Search Results: "ema"

28 April 2013

Paul Tagliamonte: Recent Hy developments

With some new patches from just oddles of interested hackers, there are some hot new changes, including a major mode for emacs and a hot new bot. I m currently blocking stuff while I m in the middle of the compiler, but new feature development will be wide open soon! Check out Hy, play with the source, install it with pip, and consider doing awesome stuff!

26 April 2013

Keith Packard: Shared Memory Fences

Shared Memory Fences In our last adventure, dri3k first steps, one of the future work items was to deal with synchronization between the direct rendering application and the X server. DRI2 handles this by performing a round trip each time the application starts using a buffer that was being used by the X server. As DRI3 manages buffer allocation within the application, there s really no reason to talk to the server, so this implicit serialization point just isn t available to us. As I mentioned last time, James Jones and Aaron Plattner added an explicit GPU serialization system to the Sync extension. These SyncFences serializing rendering between two X clients, but within the server there are hooks provided for the driver to use hardware-specific serialization primitives. The existing Linux DRM interfaces queue rendering to the GPU in the order requests are made to the kernel, so we don t need the ability to serialize within the GPU, we just need to serialize requests to the kernel. Simple CPU-based serialization gating access to the GPU will suffice here, at least for the current set of drivers. GPU access which is not mediated by the kernel will presumably require serialization that involves the GPU itself. We ll leave that for a future adventure though; the goal today is to build something that works with the current Linux DRM interfaces. SyncFence Semantics The semantics required by SyncFences is for multiple clients to block on a fence which a single client then triggers. All of the blocked clients start executing requests immediately after the trigger fires. There are four basic operations on SyncFences:

Trigger. Mark the fence as ready and wake up all waiting clients
Await. Block until the fence is ready.
Query. Retrieve the current state of the fence.
Reset. Unset the fence; future Await requests will block.

SyncFences are the same as Events as provided by Python and other systems. Of course all of the names have been changed to keep things interesting. I ll call them Fences here, to be consistent with the current X usage. Using Pthread Primitives One fact about pthreads that I recently learned is that the synchronization primitives (mutexes, barriers and semaphores) are actually supposed to work across process boundaries, if those objects are in shared memory mapped by each process. That seemed like a great simplification for this project; allocate a page of shared memory, map into the X server and direct rendering application and use the existing pthreads APIs. Alas, the pthread objects are architecture specific. I m pretty sure that when that spec was written, no-one ever thought of running multiple architectures within the same memory space. I went and looked at the code to check, and found that each of these objects has a different size and structure on x86 and x86_64 architectures. That makes it pretty hard to use this API within X as we often have both 32- and 64- bit applications talking to the same (presumably 64-bit) X server. As a last resort, I read through a bunch of articles on using futexes directly within applications and decided that it was probably possible to implement what I needed in an architecture-independent fashion. Futexes Linux Futexes live in this strange limbo of being a not-quite-public kernel interface. Glibc uses them internally to implement locking primitives, but it doesn t export any direct interface to the system call. Certainly they re easy to use incorrectly, but it s unusual in the Linux space to have our fundamental tools locked away for our own safety . Fortunately, we can still get at futexes by creating our own syscall wrappers.

static inline long sys_futex(void *addr1, int op, int val1,
                 struct timespec *timeout, void *addr2, int val3)
 
    return syscall(SYS_futex, addr1, op, val1, timeout, addr2, val3);

For this little exercise, I created two simple wrappers, one to block on a futex:

static inline int futex_wait(int32_t *addr, int32_t value)  
    return sys_futex(addr, FUTEX_WAIT, value, NULL, NULL, 0);

and one to wake up all futex waiters:

static inline int futex_wake(int32_t *addr)  
    return sys_futex(addr, FUTEX_WAKE, MAXINT, NULL, NULL, 0);

Atomic Memory Operations I need atomic memory operations to keep separate cores from seeing different values of the fence value, GCC defines a few such primitives and I picked _syncboolcompareandswap and _syncvalcompareandswap. I also need fetch and store operations that the compiler won t shuffle around:

#define barrier() __asm__ __volatile__("": : :"memory")
static inline void atomic_store(int32_t *f, int32_t v)
 
    barrier();
    *f = v;
    barrier();
 
static inline int32_t atomic_fetch(int32_t *a)
 
    int32_t v;
    barrier();
    v = *a;
    barrier();
    return v;

If your machine doesn t make these two operations atomic, then you would redefine these as needed. Futex-based Fences These wake-all semantics of Fences greatly simplify reasoning about the operation as there s no need to ensure that only a single thread runs past Await, the only requirement is that no threads pass the Await operation until the fence is triggered. A Fence is defined by a single 32-bit integer which can take one of three values:

0 - The fence is not triggered, and there are no waiters.
1 - The fence is triggered (there can be no waiters at this point).
-1 - The fence is not triggered, and there are waiters (one or more).

With those, I built the fence operations as follows. Here s Await:

int fence_await(int32_t *f)
 
    while (__sync_val_compare_and_swap(f, 0, -1) != 1)  
        if (futex_wait(f, -1))  
            if (errno != EWOULDBLOCK)
                return -1;
         
     
    return 0;

The basic requirement that the thread not run until the fence is triggered is met by fetching the current value of the fence and comparing it with 1. Until it is signaled, that comparison will return false. The compareandswap operation makes sure the fence is -1 before the thread calls futex_wait, either it was already -1 in the case where there were other waiters, or it was 0 before and is now -1 in the case where there were no waiters before. This needs to be an atomic operation so that the fence value will be seen as -1 by the trigger operation if there are any threads in the syscall. The futex_wait call will return once the value is no longer -1, it also ensures that the thread won t block if the trigger occurs between the swap and the syscall. Here s the Trigger function:

int fence_trigger(int32_t *f)
 
    if (__sync_val_compare_and_swap(f, 0, 1) == -1)  
        atomic_store(f, 1);
        if (futex_wake(f) < 0)
            return -1;
     
    return 0;

The atomic compareandswap operation will make sure that no Await thread swaps the 0 for a -1 while the trigger is changing the value from 0 to 1; either the Await switches from 0 to -1 or the Trigger switches from 0 to 1. If the value before the compareandswap was -1, then there may be threads waiting on the Fence. An atomic store, constructed with two memory barriers and a regular store operation, to mark the Fence triggered is followed by the futex_wake call to unblock all Awaiting threads. The Query function is just an atomic fetch:

int fence_query(int32_t *f)
 
    return atomic_fetch(f) == 1;

Reset requires a compareandswap so that it doesn t disturb things if the fence has already been reset and there are threads waiting on it:

void fence_reset(int32_t *f)
 
    __sync_bool_compare_and_swap(f, 1, 0);

A Request for Review Ok, so we ve all tried to create synchronization primitives only to find that our obvious implementations were full of holes. I d love to hear from you if you ve identified any problems in the above code, or if you can figure out how to use the existing glibc primitives for this operation.

Russ Allbery: WebAuth 4.5.0

Time to finally release three months of work! This is a very large feature release for the multifactor support in WebAuth and for the integration with the user information service, which is the way that WebAuth interacts with local site policy. (Currently, you still have to write your own to a well-defined protocol, although we do hope to provide Perl modules to help with this in the future.) The major driving motivation for this release is to add the infrastructure required to set long-lived persistent cookies in a browser that contribute factors to authentications, which allows WebAuth to support "remember this device" functionality and to only require multifactor from devices from which a user has not previously authenticated. This includes a new token type, new attributes the user information service can send (including a way to invalidate such tokens), and new data that's sent back to the user information service. The user information service also now has the ability to add arbitrary additional factors to the current authentication, something that is intended to provide a hook for a local help desk to bypass multifactor for a user for some time if required. This release also contains substantial contributions by Benjamin Coddington at UVM to improve multifactor interactions, including sending the OTP type back to the user information service if WebLogin knows it, a mechanism for the user information service to communicate a message to the user that's displayed on the multifactor login page, opaque state that can be sent back and forth between WebLogin and the user information service, and the ability for the user information service to add specific authentication factors to the required set for a particular authentication. Other improvements in multifactor handling include the ability to set a lifetime on factors obtained via OTP login, a fix for a long-standing bug where an initial multifactor factor would satisfy a session requirement for random multifactor, and logging of even ignored errors when contacting the user information service. There are other changes too. This release touches almost every part of WebAuth. The change to WebAuthForceLogin in 4.4.0 was reverted since, on further consideration, the original semantics seemed more useful. Password change handling in WebLogin was fixed (it's been broken for some time). Apache 2.4 error logging for all modules is much improved, and mod_webauth and mod_webkdc now produce better error logs for all versions of Apache. And WebLogin now communicates password expiration times to its templates in seconds since epoch in addition to a pre-formatted English time for better localization support. William Orr contributed a new WebAuthLdapOperationalAttribute directive for mod_webauthldap that allows it to query operational attributes and include them in the environemnt. There are two backward-incompatible changes for WebLogin. First, WebAuth now supports a user checkbox indicating either to remember their login on that device or to not remember their login (local site templates can present it either way). However, proper implementation of this matching the normal expected wording of "remember me on this device" required changing the default, so a straight upgrade from an earlier version will result in no single sign-on. To preserve behavior, either a template change to add the checkbox (checked by default) or a configuration change are required. Second, support for getting password expiration times directly with remctl to a kadmin-remctl backend has been removed in favor of using data from the user information service by way of the WebKDC. Finally, I got to do a lot of cleanup of the API, fix diagosis of undef passed to Perl XS functions, and fixed a compilation error with Heimdal. You can get the latest release from the official WebAuth distribution site or from my WebAuth distribution pages.

Vincent Sanders: When you make something, cleaning it out of structural debris is one of the most vital things you do.

Collabora recently had a problem with a project's ARM build farm. In a nice change of pace it was not that the kernel was crashing, nor indeed any of the software or hardware.
The Problem

Instead our problem was our build farm could best be described as "a pile of stuff" and we wanted to add more systems to it and have switched power control for automated testing.

Which is kinda where the Christopher Alexander quote comes into this. I suggested that I might be able to come up with a better, or at least cleaner, solution.
The Idea

Previous experience had exposed me to the idea of using 19 inch subracks for mounting circuits inside submodules.

I originally envisaged the dev boards individually mounted inside these boxes. However preliminary investigation revealed that the enclosures were both expensive and used a lot of space which would greatly increase the rack space required to house these systems.

I decided to instead look at eurocard type subracks with carriers for the systems. Using my 3D printer I came up with a carrier design for the imx53 QSB and printed it. I used the basic eurocard size of 100mm x 160mm which would allow the cards to be used within a 3U subrack.

Once assembled it became apparent that each carrier would be able to share resources like power supply, ethernet port and serial console via USB just as the existing setup did and that these would need to be housed within the subrack.
The Prototype

The carrier prototype was enough to get enough interest to allow me to move on to the next phase of the project. I purchased a Schroff 24563-194 subrack kit and three packs of guide rails from Farnell and assembled it.

Initially I had envisaged acquiring additional horizontal rails from Schroff which would enable constructing an area suitable for mounting the shared components behind the card area.

Unfortunately Schroff have no suitable horizontal profiles in their catalog and are another of those companies who seem to not want to actually sell products to end users but rather deal with wholesalers who do not have their entire product range!

Undaunted by this I created my own horizontal rail profile and 3D printed some lengths. The profile is designed to allow a 3mm thick rear cover sheet attached with M2.5 mounting bolts and fit rack sides in the same way the other profiles do.

At this point I should introduce some information on how these subracks are dimensioned. A standard 19 inch rack (as defined in IEC 60297) has a width of 17.75 inches(450.85mm) between the vertical posts. The height is measured in U (1.75 inches)

A subrack must obviously fit in the horizontal gap while providing as much internal space as possible. A subrack is generally either 3 or 6 U high. The width within a subrack is defined in units called HP (Horizontal Pitch) which are 0.2 inches(5.08 mm) and subracks like the Schroff generally list 84 usable HP.

However we must be careful (or actually just learn from me stuffing this up ;-) as the usable HP is not the same thing as the actual length of the horizontal rails! The enclosures actually leave and additional 0.1 inch at either end giving a total internal width of 85HP (17 inches, 431.8 mm) which leaves 0.75 inches for the subrack sides and some clearance.

The Schroff subrack allows eurocards to be slotted into rails where the card centre line is on HP boundaries, hence we describe the width of a card in the slot in terms of HP

I cannot manufacture aluminium extrusions (I know it is a personal failing) nor produce more than 100 mm long length of the plastic profile on my printer.

Even if full lengths are purchased from a commercial service (120 euros for a pair including tax and shipping) the plastic does not have sufficient mechanical strength.

The solution I came up with was somewhat innovative, as an alternative a M5 bolt into a thread in the aluminium extrusion I used a 444mm long length of 4mm threaded rod with nuts at either end. This arrangement puts the extrusion under compression and gives it a great deal of additional mechanical strength as the steel threaded rod is very strong.

Additionally to avoid having to print enough extrusion for the entire length I used some 6mm aluminium tube as a spacer between 6HP(30.48mm) wide sections of the printed extrusion.

It was intended to use a standard modular PC power supply which is 150mm wide which is pretty close to 30HP (6 inches) so it was decided to have a 6HP section of rail at that point to allow a rear mounting plate for the PSU to be attached.

This gives 6HP of profile, 21HP(106.68mm) of tube spacer, 6HP of profile, 46HP(233.68 mm) of tube spacer and a final 6HP profile summing to our total of 85HP. Of course this would be unnecessary if a full continuous 85HP rail had been purchased, but 6 of 6 HP long profile is only 51 euro a saving of 70 euro.

To provide a flat area on which to mount the power switching, Ethernet switch and USB hubs I ordered a 170 x 431 mm sheet of 3mm thick aluminium from inspiredsteel who, while being an ebay company, were fast, cheap and the cutting was accurate.

Do be sure to mention you would prefer it if any error made the sheet smaller rather than larger or it might not fit, for me though they were accurate to the tenth of a mm! If you would prefer the rear section of the rack to be enclosed when you are finished, buy a second sheet for the top. For my prototype I only purchased a 170 x 280mm sheet as I was unsure if I wanted a surface under the PSU (you do, buy the longer sheet)

Mounting the PSU was a simple case of constructing a 3 mm thick plate with the correct cutouts and mounting holes for an ATX supply. Although the images show the PSU mounted on the left hand side of the rack this was later reversed to improve cable management.

The subrack needed to provide Ethernet switch ports to all the systems. A TP-Link TL-SF1016DS 16-Port 10/100Mbps Switch was acquired and the switch board removed from its enclosure. The switch selected has an easily removed board and is powered by a single 3.3V input which is readily available from the ATX PSU.

Attention now returned to the eurocard carriers for the systems, the boards to be housed were iMX53 QSB and iMX6 SABRE Lite and a Raspberry Pi control system to act as USB serial console etc.

The carriers for both main boards needed to be 8HP wide, comprised of:

Combined USB and Ethernet Jack on both boards was 30 mm tall
PCB width of 2mm
underside components of 4mm
clearance between boards of 2mm

Although only 38 mm this is 7.5HP and fractions of an HP are not possible with the selected subrack.

With 8HP wide modules this would allow for ten slots, within the 84 usable HP, and an eleventh 4HP wide in which the Raspberry Pi system fits.

Carrier designs for both the i.MX53 QSB and the i.MX6 SABRE Lite boards were created and fabricated at a professional 3D print shop which gave a high quality finish product and removed the perceived risk of relying on a personal 3D printer for a quantity of parts.

This resulted in changes in the design to remove as much material as possible as commercial 3D services charge by the cubic cm. This Design For Manufacture (DFM) step removed almost 50% from the price of the initial design.

The i.MX6 design underwent a second iteration to allow for the heatsink to be mounted and not mechanically interfere with the hard drive (although the prototype carrier has been used successfully for a system that does not require a hard drive). The lesson learned here is to be aware that an design iteration or two is likely and that it is not without cost.

The initial installation was to have six i.MX53 and two i.MX6 this later changed to a five/four split, however the carrier solution allows for almost any combination, the only caveat (discovered later) is the imx53 carriers should be to the right hand side with the small 4HP gap at that end as they have a JTAG connector underneath the board which otherwise foul the hard drive of the next carrier.

Racked cards showing unwanted cable tails

A wiring loom was constructed for each board giving them a connector tail long enough to allow them to be removed. This was the wrong approach! if you implement this design (or when I do it again) the connector tails on the wiring loom should present all the connections to the rear at the same depth as the Ethernet connection.

The rack cables themselves should be long enough to allow the slides to be removed but importantly it is not desirable to have the trailing cable on the cards. I guess the original eurocard designers figured this out as they designed the cards around the standard fixed DIN connectors at the back of the card slots.

USB relay board with wiring loom attached

We will now briefly examine a misjudgement that caused the initially deployed solution to be reimplemented. As the design was going to use USB serial converters to access the serial console a USB connected relay board was selected to switch the power to each slot. I had previously used serial controlled relay boards with a USB serial convertor however these were no longer available.

Initial deployment with USB controlled relay board

All the available USB relay boards were HID controlled, this did not initially seem to be an issue and Linux software was written to provide a reasonable interface. However it soon became apparent that the firmware on the purchased board was very buggy and crashed the host computer's USB stack multiple times.

Deployed solutionOnce it became apparent that the USB controlled power board was not viable a new design was conceived. As the Ethernet switch had ports available Ethernet controlled relay boards were acquired.

It did prove necessary to design and print some PCB support posts with M3 nut traps to allow the relay boards to be easily mounted using double sided adhesive pads.

By stacking the relay boards face to face and the Ethernet switch on top separated using nylon spacers it was possible to reduce the cable clutter and provide adequate cable routing space.

A busbar for Ground (black) and unswitched 12V (yellow) was constructed from two lengths of 5A chock block.

An issue with power supply stability was noted so a load resistor was added to the 12V supply and an adhesive thermal pad used to attach it to the aluminium base plate.

It was most fortunate that the ethernet switch mounting holes lined up very well with the relay board mounting holes allowing for a neat stack.

This second edition is the one currently in use, it has proved reliable in operation and has been successfully updated with additional carriers.

The outstanding issues are mainly centered around the Raspberry Pi control board:
Needs its carrier fitting. It is currently just stuck to the subrack end plate.
Needs its Ethernet cable replacing. The existing one has developed a fault post installation.
Needs the USB hub supply separating from the device cable. The current arrangement lets the hub power the Pi which means you cannot power cycle it.
Connect its switched supply separately to the USB hub/devices.
Shopping listThe final bill of materials (excluding labour and workshop costs) which might be useful to anyone hoping to build their own version.

Prices are in GBP currency converted where appropriate and include tax at 20% and delivery to Cambridge UK and were correct as of April 2013.

The purchasing was not optimised and for example around 20GBP could be saved just by ordering all the shapeways parts in one order.
Base subrack
Item Supplier Quantity Line Price
Schroff 24563-194 subrack kit Farnell 1 41.28
Schroff 24560-351 guide rails Farnell 3 13.65
Schroff rack rear horizontal rail Shapeways 2 100.00
1000mm length of 4mm threaded rod B and Q 1 1.48
170mm x 431mm x 3mm Aluminium sheet inspired steel 2 40.00
PSU mounting plate Shapeways 1 35.42
PCB standoff Shapeways 4 22.30
160mm Deep Modular PC supply CCL 1 55.76
TP-Link TL-SF1016DS 16-Port 10/100Mbps-Switch CCL 1 23.77
8 Channel 16A Relay Board Controlled Via Ethernet Rapid 2 126.00
Raspberry Pi Farnell 1 26.48
USB Serial converters CCL 10 37.40
10 port strip style USB HUB Ebay 1 7.00
Parts for custom Ethernet cables RS 13 26.00
Parts for custom molex power cables (salvaged from scrap ATX PSU) Workshop 11 11.00
33R 10W wirewound resistor for dummy load RS 1 1.26
24pin ATX female connector pre-wired Maplin 1 2.99
Akasa double sided thermal pad Maplin 1 5.00
Small cable tie bases Maplin 1 6.49
Miscellaneous cable, connectors, nylon standoffs, solder, heatshrink, zip ties, nuts, washers etc. Workshop 1 20.00
Total for subrack 603.28

The carriers are similarly not optimally priced as over five GBP each can be saved by combining shipping on orders alone. Also the SSD drive selection was made some time ago and a newer model may be more suitable.
i.MX53 QSB carrier
Item Supplier Quantity Line Price
i.MX53 QSB Farnell 1 105.52
Intel 320 SSD 80G CCL 1 111.83
Carrier board Shapeways 1 30.00
combined sata data and power (15 to 20cm version) EBay 1 5.00
Low profile right angle 5.5mm x 2.1mm barrel jack EBay 1 0.25
Parts for 9pin serial cable extension RS 1 5.00
Miscellaneous solder, heatshrink, nylon nuts, bolts and washers Workshop 1 5.00
Total for carrier 262.60

i.MX6 SABRE Lite carrier
Item Supplier Quantity Line Price
i.MX6 SABRE Lite Farnell 1 128.06
Intel 320 SSD 80G CCL 1 111.83
Carrier board Shapeways 1 35.00
combined sata data and power (15 to 20cm version) EBay 1 5.00
Low profile right angle 5.5mm x 2.1mm barrel jack EBay 1 0.25
Parts for 9pin serial cable modification RS 1 2.00
Miscellaneous solder, heatshrink, nylon nuts, bolts and washers Workshop 1 5.00
Total for carrier 287.14
ConclusionThe solution works and in a 3U high 355mm deep subrack ten ARM development boards can be racked complete with local ethernet switching, power control and serial consoles.

The solution is neat and provides flexibility, density and reproducibility the "pile of stuff" solution failed to do.

For current prototype with nine filled slots the total cost was around 3000GBP or around 330GBP per slot which indicates a 100GBP per slot overhead over the "pile of stuff" solution. These figures omit the costs of the engineer and workshop time, which are estimated at an additional 1500GBP. Therefore a completed rack, fully filled with i.MX6 carriers costs around 5000GBP

Density could be increased if boards with lower height requirements were used however above twelve units there issues with Ethernet switch, power switch and USB port availability become a factor. For Example the 16 port Ethernet switch requires a port for uplink, one for each relay board and one for the console server which leaves only 12 ports for systems.

Addressing the outstanding issues would result in a much more user friendly solution. As the existing unit is in full time use and downtime is not easily scheduled for all ten systems, the issues are not likely to be fixed on the prototype and would have to be solved on a new build.

The solution is probably not suitable for turning into a product but that was not really the original aim. A commercial ARM blade server using this format would almost certainly use standard DIN connectors and a custom PCB design rather than adapting existing boards.

Base subrack
Schroff 24563-194 subrack kit	Farnell	1	41.28
Schroff 24560-351 guide rails	Farnell	3	13.65
Schroff rack rear horizontal rail	Shapeways	2	100.00
1000mm length of 4mm threaded rod	B and Q	1	1.48
170mm x 431mm x 3mm Aluminium sheet	inspired steel	2	40.00
PSU mounting plate	Shapeways	1	35.42
PCB standoff	Shapeways	4	22.30
160mm Deep Modular PC supply	CCL	1	55.76
TP-Link TL-SF1016DS 16-Port 10/100Mbps-Switch	CCL	1	23.77
8 Channel 16A Relay Board Controlled Via Ethernet	Rapid	2	126.00
Raspberry Pi	Farnell	1	26.48
USB Serial converters	CCL	10	37.40
10 port strip style USB HUB	Ebay	1	7.00
Parts for custom Ethernet cables	RS	13	26.00
Parts for custom molex power cables (salvaged from scrap ATX PSU)	Workshop	11	11.00
33R 10W wirewound resistor for dummy load	RS	1	1.26
24pin ATX female connector pre-wired	Maplin	1	2.99
Akasa double sided thermal pad	Maplin	1	5.00
Small cable tie bases	Maplin	1	6.49
Miscellaneous cable, connectors, nylon standoffs, solder, heatshrink, zip ties, nuts, washers etc.	Workshop	1	20.00

Total for subrack	603.28

i.MX53 QSB carrier
i.MX53 QSB	Farnell	1	105.52
Intel 320 SSD 80G	CCL	1	111.83
Carrier board	Shapeways	1	30.00
combined sata data and power (15 to 20cm version)	EBay	1	5.00
Low profile right angle 5.5mm x 2.1mm barrel jack	EBay	1	0.25
Parts for 9pin serial cable extension	RS	1	5.00
Miscellaneous solder, heatshrink, nylon nuts, bolts and washers	Workshop	1	5.00

Total for carrier	262.60

i.MX6 SABRE Lite carrier
i.MX6 SABRE Lite	Farnell	1	128.06
Intel 320 SSD 80G	CCL	1	111.83
Carrier board	Shapeways	1	35.00
combined sata data and power (15 to 20cm version)	EBay	1	5.00
Low profile right angle 5.5mm x 2.1mm barrel jack	EBay	1	0.25
Parts for 9pin serial cable modification	RS	1	2.00
Miscellaneous solder, heatshrink, nylon nuts, bolts and washers	Workshop	1	5.00

Total for carrier	287.14

Daniel Pocock: An RTC infrastructure for debian.org

A number of potential Summer of Code students are now looking at the question of how to improve RTC in Debian. Making RTC available as a service to the Debian community may be part of the solution, as this would give the community a foundation to test things against and a useful tool for communication. One core component would be a TURN server to help people traverse NAT. There are two free TURN servers already packaged in Debian, and a third one that will potentially be packaged in the near future. Challenge for SoC students: how to make the TURN authentication mechanism work with the existing passwords of people with Debian accounts or alioth accounts? All of the TURN servers would require some code changes or development of new administration tools to link the authentication systems and support LDAP. Another component would be a SIP or XMPP proxy. A very basic implementation would need to allow anybody with a debian.org account to register, make and receive calls, send and receive chat messages. Most SIP proxies and XMPP servers can do this. Challenges for SoC students: how to link up the authentication systems for user passwords? How/where can Debian/alioth users set preferences for call forwarding (adapting the code for the web interface)? How can these tools be integrated with other infrastructure such as the bug tracking system or IRC? There are many additional services that are possible with RTC. A common example is conference calling. Challenge for students: how to set up one of the free conferencing solutions so that Debian Developers can quickly set up conferences using SIP, XMPP or WebRTC? Would it be administered by a web interface? Would it generate emails inviting people to join a conference with a WebRTC link to click? Having RTC services always available on debian.org would enable RTC/VoIP packages to be automatically tested against the central infrastructure using the Jenkins continuous integration system. This would ensure that packages are always interoperable. Challenges for students: pick some of the packages and develop unit testing code for them that allows end-to-end tests against any other VoIP client software. Make it all run under Jenkins.

Jon Dowland: Debian Day #13

Wow, my first "Debian day" in 2013, but it's a bit of a misnomer because I didn't do any Debian work tonight. I spent some time poking at geary, Yorba's new email client which looks promising. I haven't managed to get it working much, though. Someone packaged an old version in Debian but it wouldn't work with either my home or my work IMAP/SMTP servers for various reasons. I did get a git checkout to build in March or April but that stopped working on Debian when they dropped "Precise" support (It seems they were backporting various GIR/Webkit bits and pieces into their own code and didn't want to carry that around any more). They've currently got a fundraiser going, they're aiming for $100k and (at the time of writing) need half of that with only 10 hours to go. Recently I've also been trying to fix a rockbox bug on Sansa Fuze v1 which really cripples writing onto the Fuze. The original firmware does not support microsd capacities >32G which rules it out for me. Sadly the v1 version of the Fuze is uncommon enough that I don't think this bug is getting much attention. My investigations have been limited to fairly dumb bisect-like approaches, combined with lots of writing onto a microsd card (which will probably be killed dead by all this). My initial attempts at a bisect have failed as it turns out the problem is the inverse of a regression: it bizarrely seems as if the bug was fixed in a version-branch, rather than stopped working in one, so it has never been squashed in the main branch at all. Furthermore, the commit that seems to fix the problem is a totally benign version bump, and almost certainly hasn't fixed the bug. So with sufficient further testing I'll probably just prove that this bug has never been fixed. It might be time to switch tack and look into diagnosing the bug rather than trying to side-step it. Or perhaps just spend 15 on a Sansa Clip Zip and forget all about the Fuze v1. Either way I hope someone releases a 128G microsd card this year! Every now and then I wonder whether it would make sense to package rockbox in Debian in some way. Probably not.

25 April 2013

David Bremner: Exporting Debian packaging patches from git, (redux)*

(Debian) packaging and Git. The big picture is as follows. In my view, the most natural way to work on a packaging project in version control [1] is to have an upstream branch which either tracks upstream Git/Hg/Svn, or imports of tarballs (or some combination thereof, and a Debian branch where both modifications to upstream source and commits to stuff in ./debian are added [2]. Deviations from this are mainly motivated by a desire to export source packages, a version control neutral interchange format that still preserves the distinction between upstream source and distro modifications. Of course, if you're happy with the distro modifications as one big diff, then you can stop reading now

gitpkg
$debian_branch $upstream_branch

and you're done. The other easy case is if your changes don't touch upstream; then 3.0 (quilt) packages work nicely with ./debian in a separate tarball. So the tension is between my preferred integration style, and making source packages with changes to upstream source organized in some nice way, preferably in logical patches like, uh, commits in a version control system. At some point we may be able use some form of version control repo as a source package, but the issues with that are for another blog post. At the moment then we are stuck with trying bridge the gap between a git repository and a 3.0 (quilt) source package. If you don't know the details of Debian packaging, just imagine a patch series like you would generate with

git
format-patch

or apply with (surprise) quilt. From Git to Quilt. The most obvious (and the most common) way to bridge the gap between git and quilt is to export patches manually (or using a helper like gbp-pq) and commit them to the packaging repository. This has the advantage of not forcing anyone to use git or specialized helpers to collaborate on the package. On the other hand it's quite far from the vision of using git (or your favourite VCS) to do the integration that I started with. The next level of sophistication is to maintain a branch of upstream-modifying commits. Roughly speaking, this is the approach taken by git-dpm, by gitpkg, and with some additional friction from manually importing and exporting the patches, by gbp-pq. There are some issues with rebasing a branch of patches, mainly it seems to rely on one person at a time working on the patch branch, and it forces the use of specialized tools or workflows. Nonetheless, both git-dpm and gitpkg support this mode of working reasonably well [3]. Lately I've been working on exporting patches from (an immutable) git history. My initial experiments with marking commits with git notes more or less worked [4]. I put this on the back-burner for two reasons, first sharing git notes is still not very well supported by git itself [5], and second Gitpkg maintainer Ron Lee convinced me to automagically pick out what patches to export. Ron's motivation (as I understand it) is to have tools which work on any git repository without extra metadata in the form of notes. Linearizing History on the fly. After a few iterations, I arrived at the following specification.

The user supplies two refs upstream and head. upstream should be suitable for export as a .orig.tar.gz file [6], and it should be an ancestor of head.
At source package build time, we want to construct a series of patches that
1. Is guaranteed to apply to upstream
2. Produces the same work tree as head, outside ./debian
3. Does not touch ./debian
4. As much as possible, matches the git history from upstream to head.

Condition (4) suggests we want something roughly like

git
format-patch

upstream..head, removing those patches which are only about Debian packaging. Because of (3), we have to be a bit careful about commits that touch upstream and ./debian. We also want to avoid outputting patches that have been applied (or worse partially applied) upstream. git patch-id can help identify cherry-picked patches, but not partial application. Eventually I arrived at the following strategy.

Use git-filter-branch to construct a copy of the history upstream..head with ./debian (and for technical reasons .pc) excised.
Filter these commits to remove e.g. those that are present exactly upstream, or those that introduces no changes, or changes unrepresentable in a patch.
Try to revert the remaining commits, in reverse order. The idea here is twofold. First, a patch that occurs twice in history because of merging will only revert the most recent one, allowing earlier copies to be skipped. Second, the state of the temporary branch after all successful reverts represents the difference from upstream not accounted for by any patch.
Generate a "fixup patch" accounting for any remaining differences, to be applied before any if the "nice" patches.
Cherry-pick each "nice" patch on top of the fixup patch, to ensure we have a linear history that can be exported to quilt. If any of these cherry-picks fail, abort the export.

Yep, it seems over-complicated to me too. TL;DR: Show me the code. You can clone my current version from

git://pivot.cs.unb.ca/gitpkg.git

This provides a script "git-debcherry" which does the history linearization discussed above. In order to test out how/if this works in your repository, you could run

git-debcherry --stat $UPSTREAM

For actual use, you probably want to use something like

git-debcherry -o debian/patches

There is a hook in hooks/debcherry-deb-export-hook that does this at source package export time. I'm aware this is not that fast; it does several expensive operations. On the other hand, you know what Don Knuth says about premature optimization, so I'm more interested in reports of when it does and doesn't work. In addition to crashing, generating multi-megabyte "fixup patch" probably counts as failure. Notes

This first part doesn't seem too Debian or git specific to me, but I don't know much concrete about other packaging workflows or other version control systems.
Another variation is to have a patched upstream branch and merge that into the Debian packaging branch. The trade-off here that you can simplify the patch export process a bit, but the repo needs to have taken this disciplined approach from the beginning.
git-dpm merges the patched upstream into the Debian branch. This makes the history a bit messier, but seems to be more robust. I've been thinking about trying this out (semi-manually) for gitpkg.
See e.g. exporting. Although I did not then know the many surprising and horrible things people do in packaging histories, so it probably didn't work as well as I thought it did.
It's doable, but one ends up spending about a bunch lines of code on duplicating basic git functionality; e.g. there is no real support for tags of notes.
Since as far as I know quilt has no way of deleting files except to list the content, this means in particular exporting upstream should yield a DFSG Free source tree.

22 April 2013

Daniel Pocock: Action week for Google Summer of Code students

This week (22 - 28 April) is the week that really matters most for those students who want to participate in Google Summer of Code. The deadline for student applications is next Friday (3 May), but if you don't spend time exploring project ideas this week, you won't be able to make a strong enough application. Here are the key things I need to see for potential students for the project areas I have offered to co-mentor in:

Register on the Google site, Click here to register yourself (registration opens 22 April and closes 3 May).
Create a page about yourself on the Debian wiki, you must copy the template used by the other students. Make sure you create a link from the student list to your own page. Think carefully about your project: what will be the final result of your work this summer? Describe what it will look like, how Debian people will use it, and explain these things on your wiki page, this will be critical to having Debian accept you.

Those are the essential things you must do before the deadline on 3 May. There are other things you should do as well:

Join the Summer of Code co-ordination mailing list
Join the mailing lists for whichever open source project and/or Debian packaging team is relevant to your proposed GSoC project
On the mailing list, send an email introducing yourself, giving us the link to your wiki page.

To maximize the chance that the Debian project will accept you, it is very important that you interact with us and show us that you understand software development. I am going to try and make this easier: for each of my projects, I've put some wishlist items in the Debian bug tracker. These are all easy tasks that a student can potentially complete. Just do the following:

Pick a task from the bug tracker
Send an email to the bug tracker (e.g. <bug-number>@bugs.debian.org>) explaining that you will work on the task (and make sure nobody else is already working on it, just look to see if any other emails are attached to the bug report)
Write the patch, test it and email the patch to the bug report

Here are the links to the bug lists for each of the packages. You can also pick any other wishlist bug in any related package, but if you are not sure, just pick one of these:

For real-time communications and VoIP, look at wish-list bugs against the reSIProcate package
For one-time password authentication, look at wish-list bugs against the dynalogin package
For Improving PKI on Debian, look at wish-list bugs against the openssl package

If you already have a history of contributing to free software, it is essential that you tell us about it. This will increase the chances that your application is successful. Specifically, on your Debian wiki page, you should include links to any of the following:

Emails you sent to open source communities contributing patches or analysis of bugs (give a link to your message in the email archive for the list)
Your github account profile page, if you have one
Your Sourceforge account profile page, if you have one
Any other similar links

Debian receives many applications every year. To help you succeed, you need to think about how Debian will benefit from your work. Use Google to search the Debian site for previous discussions about your idea. Find out the names of Debian Developers who are interested in the same subjects, and email us or email the public mailing lists of the Debian project, for example, debian-devel. Make sure your email includes links to previous discussions or wiki pages about the idea, explain how you can help and ask the community for suggestions and guidance to refine your project proposal. Debian is an open community and we are all very keen to discuss your ideas and help you get involved. DebConf For all students, attending free software events is a great opportunity to meet the community. Of particular relevance for those in Europe, attending DebConf13 in Switzerland is a great way to meet GSoC mentors and other developers. Switzerland's central location is extremely convenient for anybody in Western Europe and it is also Debian's 20th birthday, so it is anticipated DebConf could be more special than usual this year. Please put it in your calendar and subscribe to the debconf-discuss mailing list so you will know when registration opens and other practical details are announced.

18 April 2013

Daniel Pocock: Getting to Switzerland, travel costs

With so many people contemplating the cost of travel to Switzerland this summer, I thought I would share some eye-opening insights into how to get a better deal. Let's look at some costs for a hypothetical DebConf13 visitor coming from New York. Flight prices taken today from a comparison site for 8 - 18 August, car hire prices from Europcar and rail prices from SBB (regular tickets) and Swiss Travel System (tourist rail passes):

Departure airport	Arrival airport	Flight cost	Bus to Swiss border	Rail pass	Total (Flight+bus/train)	Airport car hire, 10 days	Total (Flight+car)
New York	Geneva	$1,153	n/a	CHF 82 Return ticket	$1,235	CHF 668	$1,823
New York	Milan	$990	$20	CHF 315 Flexi pass	$1,350	EUR 413 CHF 496	$1,490

Warning: car rental prices based on advance booking - if you just arrive at a Swiss airport and hire a car on the spot, it is likely to be much more expensive, maybe CHF 200 per day For a single traveler, it is slightly cheaper to fly into Geneva (closest airport) and buy a return train ticket, as long as no other travel in Switzerland is desired and sleeping on-site at Vaumarcus, not using a bus every day.

If traveling in Switzerland to see the mountains, then it works out cheaper to fly into Milan, Italy, which is very close to the Swiss border. Most of the journey from Milan to Vaumarcus can be covered using one of the Swiss railway travel passes, with stops for sight-seeing on the way, and the other days on the pass used for sight-seeing. Now imagine people arriving in a group: the rail ticket prices quickly add up. For a family of four, 4 * 315 = CHF 1,260. Car rental may be a better option. Hiring a car in Switzerland is expensive, hiring in Italy is much cheaper, the plane tickets to Italy are cheaper too, so there is a lot of money to be saved.

But who wants the hassle of travelling to Switzerland via Italy? Of course that is the silliest rhetorical question you are going to see this week: visiting Italy is not a hassle at all. Even the Swiss enjoy going there so much that they just built the world's longest rail tunnel under the Alps to bring Como and Milan that little bit closer, the Gotthard Base Tunnel. The Italian connection could involve a few days in Como or Brunate. If designer fashion labels are appealing to you (or as a gift), Vertemate has one well known outlet for a major brand and the prices really are much cheaper than any of the cities. If you want a laugh, it's worth comparing the hotel prices in Lugano on the Swiss side of the border against prices in nearby Como. Traveling north from Como, taking a train to Luzern (and a connection to Vaumarcus), it is possible to stop in Goschenen and go up into the mountains to experience the Glacier Express featured in my earlier video or get off the train at Arth-Goldau to cross Rigi on the Rigi Bahn (from my other video), take a boat across the lake to Luzern and then continue to Vaumarcus. The Italian entry point isn't just an option for those coming from the US. Even a traveler from London will find that the budget airlines have largely snubbed the Swiss airports with their high operating costs. Easyjet offers a limited number of flights to Geneva and Zurich, but the prices are always higher than their other destinations. Ryanair offers flight's to Milan's Bergamo airport and Easyjet flies to Milan Malpensa.

17 April 2013

Daniel Pocock: Switzerland Schilthorn with 007 (leg 3)

Friday's video of the 2nd leg of the cable car journey was quite popular, Ganglia stats for the video server can't be wrong:

The first leg was skipped because of the low cloud cover. Weather reports in Switzerland, such as www.meteosuisse.ch often show a distinction between the low clouds and high clouds, because many Swiss mountain resorts are high enough to enjoy ample sunshine by virtue of their altitude even when cities like Zurich and Geneva are overcast. Now for the third leg, rising up to Berg. In the video, it is hard to gauge the size of the rocky cliffs along the route, but looking at the cable car's shadow will give a clue. As the cable car approaches the station in Berg, it feels like it takes forever to complete the journey. That is because the rocks are so much bigger than they appear from a distance. <video controls="" height="340" poster="http://upload.wikimedia.org/wikipedia/commons/thumb/0/03/Schilthorn_with_Bernese_Alps%2C_2012_August.jpg/800px-Schilthorn_with_Bernese_Alps%2C_2012_August.jpg" width="560">
<source src="http://video.danielpocock.com/VID_20121025_122453.m4v" type="video/mp4"></source>
<source src="http://video.danielpocock.com/VID_20121025_122453.webm" type="video/webm"></source>
</video> The final leg to the Schilthorn summit will appear shortly. I understand many people are starting to contemplate their DebConf13 travel and summer holidays - please feel free to email me if you'd like me to post a video of some other feature in Switzerland.

Attachment	Size
ganglia-schilthorn2.png	8.71 KB

16 April 2013

Stefano Zacchiroli: bits from the DPL for March-April 2013

Dear Project Members,

   "Now that I have your attention, I would like to make the following
delegations:"

... nah, scrap that. In my last day in office I first of all owe you a report of DPL activities for the last reporting period of this term, i.e. March 8th until today. Here it is! Highlights

At LibrePlanet (see below) I've discussed at length with Karen Sandler as GNOME representative the possibility of Debian participation in the FOSS Outreach Program for Women. I've then proposed that we do participate and, as you might have read on d-d-a, we're now doing that. Many thanks to the volunteer co-organizers for Debian participation in the program: M nica Ram rez Arceda, Ana Guerrero L pez, and Patty Langasek.
A couple of years of work with the auditors has come together. At elgar.debian.org:/srv/accounting.debian.org/ledger/ you can now find Debian monetary transactions for the 2010-2013 period. Note that: 1) They are not all of our transactions, most notably because we haven't yet managed to get access to all our bank transactions at SPI (while we do have access to other transactions there, e.g. donations). Given the relevance of the missing transactions for our budget, this is a blocker for producing meaningful public periodic reports of Debian finances. This is clearly annoying, but I'm confident that our feedback to SPI over the past years has helped them better understand our needs and improve. I hope this could be finally solved during the next DPL term. And, 2) Donor names have been anonymized in the ledger files, in wait of a donation system that allow to express privacy preferences. Complete donation information is available in the companion ledger.git repository, which is accessible to auditors only. I'd like to thank the Debian auditors, and in particular Martin Michlmayr, for their amazing work on this over the past 3 years.
As you might have noticed, we now have an official Debian Project blog, finally entering the brave new Web 1.5 era I've only helped "politically" here and there with this over the years, and I'm happy to see it live. Your thanks for this should go to the blog editors---Francesca Ciceri and Ana Guerrero Lopez---and to DSA for making it real. A proper delegation for the editors is pending and I'm confident the next DPL will pick it up.

Talks Over the past month or so I've attended and spoken on behalf of Debian in the following occasions:

"Debian: 20 years and counting" talk at the New York Linux User Group; slides are available. Many thanks to Brian Gupta and Tom Limoncelli for inviting me.
"Debian and GNU" talk at LibrePlanet 2013; slides are available. Many thanks to John Sullivan and the Free Software Foundation for inviting me to talk at their main conference. My presence there has also been a chance to reassess the status of collaboration with FSF (see John's brief summary) and discuss further technical collaboration with the Trisquel maintainers.
"Legal issues from a radical community angle" keynote at the yearly workshop of FSFE's European Legal Network; slides are available. The talk has also been covered by a LWN feature article last week (the link should become unembargoed for non-LWN subscribers starting tomorrow).

Assets I've approved the budget for the following forthcoming sprints:

6th Debian Groupware Meeting (~500 EUR)
DSA sprint (~3'000 EUR)

Also, we've bought a 3-year warranty pack for the disk array that powers ftp-master.d.o (~900 USD). On the income side, Brian Gupta has started an interesting matching fund experiment, in order to raise funds for the forthcoming DebConf13. The matching fund will be open until April 30th, so your help in spreading news would be welcome. Many thanks to Brian for the idea and to his company, Brandorr Group, for funding it. DPL helpers Three more DPL helpers IRC meetings have been held; minutes are available at the usual place. Legal Spring Cleaning I've finally cleaned up the pile of pending legal matters (but I'm sure new ones will show up for the delight of the next DPL :-P)

one is merely internal for trademark@d.o: our procedures for (n)acks on incoming requests has now been vetted by our legal advisors
the second one is relevant for our mentors.debian.net service: one of the blockers to officialize it as mentors.debian.org have historically been DMCA-related concerns. We now have a DMCA policy for (wannabe) mentors.d.o, which I've shared with the service maintainers and DSA. This specific part should no longer be a concern.
the last one concerns the possibilities of playing DVDs with Debian. We now have legal guidelines on how to include installer packages that allow to do so; that should allow us to have a decent solution for our users in the Jessie time frame.

Once again, I'd like to thank SFLC for the pro bono and very high quality legal advice they keep on offering to Debian. Miscellaneous

I've mentioned last month that, as a Debian representative, I've joined a working group by the Italian public administration (PA) that should define procurement rules for software in the PA at large, together with representatives of other well-known FOSS initiatives (e.g. KDE, FSFE). The first meetings have now been held and I've participate in some, for the moment on my own budget. I'll check with the next DPL the feasibility of keep on doing so on in the future.

Now, before I get sentimental, let me thank Gergely, Lucas, and Moray for running in the recently concluded DPL election. Only thinking of running and then go through a campaign denote a very high commitment to the Project; we should all be thankful to them. Then I'd like to congratulate Lucas for his election. I've known him for a long time, and I can testify about his clear vision of the role Debian has to play in Free Software and on what Debian needs to improve to do so. Best wishes for the term ahead, Lucas! Finally, I'd like to thank you all for the support you've shown me over the past 3 years. Serving as DPL is a great honor, but also a very demanding job. Thank to you all, and to how cool Debian is, it has been for me an incredibly rewarding experience. I had no idea what I were doing when I embarked on this adventure, but in hindsight I don't regret any of it. See you around, as I don't plan to be anywhere far away from Debian anytime soon. Cheers.

PS the day-to-day activity logs for March and April 2013 are available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.20130 3,4

Craig Small: Removing itools

For very many years I have been running a set of tools on my website that basically runs whois or nslookup queries and presents them in a standard format. I have decided today to shut this part of the website down as the code running those components is very old and I ve not maintained it for years. Back when I initially wrote the tools, in 1995 or so, there wasn t many alternatives to this site but that has long changed. So thanks for those who emailled me over the years; its been an interesting journey.

Johannes Schauer: botch - the debian bootstrap software has a name

After nearly one year of development, the "Port bootstrap build-ordering tool" now finally has a name: botch. It stands for Bootstrap/Build Order Tool CHain. Now we dont have to call it "Port bootstrap build-ordering tool" anymore (nobody did that anyways) and also dont have to talk about it by referring to it as "the tools" in email, IRC or in my master thesis which is due in a few weeks. With this, the issue also doesnt block the creation of a Debian package anymore. Since only a handful of people have a clone of it anyway, I also renamed the gitorious git repository url (and updated all links in this blog and left a text file informing about the name change in the old location). The new url is: https://gitorious.org/debian-bootstrap/botch Further improvements since my last post in January include:

greatly improved speed of partial order calculation
calculation of strong edges and strong subgraphs in build graphs and source graphs
calculation of source graphs from scratch or from build graphs
calculation of strong bridges and strong articulation points
calculate betweenness of vertices and edges
find self-cycles in the source graph
add webselfcycle code to generate a HTML page of self-cycles
allow to collapse all strongly connected components of the source graph into a single vertex to make it acyclic
allow to create a build order for cyclic graphs (by collapsing SCC)
allow to specify custom installation sets
add more feedback arc set heuristics (eades, insertion sort, sifting, moving)
improve the cycle heuristic

In the mean time a paper about the theoretical aspects of this topic which Pietro Abate and I submitted to the CBSE 2013 conference got accepted (hurray!) and I will travel to the conference in Canada in June. Botch (yeey, I can call it by a name!) is also an integral part of one of the proposals for a Debian Google Summer of Code project this year, mentored by Wookey and co-mentored by myself. Lets hope it gets accepted and produces good results in the end of the summer!

14 April 2013

Vasudev Kamath: Handling C++ symbols that vary across architectures

Disclaimer I'm no symbol expert and dealing with symbol files for first time from the time I started packaging. What I did here is depending on some suggestions I got in #debian-mentors. If you think what I did was wrong please enlighten me :-). Problem Recently 2 of my library packages pugixml and ctpp2 got accepted into Debian archive and when buildd tried to build them on remaining architectures other than one for which I uploaded (amd64) builds failed. This was expected as symbols file which I generated was for amd64. As usual I got 2 serious bug reports #704718 and #705135. Solution First I was not sure how to handle this. I read article on symbols files handling by rra [1] and tried to use pkgkde-symbolshelper tool only to quickly figure out that I need to use pkgkde_symbolshelper addon for dh sequencer. But this was not possible for me as I was using CDBS for packaging. I did a quick chat on #debian-mentors and some one suggested me to tag symbols which vary across architecture with (c++) tag. First I was not sure but after reading dpkg-gensymbols man page understood that I need to replace the entire mangled symbols lines with de-mangled version tagged with (c++) in the beginning. But this was hectic job searching for each deleted symbols and replacing it. So I thought of writing a script to do the job and after struggling for 3 days (yeah I was bit dumb that I didn't read manual date: "2013-04-14 18:39:20 +0530" pugixml and ctpp2 package using it, which is now waiting for Jonas for uploading it. Here is the script

#!/bin/bash
set +x
if [ $# -lt 3 ]; then
    echo "Usage: $0 failed_buildlogs_directory symbols_file package_version"
    exit 2
fi
    
BUILD_LOG_DIRECTORY=$1
SYMBOLS_FILE=$2
PACKAGE_VERSION=$3
VERSION_TO_REPLACE=" $PACKAGE_VERSION\""
for LOGFILE in $(ls $BUILD_LOG_DIRECTORY/*.build); do
    for i in $(grep '^-\s_Z' $LOGFILE   perl -pe 's/-//g;'); do
        if [ $i = $PACKAGE_VERSION ];then
            continue
        fi
        demangled_version="\""$(echo $i" "$PACKAGE_VERSION   c++filt)"\""
        tagged_version="(c++)"$ demangled_version%$VERSION_TO_REPLACE "\" "$PACKAGE_VERSION
        escaped_tagged_version=$(echo $tagged_version   sed 's/\&/\\\&/')
        sed -i "s#$i $PACKAGE_VERSION#$escaped_tagged_version#" $SYMBOLS_FILE
   done
done

So basically to make this work we need all buildlogs to be downloaded from buildd's again this was easy thanks to rra for developing pkgkde-getbuildlogs :-). Once you have build logs directory run the above script as follows

cppsymbol_replace.sh path_to_buildlogs path_to_symbol_file upstream_version

After replacing symbols I tried to build package on i386 chroot and build passed successfully but lintian told me that there are symbols which have Debian version appended to it and this might lead to date: "2013-04-14 18:42:42 +0530" back to mentors :-). It is this time I really understood concept of mangled names generated by compiler and why it vary across the architecture ;-). This time some one by nick pochu suggested me to pass option -v with package version to dpkg-gensymbols to make it generate symbols with package version and not Debian version. The following probably needs to be done if package uses dh sequencer but I'm not sure as I've not tested it. If wrong please correct me.

override_dh_makeshlibs:
    dh_makeshlibs -- -v$PACKAGEVERION #package version needs to be either extracted using parsechangelog or manually fed

If you are using CDBS this is pretty simple. Just add following to rules

DEB_DH_MAKESHLIBS_ARGS_$(pkgname) += -- -v$(DEB_UPSTREAM_VERSION)

I noticed that when I provide (c++) tagged de-mangled name dpkg-gensymbols simply replaces it with proper mangled name but the deletion doesn't trigger an error in dpkg-gensymbols. This script me allowed to replace 128 symbols which were very tricky and long with de-mangled and tagged version in ctpp2, so I hope it should work across different packages without any problem. Only silly mistake I did was occurrence of & symbol in function making sed go mad which I took one full day to debug :facepalm:. So that's it folks, if you see something wrong in what I did please let me know through the comments. [1] http://www.eyrie.org/~eagle/journal/2012-01/008.html

12 April 2013

Craig Small: Pie Charts in TurboGears

You might of looked at Ralph Bean s tutorial on graphs and thought, that s nice but I d like something different. The great thing about ToscaWidgets using the jqPlot library is that pretty much anything you can do in jqPlot, you can do in ToscaWidgets and by extension in TurboGears. You want different graph types? jqPlot has heaps! My little application needed a Pie Chart to display the overall status of attributes. There are 6 states an attribute can be in: Up, Alert, Down, Testing, Admin Down and Unknown. The Pie Chart would show them all with some sort of colour representing the status. For this example I ve used hard-coded data but you would normally extract it from the database using a TurboGears model and possibly some bucket sorting code. I ve divided my code into widget specific, which is found in myapp/widgets/attribute.py and the controller code found unsurprisingly at myapp/controllers/attribute.py Also note that some versions of ToscaWidgets have a bug which means any jqPlot widget won t work, version 2.0.4 has the fix for issue 80 that explains briefly the bug. The widget code looks like this:

from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import pieRenderer_js
import tw2.core as twc
 
class AttributeStatusPie(JQPlotWidget):
  """
  Pie Chart of the Attributes' Status """
  id = 'attribute-status-pie'
  resources = JQPlotWidget.resources + [
  pieRenderer_js,
  ]
 
  options =  
  'seriesColors': [ "#468847", "#F89406", "#B94A48", "#999999", "#3887AD", "#222222"], 
  'seriesDefaults' :  
  'renderer': twc.js_symbol('$.jqplot.PieRenderer'),
   ,
  'legend':  
  'show': True, 
  'location': 'e',
   ,

Some important things to note are:

resources are the way of pulling in the javascript includes that actually do the work, generally if you have something like a renderer using js_symbol further on, it needs to be listed in the resources too.
seriesColors is how you make a specific data item a specific colour, or perhaps change the range of colours. It s not required if you use the default set, which is defined in the jqPlot options.
The renderer tells jqPlot what sort of graph we want, the line above says we want pie

Next the controller needs to be defined:

from myapp.widgets.attribute import AttributeStatusPie
 
@expose('myapp.templates.widget')
  def statuspie(self):
  data = [[
  ['Up', 20], ['Alert', 7], ['Down', 12], ['Admin Down', 3], ['Testing', 1], ['Unknown', 4],
  ]]
  pie = AttributeStatusPie(data=data)
  return dict(w=pie)

And that is about it, we now have a controller path attributes/statuspie which shows us the pie chart.
My template is basically a bare template with a $ w.display n in it to just show the widget for testing.

Pie Chart in Turbogears

Keith Packard: dri3k first steps

DRI3K First Steps Here s an update on DRI3000. I ll start by describing what I ve managed to get working and then summarize discussions that happened on the xorg-devel mailing list. Private Back Buffers One of the big goals for DRI3000 is to finish the job of moving buffer management out of the X server and into applications. The only thing still allocated by DRI2 in the X server are back buffers; everything else moved to the client side. Yes, I know, this breaks the GLX requirement for sharing buffers between applications, but we just don t care anymore. As a quick hack, I figured out how to do this with DRI2 today allocate our back buffers separately by creating X pixmaps for them, and then using the existing DRI2GetBuffersWithFormat request to get a GEM handle for them. Of course, now that all I ve got is a pixmap, I can t use the existing DRI2 swap buffer support, so for now I m just using CopyArea to get stuff on the screen. But, that works fine, as long as you don t care about synchronization. Handling Window Resize The biggest pain in DRI2 has been dealing with window resize. When the window resizes in the X server, a new back buffer is allocated and the old one discarded. An event is delivered to invalidate the old back buffer, but anything done between the time the back buffer is discarded and when the application responds to the event is lost. You can easily see this with any GL application today resize the window and you ll see occasional black frames. By allocating the back buffer in the application, the application handles the resize within GL; at some point in the rendering process the resize is discovered, and GL creates a new buffer, copies the existing data over, and continues rendering. So, the rendered data are never lost, and every frame gets displayed on the screen (although, perhaps at the wrong size). The puzzle here was how to tell that the window was resized. Ideally, we d have the application tell us when it received the X configure notify event and was drawing the frame at the new size. We thought of a cute hack that might do this; track GL calls to change the viewport and make sure the back buffer could hold the viewport contents. In theory, the application would receive the X configure notify event, change the viewport and render at the new size. Tracking the viewport settings for an entire frame and constructing their bounding box should describe the size of the window; at least it should describe the intended size of the window. There s at least one serious problem with this plan applications may well call glClear before calling glViewport, and as glClear does not use the current viewport, instead clearing the whole window, we couldn t use the viewport as an indication of the current window size. However, what this exercise did lead us to realize was that we don t care what size the window actually is, we only care what size the application thinks it is. More accurately, the GL library just needs to be aware of any window configuration changes before the application, so that it will construct a buffer that is not older than the application knowledge of the window size. I came up with two possible mechanisms here; the first was to construct a shared memory block between application and X server where the X server would store window configuration changes and signal the application by incrementing a sequence number in the shared page; the GL library would simply look at the sequence number and reallocate buffers when it changed. The problem with the shared memory plan was that it wouldn t work across the network, and we have a future project in mind to replace GLX indirect rendering with local direct rendering and PutImage which still needs accurate window size tracking. More about that project in a future post though X Events to the Rescue So, I decided to just have the X server send me events when the window size changed. I could simply use the existing X configure notify events, but that would require a huge infrastructure change in the application so that my GL library could get those events and have the application also see them. Not knowing what the application is up to, we d have to track every ChangeWindowAttributes call and make sure the event_mask included the right bits. Ick. Fortunately, there s another reason to use a new event we need more information than is provided in the ConfigureNotify event; as you know, the Swap extension wants to have applications draw their content within a larger buffer that can have the window decorations placed around it to avoid a copy from back buffer to window buffer. So, our new ConfigureNotify event would also contain that information. Making sure that ConfigureNotify event is delivered before the core ConfigureNotify event ensures that the GL library should always be able to know about window size changes before the application. Splitting the XCB Event Stream Ok, so I ve got these new events coming from the X server. I don t want the application to have to receive them and hand them down to the GL library; that would mean changing every application on the planet, something which doesn t seem very likely at all. Xlib does this kind of thing by allowing applications to stick themselves into the middle of the event processing code with a callback to filter out the events they re interested in before they hit the main event queue. That s how DRI2 captures Invalidate events, and it works , but using callbacks from the middle of the X event processing code creates all kinds of locking nightmares. As discussed above, I don t care when GL sees the configure events, as long as it gets them before the application finds about about the window size change. So, we don t need to synchronously handle these events, we just need to be able to know they ve arrived and then handle them on the next call to a GL drawing function. What I ve created as a prototype is the ability to identify specific events and place them in a separate event queue, and when events are placed in that event queue, to bump a sequence number so that the application can quickly identify that there s something to process. Making the Event Mask Per-API Instead of Per-Client The problem described above about using the core ConfigureNotify events made me think about how to manage multiple APIs all wanting to track window configuration. For core events, the selection of which events to receive is all based on the client; each client has a single event mask, and each client receives one copy of each event. Monolithic applications work fine with this model; there s one place in the application selecting for events and one place processing them. However, modern applications end up using different APIs for 3D, 2D and media. Getting those libraries to cooperate and use a common API for event management seems pretty intractable. Making the X server treat each API as a separate entity seemed a whole lot easier; if two APIs want events, just have them register separately and deliver two events flagged for the separate APIs. So, the new DRI3 configure notify events are created with their own XID to identify the client-side owner of the event. Within the X server, this required a tiny change; we already needed to allocate an XID for each event selection so that it could be automatically cleaned up when the client exited, so the only change was to use the one provided by the client instead of allocating one in the server. On the wire, the event includes this new XID so that the library can use it to sort out which event queue to stick the event in using the new XCB event stream splitting code. Current Status The above section describes the work that I ve got running; with it, I can run GL applications and have them correctly track window size changes without losing a frame. It s all available on the dri3 branches of my various repositories for xcb proto, libxcb, dri3proto and the X server. Future Directions The first obvious change needed is to move the configuration events from the DRI3 extension to the as-yet-unspecified new Swap extension (which I may rename as Present , as in please present this pixmap in this window ). That s because they aren t related to direct rendering, but rather to tracking window sizes for off-screen rendering, either direct, indirect or even with the CPU to memory. DRI3 and Fences Right now, I m not synchronizing the direct rendering with the CopyArea call; that means the X server will end up with essentially random contents as the application may be mid-way through the next frame before it processes the CopyArea. A simple XSync call would suffice to fix that, but I want a more efficient way of doing this. With the current Linux DRI kernel APIs, it is sufficient to serialize calls that post rendering requests to the kernel to ensure that the rendering requests are themselves serialized. So, all I need to do is have the application wait until the X server has sent the CopyArea request down to the kernel. I could do that by having the X server send me an X event, but I think there s a better way that will extend to systems that don t offer the kernel serialization guarantee. James Jones and Aaron Plattner put together a proposal to add Fences to the X Sync extension. In the X world, those offer a method to serialize rendering between two X applications, but of course the real goal is to expose those fences to GL applications through the various GL sync extensions (including GLARBsync and GLNVfence). With the current Linux DRI implementation, I think it would be pretty easy to implement these fences using pthread semaphores in a block of memory shared between the server and application. That would be DRI-specific; other direct rendering interfaces would use alternate means to share the fences between X server and application. Swap/Present The Second Extension By simply using CopyArea for my application presentation step, I think I ve neatly split this problem into manageable pieces. Once I ve got the DRI3 piece working, I ll move on to fixing the presentation issue. By making that depend solely on existing core Pixmap objects as the source of data to present, I can develop that without any reference to DRI. This will make the extension useful to existing X applications that currently have only CopyArea for this operation. Presentation of application contents occurs in two phases; the first is to identify which objects are involved in the presentation. The second is to perform the presentation operation, either using CopyArea, or by swapping pages or the entire frame buffer. For offscreen objects, these can occur at the same time. For onscreen, the presentation will likely be synchronized with the scanout engine. The second form will mean that the Fences that mark when the presentation has occurred will need to signaled only once the operation completes. A CopyArea operation means that the source pixmap is ready immediately after the Copy has completed. Doing the presentation by using the source pixmap as the new front buffer means that the source pixmap doesn t become ready until after the next swap completes. What I don t know now is whether we ll need to report up-front whether the presentation will involve a copy or a swap. At this point, I don t think so the application will need two back buffers in all cases to avoid blocking between the presentation request and the presentation execution. Yes, it could use a fence for this, but that still sticks a bubble in the 3D hardware where it s blocked waiting for vblank instead of starting on the next frame immediately. Plan of Attack Right now, I m working on finishing up the DRI3 piece:

Replace the DRI2 buffer allocation kludge with actual local buffer allocation, mapping them into pixmaps using FD passing.
Replace the DRI2 authentication scheme with having the X server open the DRI object, preparing it for rendering and passing it back to the application.
Working on the XCB pieces to get the split event-queue stuff landed upstream.
Implementing the Fencing stuff to correctly serialize access to the pixmap.

The first three seem fairly straight forward. The fencing stuff will involve working with James and Aaron to integrate their XSync changes into the server. After that, I ll start working on the presentation piece. Foremost there is figuring out the right name for this new extension; I started with the name Swap as that s the GL call it implements. However, Swap is quite misleading as to the actual functionality; a name more like Present might provide a better indication of what it actually does. Of course, Present is both a verb and a noun, with very different connotations. Suggestions on this most complicated part of the project are welcome!

Michal Čihař: Hackweek 9 is over

Hackweek 9 is over and it's time to share what I've done on Weblate during that. I think everything went quite well and Weblate is now ready for 1.5 release. I'm slowly deploying it on my installations (unfortunately this release migration will need some noticeable downtime for bigger installations) and everything seems to work fine so far. I believe this is possible thanks to massive test coverage - all important code is covered by testcases. So what you can expect in 1.5 release? The most visible change is probably new machine translation support, providing support for way more backends and allow you to plug in own services as well. The other changes include word counting (what might give you more idea how much work is remaining) or fancy progress bars in all places (they used to be available for translations only). From the other side, Weblate can now run custom scripts to pre-process translations before commit, what can be used for various things from generating byte compiled files to sorting or cleaning up the translation files. Also Weblate should be now much faster - there were dozen of optimizations done, leading to much lower press on database server. If you want to see more detailed work progress, check Hackweek project page or Weblate changelog. PS: In case no problems appear, Weblate 1.5 should be released on Sunday.

Filed under: English Phpmyadmin Suse Weblate 0 comments Flattr this!

11 April 2013

Russ Allbery: Hugo nominee haul

I need to write up new Kindle books, of which there are now quite a few due to various sales plus the Hugo nominee slate, but I got another set of paper books and they're sitting in front of me. So here's a list. Saladin Ahmed Throne of the Crescent Moon (sff)
Elizabeth Bear Shattered Pillars (sff)
Ta-Nehisi Coates The Beautiful Struggle (non-fiction)
Guy Gavriel Kay River of Stars (sff)
Jenny Larson Let's Pretend This Never Happened (non-fiction)
Domenica Ruta With or Without You (non-fiction)
Jay Wexler The Odd Clauses (non-fiction) Ahmed's book is the remaining Hugo nominee that I didn't already pick up. I'm delighted to see the diversity on the Hugo and particularly the Nebula slate this year, and I'm curious to see the spin that Ahmed brings to the epic fantasy genre. Bear's book is the sequel to Range of Ghosts, another book I'm very much looking forward to reading to but haven't yet. I'm rather behind on reading Bear's work right now. Let's Pretend This Never Happened, With or Without You, and The Beautiful Struggle are all memoirs, of varying degrees of seriousness. I've gotten hooked on Coates's writing at The Atlantic, and I highly recommend it if you've not yet seen it. Domenica Ruta's memoir is the latest Indiespensible selection. Jay Wexler's book was recommended by Lowering the Bar, a legal humor blog whose entire archives I'm slowly reading. But the highlight of this order is Kay's River of Stars, which is a sequel of sorts to one of my favorite books ever. This is probably the book I'll read during my next vacation.

Bdale Garbee: Batteries and Pyro Circuits

Keith and I have discovered a change in the behavior of the protection circuits integrated in the LiPo batteries we sell for use with Altus Metrum products that poses a risk for our customers. This post is meant to document what we now know, communicate changes we're planning as a result, and explain what we think flyers of our existing electronics and batteries may want to do to maximize their chances of successful flights. Background Choosing batteries and designing pyro circuits for high power rocketry avionics involves a variety of trade-offs. Reliability is the highest concern, both because nobody wants to lose an airframe due to a failed pyro event, and also because airframes recovering anomalously have safety implications. But we also care about other factors including cost and weight, and usually want to minimize the complexity of both the electronics and the overall installation. The objective of a pyro circuit is to dump enough energy rapidly into an electric match to cause it to catch fire. We need batteries both to power the electronics that decide when to fire the charge, and to provide the energy that actually ignites the match. The two most common battery types seen in the rocketry hobby are alkaline cells, often the nominal 9V rectangular variety, and rechargeable batteries based on Lithium Polymer (LiPo) chemistry. LiPo cells are 3.7 volts per cell nominal voltage, are very light, and have a high energy density. LiPo capacity is measured in units of current times time, so an 850mAh cell should be able to deliver 850 milliamps for an hour. The battery industry also uses something called a "C rate" to describe how fast the battery can be usefully discharged, wich is a multiplier relative to the battery capacity. So a battery with 850mAh capacity and a "2C" rating can deliver current at a sustained rate of 1700mA until discharged, while the same capacity at a "5C" rating can deliver 4250mA. By comparison, most 9V alkaline batteries are actually composed of 6 individual 1.5V cells enclosed in a wrapper. It's hard to get hard numbers for capacity and discharge rate, since in an alkaline cell the two are not independent, and the discharge rate is related to the volume of each individual cell. The data sheet for an Energizer 522 shows just over 600mAh at a 25mA discharge rate, dropping to about 300mAh at a 500mA discharge rate. Importantly for use in pyro circuits, LiPo cells have a very low source impedance, which means they can source immense amounts of current. It's not unusual for cells in the 1000mAh range to have ratings in excess of 30C! Because this rapid discharge ability can pose a risk of fire, it's common for LiPo cells to come with a "protection board" integrated into the battery assembly that is designed to limit the current to some rate such as 2C continuous duty. In large airframes, or projects that involve staging, air-starts, or other complex pyro event sequences, the most reliable approach will always be to use separate batteries for the control electronics and the source of pyro firing power. In the limit, having separate pyro batteries for each pyro charge with the control electronics only providing the switching to connect the batteries to the charges could even make sense. But for most airframes, this is overkill, and the increases in mass, volume, and wiring complexity just don't make sense. The challenge, then, is how to design electronics that will robustly initiate pyro events without negatively affecting the rest of the electronics when operating from a single shared battery. Altus Metrum Pyro Circuits The very first prototypes of TeleMetrum were designed to use a single-cell LiPo battery, and had an on-board 100mA charging circuit. Because we needed 5 volts to power the accelerometer, we had a small switching regulator that up-converted the LiPo voltage, and we used some of that regulator's output to charge a 1000uF capacitor. The pyro circuit used Fairchild FDN335N N-channel MOSFET switches in a low-side switching configuration to dump the energy stored in the capacitor through an attached ematch. Those FETs had an on resistance of under a tenth of an ohm in our operating conditions. The circuit worked very reliably, but the 1000uF electrolytic cap was huge and we struggled with the mechanics of such a large part hanging off the board... It turns out that 3.7 volts is way more than enough to fire a typical low-current e-match or equivalents like the Quest Q2G2 igniters. In fact, bench testing with a good digital oscilloscope showed that a typical e-match with resistance of 1.3-1.8 ohms will fire in approximately 13 microseconds when hit with the nominal 3.7 volts from a LiPo. So, starting with our v0.2 boards, we switched to using the LiPo battery voltage directly to fire the pyro charges, eliminating the clunky electrolytic capacitor entirely. We also switched to the Fairchild FDS9926A dual N-channel MOSFET whose specs are better in all regards for our application. The on resistance is down around 40 milli-ohms in our circuit, such that the current ratings are much higher (FET current limits are primarily driven by how much heat is generated due to current flowing through the channel's on resistance). Because using the LiPo voltage directly means we're effectively temporarily putting a very low resistance across the battery during the pyro events, the input voltage to the voltage regulator gets pulled down. To ensure the processor could "ride through" these events, we added a 100uF surface mount bulk capacitor on the 3.3 volt regulated voltage rail, which bench testing demonstrated was more than sufficient to maintain processor operation through pyro events. And that is essentially the same pyro circuit on all the boards, both TeleMetrum and TeleMini, that we have shipped to date. What's Changed The LiPo batteries we source and sell with our electronics come with a protection board and cable terminated in a 2-pin, 2mm "JST" connector. The specs on the protection board have always been "2C continuous", but we observed the ability to source much higher currents for short periods such as the 13 micro-seconds or so required to fire an e-match. Thus these protection circuits seemed just fine .. we could fire e-matches with a burst of current but were protected against short circuits in the wiring or our boards by the 2C continuous limit. Unfortunately, the most recent batch of batteries we sourced seem to have a much "twitchier" protection circuit. We can draw more than 2C for short bursts, but not as much as with prior boards, and not for as long an interval. With a 1 ohm power resistor on the pyro terminals of one of our boards, we get about 9 milliseconds of power before the protection circuit cuts in and shuts the battery down. The power stays down until all load is removed, which at least means the board is turned off and back on again, and in some cases could even mean the battery is unplugged and re-plugged since we draw trace current to keep the GPS memory alive even when the power is turned off, and at least some of the new batteries see that as enough to keep the power turned off after an over-current event. For many e-matches, this isn't an issue, since 9 milliseconds is way longer than the 13 microseconds needed to fire the charge. Unfortunately, we've discovered that many of the e-matches bought and used in the rocketry hobby are actually made for use in the fireworks industry, where it is desireable to retain continuity after firing so that series connections of e-matches all can fire even if some fire faster than others! This means that while the resistance goes up some after firing, sometimes the drain on the battery remains sufficient to cause the protection circuit to kick in even after the pyro charge has fired. What We're Doing About It If we remove the protection circuit from the LiPo (or jumper around it), all existing Altus Metrum products will operate successfully with pyro charges thave have an effective resistance of as low as 1 ohm. That's lower than any e-match or Q2G2 we've ever seen, so in effect what this means is that if you have an existing Altus Metrum flight computer, and you remove the LiPo protection circuit, you're good to go. This does not really make things any more "dangerous", since our battery chargers are all current limited and our discharge patterns will never cause heating of the battery. Frankly, in a rocket, the most likely way to cause a problem with a LiPo is by smashing or puncturing the actual battery during bench work or during a crash... and those cause the same problems with or without a protection board present. In the future, we will ship batteries that have either a much higher C rating on the protection circuit, or have no protection circuit at all. The number of problems reported by actual customers that we think should be attributed to LiPo protection circuit boards is very low, and we suspect most of our customers who are happily flying their boards can continue to do so. Ground testing where you fire pyro charges (or at least e-matches) using RF to issue the commands (not USB, since the LiPo charger is running any time USB is connected!) will confirm whether there's a problem. If the board resets (does startup beeps) after a pyro event, or shuts down completely (no LED activity), then you have a problem. If the matches light and the board keeps running, you're good to go. However, any Altus Metrum customer with LiPo batteries sourced from us or our distributors who is worried about this problem (even if you haven't seen problems in ground testing or previous flights), and who doesn't want to try soldering on the battery circuit board yourself, is welcome to contact us about removing the protection circuit for you. We won't charge anything other than shipping. To take advantage of this offer, just send email to fixmybattery@altusmetrum.org telling us how many of which capacity batteries you have that you'd like updated, and we'll respond with an RMA number and shipping details. Going Even Further As previously indicated, with the LiPo protection circuits removed, all of our current products will work reliably with at least 1 ohm across the pyro terminals. That should cover all real-world flying conditions just fine, but we're not satisfied yet. We've designed a new pyro control circuit that transfers the maximum possible energy to the load regardless of battery voltage without ever allowing the voltage to the processor to droop at all. We're testing this new circuit in various prototypes now, and if it pans out it will probably show up first in MegaMetrum and then trickle down to TeleMetrum and TeleMini as those products are updated later this year. The new pyro circuit tolerates 0 ohms (dead shorts) on the pyro terminals for as long as the battery can provide current, which is as good as it gets. We think the circuit is clever enough that we'll probably write more about it once we're finished validating it.

10 April 2013

Lars Wirzenius: Machine-parseable embedded license summary information

There is no common way to express license information in each source file at the moment. Some people embed license information in each file, others keep it in README or another file at the top of the source tree. Worse, there is no common syntax to express the license information in a machine-parseable way. If we had this, we could have tools that, for example, tell you if you're trying to merge code that has an incompatible license. Obviously, this kind of thing can never work perfectly. People keep inventing new licenses, and it is not possible for a computer program to fully understand any license. It is not clear humans can do that, either. However, it would be possible to do it to a number of well-known licenses, which would help most of the time. A classic 20/80 situation. I would like to suggest a syntax, similar to Emacs's "Hey Emacs" modelines, for embedding a summary of the license, or licenses, for a source file, in such a way that it can be programmatically extracted and parsed and analysed. With this syntax, one could then write a tool to ensure that all files in a project have the same license, or that all licenses are compatible with the project's overall license. Such a tool will, of course, rely on heuristics and assumptions. For example, it needs to assume the machine-parseable license summary is correct, and rely on a ruleset on what licenses are compatible with what licenses. Things can go wrong. That's life. Remember, this is aiming at doing the 20% of the work that will work 80% of the time, not perfection. I don't have a tool written, but I have a suggestion for the syntax.

/*
 * Copyright 2013 Lars Wirzenius
 *
 * tl;dr =*= Licenses: GPL-3+ or Expat, and Artistic =*=
 *
 * Blah. Blah. Blah. Imagine long, boring license texts 
 * here.
 */

The important part is this:

=*= Licenses: GPL-3+ or Expat, and Artistic =*=

The =*= prefix and suffix and the word Licenses are there to make grepping reasonably reliable without too many false positives, and to allow comment characters and other text on the same line. The actual license summary follows the syntax and semantics of the Debian copyright-format 1.0 specification, which I chose because it exists and has had a fair bit of review so far, and is reasonably expressive. The license summaries can be extracted with the following GNU sed invocation:

sed -n '/.*=\*= [Ll]icen[cs]es\?: \(.*\)=\*=.*/s//\1/p'

I allowed various forms of the word license, since it's a word that a lot of people will get wrong, and it's easy to catch all four common forms. So, does anyone else think this might be useful? Would you use it in your own projects?

Next.

Previous.

Base subrack
Item	Supplier	Quantity	Line Price
Schroff 24563-194 subrack kit	Farnell	1	41.28
Schroff 24560-351 guide rails	Farnell	3	13.65
Schroff rack rear horizontal rail	Shapeways	2	100.00
1000mm length of 4mm threaded rod	B and Q	1	1.48
170mm x 431mm x 3mm Aluminium sheet	inspired steel	2	40.00
PSU mounting plate	Shapeways	1	35.42
PCB standoff	Shapeways	4	22.30
160mm Deep Modular PC supply	CCL	1	55.76
TP-Link TL-SF1016DS 16-Port 10/100Mbps-Switch	CCL	1	23.77
8 Channel 16A Relay Board Controlled Via Ethernet	Rapid	2	126.00
Raspberry Pi	Farnell	1	26.48
USB Serial converters	CCL	10	37.40
10 port strip style USB HUB	Ebay	1	7.00
Parts for custom Ethernet cables	RS	13	26.00
Parts for custom molex power cables (salvaged from scrap ATX PSU)	Workshop	11	11.00
33R 10W wirewound resistor for dummy load	RS	1	1.26
24pin ATX female connector pre-wired	Maplin	1	2.99
Akasa double sided thermal pad	Maplin	1	5.00
Small cable tie bases	Maplin	1	6.49
Miscellaneous cable, connectors, nylon standoffs, solder, heatshrink, zip ties, nuts, washers etc.	Workshop	1	20.00

Total for subrack			603.28