• source tarballs
• Win32 binaries
• DOS binaries
• drop-in jupprc for JOE 2.8, or to update jupp 2.8
• drop-in jupprc for JOE 3.7
• drop-in jupprc for JOE 4.4

1. Building a bootable disk image that can be used as both a live system and as an OS installer.
2. Running on that live system, to install the target system. Which can just involve copying the live system to the target disk and then letting the configuration management system make the necessary changes to get from the live system configuration to the target system configuration.
3. To support such things as headless arm boards, building customized images tuned for the target board and use case, that can then simply be copied to the board to install.
4. Optionally, running on the installed system later, to futher customize it. Starting from the same configuration that produced the installed system in the first place.

• A user interface that runs on the live system and gets whatever input is needed to install to the target system. This is really just a config editor underneath. I built a prototype gamified interface that's as minimal as such an interface could get.
• With a regular text editor, of course. This is the equivilant of preseeding in d-i, giving advanced users full control over the system that gets built. Unlike with preseeding, users have the full power of a configuration management system, so can specify precisely the system they want installed.
• A separate user interface for customizing disk images, for arm boards and similar use cases. This would run on a server, or on the user's own laptop.

• In Functional Reactive Propellor I found a way to express the commonalities and differences between the installer's configuration and the target system's configuration. This lets the installer disk image be copied to the target and the minimum work be done to convert it into the desired target system.
• In disk partitioning nitty gritty I tackled configuring the partition table of the target system. I extended a DSL propellor already used for partitioning disk images. The result is basically partman in 1/100th the lines of code.
• In end in sight I found a way to make propellor build disk images super fast, so a new 5 gb disk image can be ready in 30 seconds, a quarter of the time that it takes to write a 5 gb file with dd.
• For completeness sake, I also devblogged about unfortunately needing a progress bar, picking the disk to install to and installing grub, dependency yak shaving, high bandwidth propellor hacking and finishing touches

From: Sebastian R. < @gmail.com>
To: joachim@cis.upenn.edu
Subject: re: Errors I've spotted a basic error in your course on Haskell (https://www.seas.upenn.edu/~cis194/fall16/). Before I proceed, it's cool if you're not receptive to errors being indicated; I've come across a number of professors who would rather take offense than admit we're all human and thus capable of making mistakes... My goal is to find a resource that might be useful well into the future, and a good indicator of that is how responsive the author is to change. In your introduction note you have written:

n contrast to a classical intro into Haskell, we do not start with numbers, booleans, tuples, lists and strings, but we start with pictures. These are of course library-defined (hence the input CodeWorld) and not part of the language . But that does not make them less interesting, and in fact, even the basic boolean type is library defined it just happens to be the standard library.

Howeverm there is no input CodeWorld in the code above. Have you been made aware of this error earlier? Regards, ...

From: Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript>
To: Sebastian R. < @gmail.com>
Subject: Re: Errors Dear Sebastian, thanks for pointing out errors. But the first piece of code under Basic Haskell starts with

 -# LANGUAGE OverloadedStrings #- 
import CodeWorld

so I am not sure what you are referring to. Note that these are lecture notes, so you have to imagine a lecturer editing code live on stage along with it. If you only have the notes, you might have to infer a few things. Regards, Joachim

From: Sebastian R. < @gmail.com>
To: Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript>
Subject: Re: Errors Greetings, Joachim. Kindly open the lecture slides and search for "input CodeWorld" to find the error; it is not in the code, but in the paragraph that implicitly refers back to the code. You might note that I quoted this precisely from the lectures... and so I repeat myself... this came from your lectures; they're not my words!

In contrast to a classical intro into Haskell, we do not start with numbers, booleans, tuples, lists and strings, but we start with pictures. These are of course library-defined (hence the input CodeWorld) and not part of the language . But that does not make them less interesting, and in fact, even the basic boolean type is library defined it just happens to be the standard library.

This time around, I've highlighted the issue. I hope that made it easier for you to spot... Nonetheless, I got my answer. Don't reply if you're going to fight tooth and nail about such a basic fix; it's simply a waste of both of our time. I'd rather learn from somewhere else... On Tue, Aug 1, 2017 at 11:19 PM, Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript> wrote:

From: Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript>
To: Sebastian R. < @gmail.com>
Betreff: Re: Errors Dear Sebastian, nobody is fighting, and I see the mistake now: The problem is not that the line is not in the code, the problem is that there is a typo in the line and I wrote input instead of import . Thanks for the report, although you did turn it into quite a riddle a simple you wrote import when it should have been import would have been a better user of both our time. Regards, Joachim Am Donnerstag, den 03.08.2017, 13:32 +1000 schrieb Sebastian R.:

From: Sebastian R. < @gmail.com>
To: Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript>
Subject: Re: Errors

a simple you wrote import when it should have been import would have been a better user of both our time.

We're both programmers. How about I cut ALL of the unnecessary garbage and just tell you to s/import/input/ on that last quotation (the thing immediately before this paragraph, in case you didn't know). I blatantly quoted the error, like this:

In your introduction note you have written:

n contrast to a classical intro into Haskell, we do not start with numbers, booleans, tuples, lists and strings, but we start with pictures. These are of course library-defined (hence the input CodeWorld) and not part of the language . But that does not make them less interesting, and in fact, even the basic boolean type is library defined it just happens to be the standard library.

Howeverm there is no input CodeWorld in the code above.

Since that apparently wasn't clear enough, in my second email to you I had to highlight it like so:

You might note that I quoted this precisely from the lectures... and so I repeat myself... this came from your lectures; they're not my words!

In contrast to a classical intro into Haskell, we do not start with numbers, booleans, tuples, lists and strings, but we start with pictures. These are of course library-defined (hence the input CodeWorld) and not part of the language . But that does not make them less interesting, and in fact, even the basic boolean type is library defined it just happens to be the standard library.

This time around, I've highlighted the issue. I hope that made it easier for you to spot...

I'm not sure if you're memeing at me or not now, but it seems either your reading comprehension, or your logical deduction skills might be substandard. Unfortunately, there isn't much either of us can do about that, so I'm happy to accept that some people will be so stupid; after all, it's to be expected and if we don't accept that which is to be expected then we live our lives in denial. Happy to wrap up this discusson here, Seb... On Fri, Aug 4, 2017 at 12:22 AM, Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript> wrote:

Note, I'll again be talking mostly about USB2.0 low- and full-speed. I believe that high speed is approximately the same but with faster wires, except not quite that simple.

Down to the wire I don't intend to talk about the actual electrical signalling, though it's not un-reasonable for you to know that USB is a pair of wires forming a differentially signalled bidirectional serial communications link. The host is responsible for managing all the framing and timing on the link, and for formatting the communications into packets. There are a number of packet types which can appear on the USB link:

Packet type	Purpose
Token Packet	When the host wishes to send a message to the Control endpoint to configure the device, read data IN, or write data OUT, it uses this to start the transaction.
Data(0/1) Packet	Following a Setup, In, or Out token, a Data packet is a transfer of data (in either direction). The 0 and 1 alternate to provide a measure of confidence against lost packets.
Handshake Packet	Following a data packet of some kind, the other end may ACK the packet (all was well), NAK the packet (report that the device cannot, temporarily, send/receive data, or that an interrupt endpoint isn't triggered), or STALL the bus in which case the host needs to intervene.
Start of Frame	Every 1ms (full-speed) the host will send a SOF packet which carries a frame number. This can be used to help keep time on very simple devices. It also divides the bus into frames within which bandwidth is allocated.

As an example, when the host wishes to perform a control transfer, the following packets are transacted in turn:

1. Setup Token - The host addresses the device and endpoint (OUT0)
2. Data0 Packet - The host transmits a GET_DESCRIPTOR for the device descriptor
3. Ack Packet - The device acknowledges receipt of the request

This marks the end of the first transaction. The device decodes the GET_DESCRIPTOR request and prepares the device descriptor for transmission. The transmission occurs as the next transaction on the bus. In this example, we're assuming 8 byte maximum transmission sizes, for illustrative purposes.

1. In Token - The host addresses the device and endpoint (IN0)
2. Data1 Packet - The device transmits the first 8 bytes of the descriptor
3. Ack Packet - The host acknowledges the data packet
4. In Token - The host addresses the device and endpoint (IN0)
5. Data0 Packet - The device transmits the remaining 4 bytes of the descriptor (padded)
6. Ack Packet - The host acknowledges the data packet

The second transaction is now complete, and the host has all the data it needs to proceed. Finally a status transaction occurs in which:

1. Out Token - The host addresses the device and endpoint (OUT0)
2. Data1 Packet - The host transmits a 0 byte data packet to indicate successful completion
3. Ack Packet - The device acknowledges the completion, indicating its own satisfaction

And thus ends the full control transaction in which the host retrieves the device descriptor. From a high level, we need only consider the activity which occurs at the point of the acknowledgement packets. In the above example:

1. On the first ACK the device prepares IN0 to transmit the descriptor, readying whatever low level device stack there is with a pointer to the descriptor and its length in bytes.
2. On the second ACK the low levels are still thinking.
3. On the third ACK the transmission from IN0 is complete and the endpoint no longer expects to transfer data.
4. On the fourth ACK the control transaction is entirely complete.

Thinking at the low levels of the control interface Before we can build a high level USB stack, we need to consider the activity which might occur at the lower levels. At the low levels, particularly of the device control interface, work has to be done at each and every packet. The hardware likely deals with the token packet for us, leaving the data packets for us to process, and the resultant handshake packets will be likely handled by the hardware in response to our processing the data packets. Since every control transaction is initiated by a setup token, let's look at the setup requests which can come our way...

Setup Packet (Data) Format

Field Name	Byte start	Byte length	Encoding	Meaning
bmRequestType	0	1	Bitmap	Describes the kind of request, and the target of it. See below.
bRequest	1	1	Code	The request code itself, meanings of the rest of the fields vary by bRequest
wValue	2	2	Number	A 16 bit value whose meaning varies by request type
wIndex	4	2	Number	A 16 bit value whose meaning varies by request type but typically encodes an interface number or endpoint.
wLength	6	2	Number	A 16 bit value indicating the length of the transfer to come.

Since bRequest is essentially a switch against which multiple kinds of setup packet are selected between, here's the meanings of a few...

GET_DESCRIPTOR (Device) setup packet

Field Name	Value	Meaning
bmRequestType	0x08	Data direction is IN (from device to host), recipient is the device
bRequest	0x06	GET_DESCRIPTOR (in this instance, the device descriptor is requested)
wValue	0x0001	This means the device descriptor
wIndex	0x0000	Irrelevant, there's only 1 device descriptor anyway
wLength	12	This is the length of a device descriptor (12 bytes)

SET_ADDRESS to set a device's USB address

Field Name	Value	Meaning
bmRequestType	0x00	Data direction is OUT (from host to device), recipient is the device
bRequest	0x05	SET_ADDRESS (Set the device's USB address)
wValue	0x00nn	The address for the device to adopt (max 127)
wIndex	0x0000	Irrelevant for address setting
wLength	0	There's no data transfer expected for this setup operation

Most hardware blocks will implement an interrupt at the point that the Data packet following the Setup packet has been receive. This is typically called receiving a 'Setup' packet and then it's up to the device stack low levels to determine what to do and dispatch a handler. Otherwise an interrupt will fire for the IN or OUT tokens and if the endpoint is zero, the low level stack will handle it once more. One final thing worth noting about SET_ADDRESS is that it doesn't take effect until the completion of the zero-length "status" transaction following the setup transaction. As such, the status request from the host will still be sent to address zero (the default for new devices).

A very basic early "packet trace" This is an example, and is not guaranteed to be the packet sequence in all cases. It's a good indication of the relative complexity involved in getting a fresh USB device onto the bus though... When a device first attaches to the bus, the bus is in RESET state and so the first event a device sees is a RESET which causes it to set its address to zero, clear any endpoints, clear the configuration, and become ready for control transfers. Shortly after this, the device will become suspended. Next, the host kicks in and sends a port reset of around 30ms. After this, the host is ready to interrogate the device. The host sends a GET_DESCRIPTOR to the device, whose address at this point is zero. Using the information it receives from this, it can set up the host-side memory buffers since the device descriptor contains the maximum transfer size which the device supports. The host is now ready to actually 'address' the device, and so it sends another reset to the device, again around 30ms in length. The host sends a SET_ADDRESS control request to the device, telling it that its new address is nn. Once the acknowledgement has been sent from the host for the zero-data status update from the device, the device sets its internal address to the value supplied in the request. From now on, the device shall respond only to requests to nn rather than to zero. At this point, the host will begin interrogating further descriptors, looking at the configuration descriptors and the strings, to build its host-side representation of the device. These will be GET_DESCRIPTOR and GET_STRING_DESCRIPTOR requests and may continue for some time. Once the host has satisfied itself that it knows everything it needs to about the device, it will issue a SET_CONFIGURATION request which basically starts everything up in the device. Once the configuration is set, interrupt endpoints will be polled, bulk traffic will be transferred, Isochronous streams begin to run, etc.

Okay, but how do we make this concrete? So far, everything we've spoken about has been fairly abstract, or at least "soft". But to transfer data over USB does require some hardware. (Okay, okay, we could do it all virtualised, but there's no fun in that). The hardware I'm going to be using for the duration of this series is the STM32 on the blue-pill development board. This is a very simple development board which does (in theory at least) support USB device mode. If we view the schematic for the blue-pill, we can see a very "lightweight" USB interface which has a pullup resistor for D+. This is the way that a device signals to the host that it is present, and that it wants to speak at full-speed. If the pullup were on D- then it would be a low-speed device. High speed devices need a little more complexity which I'm not going to go into for today. The USB lines connect to pins PA11 and PA12 which are the USB pins on the STM32 on the board. Since USB is quite finicky, the STM32 doesn't let you remap that function elsewhere, so this is all looking quite good for us so far. The specific STM32 on the blue-pill is the STM32F103C8T6. By viewing its product page on ST's website we can find the reference manual for the part. Jumping to section 23 we learn that this STM32 supports full-speed USB2.0 which is convenient given the past article and a half. We also learn it supports up to eight endpoints active at any one time, and offers double-buffering for our bulk and isochronous transfers. It has some internal memory for packet buffering, so it won't use our RAM bandwidth while performing transfers, which is lovely. I'm not going to distill the rest of that section here, because there's a large amount of data which explains how the USB macrocell operates. However useful things to note are:

• How IN OUT and SETUP transfers work.
• How the endpoint buffer memory is configured.
• That all bus-powered devices MUST respond to suspend/resume properly
• That the hardware will prioritise endpoint interrupts for us so that we only need deal with the most pressing item at any given time.
• There is an 'Enable Function' bit in the address register which must be set or we won't see any transactions at all.
• How the endpoint registers signal events to the device firmware.

Next time, we're going to begin the process of writing a very hacky setup routine to try and initialise the USB device macrocell so that we can see incoming transactions through the ITM. It should be quite exciting, but given how complex this will be for me to learn, it might be a little while before it comes through.

A brief introduction to USB In order to go much further, I needed a grounding in USB. It's a multi-layer protocol as you might expect, though we can probably ignore the actual electrical layer since any device we might hope to support will have to have a hardware block to deal with that. We will however need to consider the packet layer (since that will inform how the hardware block is implemented and thus its interface) and then the higher level protocols on top. USB is a deliberately asymmetric protocol. Devices are meant to be significantly easier to implement, both in terms of hardware and software, as compared with hosts. As such, despite some STM32s having OTG ports, I have no intention of supporting host mode at this time. USB is arranged into a set of busses which are, at least in the USB1.1 case, broadcast domains. As such, each device has an address assigned to it by the host during an early phase called 'configuration'. Once the address is assigned, the device is expected to only ever respond to messages addressed to it. Note that since everything is asymmetric in USB, the device can't send messages on its own, but has to be asked for them by the host, and as such the addressing is always from host toward device. USB devices then expose a number of endpoints through which communication can flow IN to the host or OUT to the device. Endpoints are not bidirectional, but the in and out endpoints do overlap in numbering. There is a special pair of endpoints, IN0 and OUT0 which, between them, form what I will call the device control endpoints. The device control endpoints are important since every USB device MUST implement them, and there are a number of well defined messages which pass over them to control the USB device. In theory a bare minimum USB device would implement only the device control endpoints.

Configurations, and Classes, and Interfaces, Oh My! In order for the host to understand what the USB device is, and what it is capable of, part of the device control endpoints' responsibility is to provide a set of descriptors which describe the device. These descriptors form a heirarchy and are then glommed together into a big lump of data which the host can download from the device in order to decide what it is and how to use it. Because of various historical reasons, where a multi-byte value is used, they are defined to be little-endian, though there are some BCD fields. Descriptors always start with a length byte and a type byte because that way the host can parse/skip as necessary, with ease. The first descriptor is the device descriptor, is a big one, and looks like this:

Device Descriptor

Field Name	Byte start	Byte length	Encoding	Meaning
bLength	0	1	Number	Size of the descriptor in bytes (18)
bDescriptorType	1	1	Constant	Device Descriptor (0x01)
bcdUSB	2	2	BCD	USB spec version compiled with
bDeviceClass	4	1	Class	Code, assigned by USB org (0 means "Look at interface descriptors", common value is 2 for CDC)
bDeviceSubClass	5	1	SubClass	Code, assigned by USB org (usually 0)
bDeviceProtocol	6	1	Protocol	Code, assigned by USB org (usually 0)
bMaxPacketSize	7	1	Number	Max packet size for IN0/OUT0 (Valid are 8, 16, 32, 64)
idVendor	8	2	ID	16bit Vendor ID (Assigned by USB org)
idProduct	10	2	ID	16bit Product ID (Assigned by manufacturer)
bcdDevice	12	2	BCD	Device version number (same encoding as bcdUSB)
iManufacturer	14	1	Index	String index of manufacturer name (0 if unavailable)
iProduct	15	1	Index	String index of product name (0 if unavailable)
iSerialNumber	16	1	Index	String index of device serial number (0 if unavailable)
bNumConfigurations	17	1	Number	Count of configurations the device has.

This looks quite complex, but breaks down into a relatively simple two halves. The first eight bytes carries everything necessary for the host to be able to configure itself and the device control endpoints properly in order to communicate effectively. Since eight bytes is the bare minimum a device must be able to transmit in one go, the host can guarantee to get those, and they tell it what kind of device it is, what USB protocol it supports, and what the maximum transfer size is for its device control endpoints. The encoding of the bcdUSB and bcdDevice fields is interesting too. It is of the form 0xMMmm where MM is the major number, mm the minor. So USB2.0 is encoded as 0x0200, USB1.1 as 0x0110 etc. If the device version is 17.36 then that'd be 0x1736. Other fields of note are bDeviceClass which can be 0 meaning that interfaces will specify their classes, and idVendor/idProduct which between them form the primary way for the specific USB device to be identified. The Index fields are indices into a string table which we'll look at later. For now it's enough to know that wherever a string index is needed, 0 can be provided to mean "no string here". The last field is bNumConfigurations and this indicates the number of ways in which this device might function. A USB device can provide any number of these configurations, though typically only one is provided. If the host wishes to switch between configurations then it will have to effectively entirely quiesce and reset the device. The next kind of descriptor is the configuration descriptor. This one is much shorter, but starts with the same two fields:

Configuration Descriptor

Field Name	Byte start	Byte length	Encoding	Meaning
bLength	0	1	Number	Size of the descriptor in bytes (9)
bDescriptorType	1	1	Constant	Configuration Descriptor (0x02)
wTotalLength	2	2	Number	Size of the configuration in bytes, in total
bNumInterfaces	4	1	Number	The number of interfaces in this configuration
bConfigurationValue	5	1	Number	The value to use to select this configuration
iConfiguration	6	1	Index	The name of this configuration (0 for unavailable)
bmAttributes	7	1	Bitmap	Attributes field (see below)
bMaxPower	8	1	Number	Maximum bus power this configuration will draw (in 2mA increments)

An important field to consider here is the bmAttributes field which tells the host some useful information. Bit 7 must be set, bit 6 is set if the device would be self-powered in this configuration, bit 5 indicates that the device would like to be able to wake the host from sleep mode, and bits 4 to 0 must be unset. The bMaxPower field is interesting because it encodes the power draw of the device (when set to this configuration). USB allows for up to 100mA of draw per device when it isn't yet configured, and up to 500mA when configured. The value may be used to decide if it's sensible to configure a device if the host is in a low power situation. Typically this field will be set to 50 to indicate the nominal 100mA is fine, or 250 to request the full 500mA. Finally, the wTotalLength field is interesting because it tells the host the total length of this configuration, including all the interface and endpoint descriptors which make it up. With this field, the host can allocate enough RAM to fetch the entire configuration descriptor block at once, simplifying matters dramatically for it. Each configuration has one or more interfaces. The interfaces group some endpoints together into a logical function. For example a configuration for a multifunction scanner/fax/printer might have an interface for the scanner function, one for the fax, and one for the printer. Endpoints are not shared among interfaces, so when building this table, be careful. Next, logically, come the interface descriptors:

Interface Descriptor

Field Name	Byte start	Byte length	Encoding	Meaning
bLength	0	1	Number	Size of the descriptor in bytes (9)
bDescriptorType	1	1	Constant	Interface Descriptor (0x04)
bInterfaceNumber	2	1	Number	The number of the interface
bAlternateSetting	3	1	Number	The interface alternate index
bNumEndpoints	4	1	Number	The number of endpoints in this interface
bInterfaceClass	5	1	Class	The interface class (USB Org defined)
bInterfaceSubClass	6	1	SubClass	The interface subclass (USB Org defined)
bInterfaceProtocol	7	1	Protocol	The interface protocol (USB Org defined)
iInterface	8	1	Index	The name of the interface (or 0 if not provided)

The important values here are the class/subclass/protocol fields which provide a lot of information to the host about what the interface is. If the class is a USB Org defined one (e.g. 0x02 for Communications Device Class) then the host may already have drivers designed to work with the interface meaning that the device manufacturer doesn't have to provide host drivers. The bInterfaceNumber is used by the host to indicate this interface when sending messages, and the bAlternateSetting is a way to vary interfaces. Two interfaces with the came bInterfaceNumber but different bAlternateSettings can be switched between (like configurations, but) without resetting the device. Hopefully the rest of this descriptor is self-evident by now. The next descriptor kind is endpoint descriptors:

Endpoint Descriptor

Field Name	Byte start	Byte length	Encoding	Meaning
bLength	0	1	Number	Size of the descriptor in bytes (7)
bDescriptorType	1	1	Constant	Endpoint Descriptor (0x05)
bEndpointAddress	2	1	Endpoint	Endpoint address (see below)
bmAttributes	3	1	Bitmap	Endpoint attributes (see below)
wMaxPacketSize	4	2	Number	Maximum packet size this endpoint can send/receive
bInterval	6	1	Number	Interval for polling endpoint (in frames)

The bEndpointAddress is a 4 bit endpoint number (so there're 16 endpoint indices) and a bit to indicate IN vs. OUT. Bit 7 is the direction marker and bits 3 to 0 are the endpoint number. This means there are 32 endpoints in total, 16 in each direction, 2 of which are reserved (IN0 and OUT0) giving 30 endpoints available for interfaces to use in any given configuration. The bmAttributes bitmap covers the transfer type of the endpoint (more below), and the bInterval is an interval measured in frames (1ms for low or full speed, 125 s in high speed). bInterval is only valid for some endpoint types. The final descriptor kind is for the strings which we've seen indices for throughout the above. String descriptors have two forms:

String Descriptor (index zero)

Field Name	Byte start	Byte length	Encoding	Meaning
bLength	0	1	Number	Size of the descriptor in bytes (variable)
bDescriptorType	1	1	Constant	String Descriptor (0x03)
wLangID[0]	2	2	Number	Language code zero (e.g. 0x0409 for en_US)
wLangID[n]	4..	2	Number	Language code n ...

This form (for descriptor 0) is that of a series of language IDs supported by the device. The device may support any number of languages. When the host requests a string descriptor, it will supply both the index of the string and also the language id it desires (from the list available in string descriptor zero). The host can tell how many language IDs are available simply by dividing bLength by 2 and subtracting 1 for the two header bytes. And for string descriptors of an index greater than zero:

String Descriptor (index greater than zero)

Field Name	Byte start	Byte length	Encoding	Meaning
bLength	0	1	Number	Size of the descriptor in bytes (variable)
bDescriptorType	1	1	Constant	String Descriptor (0x03)
bString	2..	..	Unicode	The string, in "unicode" format

This second form of the string descriptor is simply the the string is in what the USB spec calls 'Unicode' format which is, as of 2005, defined to be UTF16-LE without a BOM or terminator. Since string descriptors are of a variable length, the host must request strings in two transactions. First a request for 2 bytes is sent, retrieving the bLength and bDescriptorType fields which can be checked and memory allocated. Then a request for bLength bytes can be sent to retrieve the entire string descriptor.

Putting that all together Phew, this is getting to be quite a long posting, so I'm going to leave this here and in my next post I'll talk about how the host and device pass packets to get all that information to the host, and how it gets used.

• I backported freeciv, freeorion and minetest to stretch-backports.
• The bug fix (#866378) for 3dchess also landed in Stretch and Jessie.
• I sponsored Lugaru for Vincent Prat and Martin Erik Werner, a really cool 3D fighting game featuring a rabbit. The game is dfsg-free now and will replace openlugaru.
• I uploaded fifechan to unstable and packaged new upstream versions of fife, unknown-horizons, adonthell-data and hyperrogue.
• I fixed bugs in bloboats (#864534), lordsawar (RC #866988), kraptor (#826423), pathogen (#845991), fretsonfire (#866426), blockout2 (#826416), boswars (#827112), kanatest (RC #868315, fix also backported to Stretch), overgod (#827114), morris (#829948, #721834, #862224), mousetrap (#726842), alsoft-conf (#784052, #562898) and nikwi (#835625)
• I uploaded a new revision of clanlib and teg fixing Perl transition bugs. The patches were provided by gregor herrmann. I added myself to Uploaders in case of teg because the package was missing a human maintainer.
• I adopted trackballs after I discovered #868983 where Henrique de Moraes Holschuh called attention to a new fork of Trackballs. The current version was broken and unplayable and it was only a matter of time before the game was removed from Debian. I could fix a couple of bugs, forwarded some issues upstream and I believe a nice game was saved.
• I uploaded Bullet 2.86.1 to unstable and completed another Bullet transition.

• Almost all PDFsam dependencies were accepted into the archive this month. I had to update some of them and removed patches that were needed for the initial bootstrapping. I also uploaded libmetadata-extractor-java to unstable. Now only fontawesomefx is missing to complete the PDFsam update.
• I cleaned up vecmath.
• I prepared a security update for undertow issued as DSA 3906-1.
• New upstream releases: jansi, jansi-native, libpdfbox-java, qdox2.
• Bug fixes: felix-bundlerepository (RC #868074), netbeans (RC #868931, #865715), libnb-platform18-java (RC #868923), easymock (RC #869018).

• From 24. July until 31. July I was in charge of our LTS frontdesk. I triaged bugs in tinyproxy, varnish, freerdp, ghostscript, gcc-4.6, gcc-4.7, fontforge, teamspeak-server, teamspeak-client, qpdf, nvidia-graphics-drivers and sipcrack. I also pinged Diego Biurrun for more information about the next libav update and replied to questions on the debian-lts mailing list and LTS IRC channel.
• DLA 1034-1. Issued a security update for php5 fixing 5 CVE. I discussed CVE-2017-11362 with the security team. We came to the conclusion that it was no security issue but just a normal bug.
• DLA 1036-1. Issued a security update for gsoap fixing 1 CVE.
• DLA 1037-1. Issued a security update for catdoc fixing 1 CVE.
• DLA 613-2. Issued a regression update for roundcube.
• DLA 1045-1. Issued a security update for graphicsmagick fixing 10 CVE.
• DLA 1047-1. Issued a security update for supervisor fixing 1 CVE.
• DLA-1048-1. Issued a security update for ghostscript fixing 8 CVE.

• I uploaded the security fix for spice to unstable which was already fixed in Stretch and earlier versions.

Changes

• apt: more flexible timing (1, 2, 3), /org to /srv
• apt-offline: remove generated files
• check-all-the-things: add more shell checks
• spelling dictionaries:
  • codespell: bettery, ciruit(s), debugginf, treshold, actuion(able), apendix, behaivior, confimred, meaningfull, registeres, sence, systen, tryed, infrasctructure, current(ly), defaut, pedning, variabele, upstreamedd, nework, ouput
  • lintian: resistence, bettery, ciruit(s), debugginf, actuion, actuionable, apendix, behaivior, confimred, finaly, meaningfull, registeres, sence, systen, tryed, infrasctructure, currnt(ly), defaut, pedning, variabele, upstreamedd
• Debian security tracker: add gsoap CVE-2017-9765
• Debian puppet: stretch fixes (1, 2), failed attempt to kill a mail flood, disable web cache backups
• Debian QA services: email update, https for lintian.d.o, fix issue with by-hash
• Debian build log scanner: https, html log links, ignore uploaders brokenness
• Debian Planets: https for Grml, add Zevenet, Deepin, remove HandyLinux
• Debian website: use ISO dates, no https for archive.d.o
• Debian package uploads: flasm 1.62-8
• Debian wiki pages: DebianDay/2017/Venezuela/Caracas, Derivatives/Census (Template, Deepin, Grml, HandyLinux, Proxmox, SELKS, xanadu, Zevenet), Hardware/Wanted, Infinity0, IRC, JavierFernandezSanguino, LocalGroups/Debian-CN, LTS/Team, MemberBenefits, Mobile, PaulWise/InterestingSoftware, qa.debian.org, ReleasePartyStretch/India/Hyderabad/MGIT, SystemBuildTools
• DebConf wiki pages: DebConf17/DebconfShirts

Issues

• Crashes in MoinMoin, aspell
• Missing sources in apt-offline (reported privately)
• Malformed data in ftp.d.o
• Debug symbols in perl
• Media playback in gstreamer
• Race condition in dehydrated
• Weird errors in translate-shell

Review

• Spam: reported several Debian bug reports and 359 Debian mailing list posts
• Debian wiki: RecentChanges for the month
• Debian screenshots:
  • approved baresip, blender, caja-dropbox, cups, fastnetmon, firetools, fonts-firacode, freecad, gemrb-baldurs-gate-2, gjiten, gscan2pdf, kdevelop, ketm, liguidsoap, lordsawar, mdadm, minizinc-ide, mupdf, musique, nautilus-dropbox, pcsxr, projectl, quiterss, rafkill, redis-server, redshift-gtk, residualvm, sigil, sqlitebrowser, surf, trigger-rally, virtualbox, vis, vizigrep, vokoscreen, xinv3d, yarssr
  • approved deletion of phototonic (Windows screenshot), pipebench (inappropriate background), autossh/gtk3-nocsd (not the package)
  • rejected deletion of muon-discover (we do not remove old screenshots), apt (not a bug reporting site)

Administration

• Debian: fsck/reboot a buildd, reboot a segfaulting buildd, report/fix broken hoster contact, ping hoster about down machines, forcibly reset backup machine, merged cache patch for network-test.d.o, do some samhain dances, fix two stunnel services, update an IP address in LDAP, fix /etc/aliases on one host, reboot 1 non-responsive VM
• Debian mentors: security updates, reboot
• Debian wiki: whitelist several email addresses
• Debian build log scanner: deploy my changes
• Debian PTS: deploy my changes
• Openmoko: security updates & reboots

Communication

• Ping Advogato users on Planet Debian about updating/removing their feeds since it shut down
• Invite deepin to the Debian derivatives census
• Welcome Deepin to the Debian derivatives census
• Inquire about the status of GreenboneOS, HandyLinux

Sponsors All work was done on a volunteer basis.

• [DLA 1025-1] bind9 security update for two CVEs
• [DLA 1038-1] libtasn1-3 security update for one CVE
• [DLA 1025-2] bind9 regression update
• [DLA 1039-1] rkhunter security update for one CVE
• [DLA 1040-1] resiprocate security update for one CVE
• [DLA 1041-1] nasm security update for two CVEs
• [DLA 1042-1] libquicktime security update for seven CVEs

• [1] https://etbe.coker.com.au/2015/01/28/play-machine-tor/
• [2] https://www.eff.org/torchallenge/what-is-tor.html
• [3] https://www.torproject.org/docs/tor-doc-relay.html.en

• It will remove the need for littering your rules file for supporting noudeb (and in some cases also other noX profiles).
• It will remove the need for overriding the dh_install* tools just to juggle with sourcedir and -p/-N.

• foo
• libfoo1
• libfoo-dev
• foo-udeb
• libfoo1-udeb

[...]
override_dh_auto_configure:
    dh_auto_configure -B build-deb -- --with-feature1 --with-feature2
    dh_auto_configure -B build-udeb -- --without-feature1 --without-feature2
[...]

1. Pick a reasonable label name for the secondary build. In the example, I will use udeb .
2. Add buildlabel=$LABEL to all dh_auto_* calls related to your secondary build.
3. Tag all packages related to my-label with X-DH-Buildlabel: $LABEL in debian/control. (For udeb packages, you may want to add Build-Profiles: <!noudeb> while you are at it).

[...]
override_dh_auto_configure:
    dh_auto_configure -B build-deb -- --with-feature1 --with-feature2
    dh_auto_configure --buildlabel=udeb -B build-udeb -- --without-feature1 --without-feature2
[...]

• debhelper will skip the calls to dh_auto_* with buildlabel=udeb if the udeb packages are skipped.
• dh_auto_install will automatically pick a separate destination directory by default for the udeb build (assuming you do not explicitly override it with destdir).
• dh_install will now automatically pick up files from the destination directory.that dh_auto_install used for the given package (even if you overwrote it with destdir). Note that you have to remove any use of sourcedir first as this disables the auto-detection. This also works for other dh_install* tools supporting sourcedir in compat 11 or later.

• Apply labels to packages and dh_auto_* tools: https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/?h=wip-dh-prototype-smarter-multi-builds&id=880a80bc392a298bed033c047ccbef65e50acc58
• Clean up most make conditionals: https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/?h=wip-dh-prototype-smarter-multi-builds&id=96c0ab8b35ce04eab031a4a9ee4c4ef686f76977
• Merge multiple dh_install calls into one: https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/?h=wip-dh-prototype-smarter-multi-builds&id=67994cb5e5b451ebcebb0dccfdd97e606ac4a172
  ( fail-missing was never applied to the udeb before; I kept it that way by passing sourcedir to dh_missing because I was to lazy to review the missing files).

• It is experimental feature and may change without notice.
• The current prototype may break existing packages as it is not guarded by a compat bump to ease your testing. I am still very curious to hear about any issues you may experience.
• The default build directory is re-used even with different buildlabels, so you still need to use explicit build dirs for buildsystems that prefer building in a separate directory (e.g. meson).
• udebs are not automatically tagged with an udeb buildlabel. This is partly by design as some source packages only build udebs (and no regular debs). If they were automatically tagged, the existing packages would break.
• Label names are used in path names, so you may want to refrain from using too exciting label names.
• It is experimental feature and may change without notice. (Yes, I thought it deserved repeating)

Debian Long Term Support (LTS) This is my monthly working on Debian LTS. This time I worked on various hairy issues surrounding ca-certificates, unattended-upgrades, apache2 regressions, libmtp, tcpdump and ipsec-tools.

ca-certificates updates I've been working on the removal of the Wosign and StartCom certificates (Debian bug #858539) and, in general, the synchronisation of ca-certificates across suites (Debian bug #867461) since at least last march. I have made an attempt at summarizing the issue which led to a productive discussion and it seems that, in the end, the maintainer will take care of synchronizing information across suites. Guido was right in again raising the question of synchronizing NSS across all suites (Debian bug #824872) which itself raised the other question of how to test reverse dependencies. This brings me back to Debian bug #817286 which, basically proposed the idea of having "proposed updates" for security issues. The problem is while we can upload test packages to stable proposed-updates, we can't do the same in LTS because the suite is closed and we operate only on security packages. This issue came up before in other security upload and we need to think better about how to solve this.

unattended-upgrades Speaking of security upgrades brings me to the question of a bug (Debian bug #867169) that was filed against the wheezy version of unattended-upgrades, which showed that the package simply stopped working since the latest stable release, because wheezy became "oldoldstable". I first suggested using the "codename" but that appears to have been introduced only after wheezy. In the end, I proposed a simple update that would fix the configuration files and uploaded this as DLA-1032-1. This is thankfully fixed in later releases and will not require such hackery when jessie becomes LTS as well.

libmtp Next up is the work on the libmtp vulnerabilities (CVE-2017-9831 and CVE-2017-9832). As I described in my announcement, the work to backport the patch was huge, as upstream basically backported a whole library from the gphoto2 package to fix those issues (and probably many more). The lack of a test suite made it difficult to trust my own work, but given that I had no (negative) feedback, I figured it was okay to simply upload the result and that became DLA-1029-1.

tcpdump I then looked at reproducing CVE-2017-11108, a heap overflow triggered tcpdump would parse specifically STP packets. In Debian bug #867718, I described how to reproduce the issue across all suites and opened an issue upstream, given that the upstream maintainers hadn't responded responded in weeks according to notes in the RedHat Bugzilla issue. I eventually worked on a patch which I shared upstream, but that was rejected as they were already working on it in their embargoed repository. I can explain this confusion and duplication of work with:

1. the original submitter didn't really contact security@tcpdump.org
2. he did and they didn't reply, being just too busy
3. they replied and he didn't relay that information back

I think #2 is most likely: the tcpdump.org folks are probably very busy with tons of reports like this. Still, I should probably have contacted security@tcpdump.org directly before starting my work, even though no harm was done because I didn't divulge issues that were already public. Since then, tcpdump has released 4.9.1 which fixes the issue, but then new CVEs came out that will require more work and probably another release. People looking into this issue must be certain to coordinate with the tcpdump security team before fixing the actual issues.

ipsec-tools Another package that didn't quite have a working solution is the ipsec-tools suite, in which the racoon daemon was vulnerable to a remotely-triggered DOS attack (CVE-2016-10396). I reviewed and fixed the upstream patch which introduced a regression. Unfortunately, there is no test suite or proof of concept to control the results. The reality is that ipsec-tools is really old, and should maybe simply be removed from Debian, in favor of strongswan. Upstream hasn't done a release in years and various distributions have patched up forks of those to keep it alive... I was happy, however, to know that a maintainer will take care of updating the various suites, including LTS, with my improved patch. So this fixes the issue for now, but I would strongly encourage users to switch away from ipsec-tools in the future.

apache2 Finally, I was bitten by the old DLA-841-1 upload I did all the way back in February, as it introduced a regression (Debian bug #858373). It turns out it was possible to segfault Apache workers with a trivial HTTP request, in certain (rather exotic, I might add) configurations (ErrorDocument 400 directive pointing to a cgid script in worker mode). Still, it was a serious regression and I found a part of the nasty long patch we worked on back then that was faulty, and introduced a small fix to correct that. The proposed package unfortunately didn't yield any feedback, and I can only assume it will work okay for people. The result is the DLA-841-2 upload which fixes the regression. I unfortunately didn't have time to work on the remaining CVEs affecting apache2 in LTS at the time of writing.

Triage I also did some miscellaneous triage by filing Debian bug #867477 for poppler in an effort to document better the pending issue. Next up was some minor work on eglibc issues. CVE-2017-8804 has a patch, but it's been disputed. since the main victim of this and the core of the vulnerability (rpcbind) has already been fixed, I am not sure this vulnerability is still a thing in LTS at all. I also looked at CVE-2014-9984, but the code is so different in wheezy that I wonder if LTS is affected at all. Unfortunately, the eglibc gymnastics are a little beyond me and I do not feel confident enough to just push those issues aside for now and let them open for others to look at.

Other free software work And of course, there's my usual monthly volunteer work. My ratio is a little better this time, having reached an about even ratio between paid and volunteer work, whereas this was 60% volunteer work in march.

Announcing ecdysis I recently published ecdysis, a set of template and code samples that I frequently reuse across project. This is probably the least pronounceable project name I have ever chosen, but this is somewhat on purpose. The goal of this project is not collaboration or to become a library: it's just a personal project which I share with the world as a curiosity. To quote the README file:

The name comes from what snakes and other animals do to "create a new snake": they shed their skin. This is not so appropriate for snakes, as it's just a way to rejuvenate their skin, but is especially relevant for anthropods since then "ecdysis" may be associated with a metamorphosis:

Ecdysis is the moulting of the cuticle in many invertebrates of the clade Ecdysozoa. Since the cuticle of these animals typically forms a largely inelastic exoskeleton, it is shed during growth and a new, larger covering is formed. The remnants of the old, empty exoskeleton are called exuviae. Wikipedia

So this project is metamorphosed into others when the documentation templates, code examples and so on are reused elsewhere. For that reason, the license is an unusally liberal (for me) MIT/Expat license. The name also has the nice property of being absolutely unpronounceable, which makes it unlikely to be copied but easy to search online.

It was an interesting exercise to go back into older projects and factor out interesting code. The process is not complete yet, as there are older projects I'm still curious in reviewing. A bunch of that code could also be factored into upstream project and maybe even the Python standard library. In short, this is stuff I keep on forgetting how to do: a proper setup.py config, some fancy argparse extensions and so on. Instead of having to remember where I had written that clever piece of code, I now shove it in the crazy chaotic project where I can find it again in the future.

Beets experiments Since I started using Subsonic (or Libresonic) to manage the music on my phone, album covers are suddenly way more interesting. But my collection so far has had limited album covers: my other media player (gmpc) would download those on the fly on its own and store them in its own database - not on the filesystem. I guess this could be considered to be a limitation of Subsonic, but I actually appreciate the separation of duty here. Garbage in, garbage out: the quality of Subsonic's rendering depends largely on how well setup your library and tags are. It turns out there is an amazing tool called beets to do exactly that kind of stuff. I originally discarded that "media library management system for obsessive-compulsive [OC] music geeks", trying to convince myself i was not an "OC music geek". Turns out I am. Oh well. Thanks to beets, I was able to download album covers for a lot of the albums in my collection. The only covers that are missing now are albums that are not correctly tagged and that beets couldn't automatically fix up. I still need to go through those and fix all those tags, but the first run did an impressive job at getting album covers. Then I got the next crazy idea: after a camping trip where we forgot (again) the lyrics to Georges Brassens, I figured I could start putting some lyrics on my ebook reader. "How hard can that be?" of course, being the start of another crazy project. A pull request and 3 days later, I had something that could turn a beets lyrics database into a Sphinx document which, in turn, can be turned into an ePUB. In the process, I probably got blocked from MusixMatch a hundred times, but it's done. Phew! The resulting e-book is about 8000 pages long, but is still surprisingly responsive. In the process, I also happened to do a partial benchmark of Python's bloom filter libraries. The biggest surprise there was the performance of the set builtin: for small items, it is basically as fast as a bloom filter. Of course, when the item size grows larger, its memory usage explodes, but in this case it turned out to be sufficient and bloom filter completely overkill and confusing. Oh, and thanks to those efforts, I got admitted in the beetbox organization on GitHub! I am not sure what I will do with that newfound power: I was just scratching an itch, really. But hopefully I'll be able to help here and there in the future as well.

Debian package maintenance I did some normal upkeep on a bunch of my packages this month, that were long overdue:

• uploaded slop 6.3.47-1: major new upstream release
• uploaded an NMU for maim 5.4.64-1.1: maim was broken by the slop release
• uploaded pv 1.6.6-1: new upstream release
• uploaded kedpm 1.0+deb8u1 to jessie (oldstable): one last security fix (Debian bug #860817, CVE-2017-8296) for that derelict password manager
• uploaded charybdis 3.5.5-1: new minor upstream release, with optional support for mbedtls
• filed Debian bug #866786 against cryptsetup to make the remote initramfs SSH-based unlocking support multiple devices: thanks to the maintainer, this now works flawlessly in buster and may be backported to stretch
• expanded on Debian bug #805414 against gdm3 and Debian bug #845938 against pulseaudio, because I had trouble connecting my computer to this new Bluetooth speaker. turns out this is a known issue in Pulseaudio: whereas it releases ALSA devices, it doesn't release Bluetooth devices properly. Documented this more clearly in the wiki page
• filed Debian bug #866790 regarding old stray Apparmor profiles that were lying around my system after an upgrade, which got me interested in Debian bug #830502 in turn
• filed Debian bug #868728 against cups regarding a weird behavior I had interacting with a network printer. turns out the other workstation was misconfigured... why are printers still so hard?
• filed Debian bug #870102 to automate sbuild schroots upgrades
• after playing around with rash tried to complete the packaging (Debian bug #754972) of percol with this pull request upstream. this ended up to be way too much overhead and I reverted to my old normal history habits.

## plotOBOS -- displaying overbough/oversold as eg in Bespoke's plots
##
## Copyright (C) 2010 - 2017  Dirk Eddelbuettel
##
## This is free software: you can redistribute it and/or modify it
## under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 2 of the License, or
## (at your option) any later version.
suppressMessages(library(quantmod))     # for getSymbols(), brings in xts too
suppressMessages(library(TTR))          # for various moving averages
plotOBOS <- function(symbol, n=50, type=c("sma", "ema", "zlema"),
                     years=1, blue=TRUE, current=TRUE, title=symbol,
                     ticks=TRUE, axes=TRUE)  
    today <- Sys.Date()
    if (class(symbol) == "character")  
        X <- getSymbols(symbol, from=format(today-365*years-2*n), auto.assign=FALSE)
        x <- X[,6]                          # use Adjusted
      else if (inherits(symbol, "zoo"))  
        x <- X <- as.xts(symbol)
        current <- FALSE                # don't expand the supplied data
     
    n <- min(nrow(x)/3, 50)             # as we may not have 50 days
    sub <- ""
    if (current)  
        xx <- getQuote(symbol)
        xt <- xts(xx$Last, order.by=as.Date(xx$ Trade Time ))
        colnames(xt) <- paste(symbol, "Adjusted", sep=".")
        x <- rbind(x, xt)
        sub <- paste("Last price: ", xx$Last, " at ",
                     format(as.POSIXct(xx$ Trade Time ), "%H:%M"), sep="")
     
    type <- match.arg(type)
    xd <- switch(type,                  # compute xd as the central location via selected MA smoother
                 sma = SMA(x,n),
                 ema = EMA(x,n),
                 zlema = ZLEMA(x,n))
    xv <- runSD(x, n)                   # compute xv as the rolling volatility
    strt <- paste(format(today-365*years), "::", sep="")
    x  <- x[strt]                       # subset plotting range using xts' nice functionality
    xd <- xd[strt]
    xv <- xv[strt]
    xyd <- xy.coords(.index(xd),xd[,1]) # xy coordinates for direct plot commands
    xyv <- xy.coords(.index(xv),xv[,1])
    n <- length(xyd$x)
    xx <- xyd$x[c(1,1:n,n:1)]           # for polygon(): from first point to last and back
    if (blue)  
        blues5 <- c("#EFF3FF", "#BDD7E7", "#6BAED6", "#3182BD", "#08519C") # cf brewer.pal(5, "Blues")
        fairlylight <<- rgb(189/255, 215/255, 231/255, alpha=0.625) # aka blues5[2]
        verylight <<- rgb(239/255, 243/255, 255/255, alpha=0.625) # aka blues5[1]
        dark <<- rgb(8/255, 81/255, 156/255, alpha=0.625) # aka blues5[5]
        ## buglet in xts 0.10-0 requires the <<- here
      else  
        fairlylight <<- rgb(204/255, 204/255, 204/255, alpha=0.5)  # two suitable grays, alpha-blending at 50%
        verylight <<- rgb(242/255, 242/255, 242/255, alpha=0.5)
        dark <<- 'black'
     
    plot(x, ylim=range(range(x, xd+2*xv, xd-2*xv, na.rm=TRUE)), main=title, sub=sub, 
         major.ticks=ticks, minor.ticks=ticks, axes=axes) # basic xts plot setup
    addPolygon(xts(cbind(xyd$y+xyv$y, xyd$y+2*xyv$y), order.by=index(x)), on=1, col=fairlylight)  # upper
    addPolygon(xts(cbind(xyd$y-xyv$y, xyd$y+1*xyv$y), order.by=index(x)), on=1, col=verylight)    # center
    addPolygon(xts(cbind(xyd$y-xyv$y, xyd$y-2*xyv$y), order.by=index(x)), on=1, col=fairlylight)  # lower
    lines(xd, lwd=2, col=fairlylight)   # central smooted location
    lines(x, lwd=3, col=dark)           # actual price, thicker
 

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Blends

Andreas Tille

Debian Science

Michael Banck

Cloud and containers

Luca Filipozzi

Embedded

Pending

Systems administration, automation and orchestation

Pending

Security

Gunnar Wolf

• #783656 libhtml-microformats-perl: "libhtml-microformats-perl: missing dependency on libmodule-pluggable-perl"
  fix in jessie (pkg-perl)
• #788008 libcgi-application-plugin-anytemplate-perl: "libcgi-application-plugin-anytemplate-perl: missing dependency on libclone-perl"
  fix in jessie (pkg-perl)
• #788350 libhttp-proxy-perl: "libhttp-proxy-perl: FTBFS - proxy tests"
  fix in jessie (pkg-perl)
• #808454 src:libdata-faker-perl: "libdata-faker-perl: FTBFS under some locales (eg. fr_CH.UTF-8)"
  fix in jessie (pkg-perl)
• #824843 libsys-syscall-perl: "libsys-syscall-perl: FTBFS on arm64: test suite failures"
  fix in jessie (pkg-perl)
• #824936 libsys-syscall-perl: "libsys-syscall-perl: FTBFS on mips*: test failures"
  fix in jessie (pkg-perl)
• #825012 libalgorithm-permute-perl: "libalgorithm-permute-perl: FTBFS with Perl 5.24: undefined symbol: PUSHBLOCK"
  upload new upstream release (pkg-perl)
• #826136 libsys-syscall-perl: "libsys-syscall-perl FTBFS on hppa arch (with patch)"
  fix in jessie (pkg-perl)
• #826471 intltool: "intltool: Unescaped left brace in regex is deprecated at /usr/bin/intltool-update line 1070"
  extend existing patch, uploaded by maintainer
• #826502 quilt: "quilt: FTBFS with perl 5.26: Unescaped left brace in regex"
  apply patch from ntyni (pkg-perl)
• #826505 swissknife: "swissknife: FTBFS with perl 5.26: Unescaped left brace in regex"
  add patches from ntyni, upload to DELAYED/5
• #839208 libio-compress-perl: "libio-compress-perl: uninstallable, current version superseded by Perl 5.24.1"
  upload newer upstream release (pkg-perl)
• #839218 nama: "nama: FTBFS because of perl's lack of stack reference counting"
  apply patch from Balint Reczey (pkg-perl)
• #848060 src:libx11-protocol-other-perl: "libx11-protocol-other-perl: FTBFS randomly (failing tests)"
  fix in jessie (pkg-perl)
• #855920 src:fail2ban: "fail2ban: FTBFS: test_rewrite_file: AssertionError: False is not true"
  try to reproduce, later propose different fix
• #855951 src:libsecret: "libsecret FTBFS with test failures on many architectures"
  try to reproduce
• #856133 src:shiboken: "shiboken FTBFS on i386/armel/armhf: other_collector_external_operator test failed"
  try to reproduce
• #857087 nsscache: "nsscache: /usr/share/nsscache not installed"
  triaging, later fixed by maintainer
• #858501 libmojomojo-perl: "libmojomojo-perl: broken symlink: /usr/share/perl5/MojoMojo/root/static/js/jquery.autocomplete.js -> ../../../../../javascript/jquery-ui/ui/jquery.ui.autocomplete.min.js"
  fix symlink (pkg-perl)
• #858509 cl-csv-data-table: "cl-csv-data-table: broken symlink: /usr/share/common-lisp/systems/cl-csv-data-table.asd -> ../source/cl-csv-data-table/cl-csv-data-table.asd"
  propose a patch
• #859539 filezilla: "filezilla: Filezilla crashes at startup"
  simply close the bug :)
• #859963 src:mimetic: "mimetic FTBFS on architectures where char is unsigned"
  add patch to make variable signed
• #860142 libgeo-ip-perl: "libgeo-ip-perl: should recommend geoip-database and geoip-database-extra"
  analyse, lower severity (pkg-perl)
• #863049 release.debian.org: "jessie-pu: package shutter/0.92-0.1+deb8u2"
  fix in jessie (pkg-perl)
• #865045 xmltv: "xmltv: FTBFS with Perl 5.26: t/test_filters.t failure"
  sponsor maintainer upload
• #865224 src:uwsgi: "uwsgi: ftbfs with multiple supported python3 versions"
  adjust build dependencies, upload NMU with maintainer's permission
• #865380 libtest-unixsock-perl: "libtest-unixsock-perl: Build-Conflicts-Indep: libtest-simple-perl (>= 1.3), including Perl 5.26"
  conditionally skip a test (pkg-perl)
• #865888 nagios-plugin-check-multi: "nagios-plugin-check-multi: FTBFS with Perl 5.26: Unescaped left brace in regex is deprecated here"
  update unescaped-left-brace-in-regex.patch, upload to DELAYED/5
• #866317 html2ps: "html2ps: relies on deprecated Perl syntax/features, breaks with 5.26"
  prepare a patch, later do QA upload
• #866934 libhttp-oai-perl: "libhttp-oai-perl: /usr/bin/oai_pmh uses the 'encoding' pragma, breaks with Perl 5.26"
  patch out "use encoding" (pkg-perl)
• #866944 libmecab-perl: "libmecab-perl: FTBFS: no such file or directory: /var/lib/mecab/dic/debian/dicrc"
  build-depend on fixed version (pkg-perl)
• #866981 src:libpgobject-type-datetime-perl: "libpgobject-type-datetime-perl FTBFS: You planned 15 tests but ran 5."
  upload new upstream release (pkg-perl)
• #866984 src:libpgobject-type-json-perl: "libpgobject-type-json-perl FTBFS: You planned 9 tests but ran 5."
  upload new upstream release (pkg-perl)
• #866991 libcpan-meta-perl: "libcpan-meta-perl: uninstallable in unstable"
  fix versioned Breaks/Replaces (pkg-perl)
• #867046 gwhois: "gwhois: fails with perl 5.26: The encoding pragma is no longer supported at /usr/bin/gwhois line 80."
  drop "use encoding", upload to DELAYED/5
• #867208 src:libpgobject-type-bytestring-perl: "libpgobject-type-bytestring-perl FTBFS: Failed 1/5 test programs. 0/8 subtests failed."
  upload new upstream release (pkg-perl)
• #867210 libtext-mecab-perl: "libtext-mecab-perl: FTBFS: test failures: Failed to create mecab instance"
  build-depend on fixed version (pkg-perl)
• #867983 libclass-load-perl: "libclass-load-perl: FTBFS: t/012-without-implementation.t failure"
  upload new upstream release (pkg-perl)
• #867984 libclass-load-xs-perl: "libclass-load-xs-perl: FTBFS: t/012-without-implementation.t failure"
  upload new upstream release (pkg-perl)
• #868069 liburi-namespacemap-perl: "liburi-namespacemap-perl: unbuildable with sbuild"
  fix build dependencies (pkg-perl)
• #868075 libperlx-assert-perl: "libperlx-assert-perl: missing dependency on libkeyword-simple-perl libdevel-declare-perl"
  fix dependencies (pkg-perl)
• #868613 src:liblog-report-lexicon-perl: "liblog-report-lexicon-perl FTBFS: Failed 2/10 test programs. 2/305 subtests failed."
  upload new upstream release (pkg-perl)
• #868888 get-flash-videos: "Fails: Can't locate Term/ProgressBar.pm"
  add missing dependency (pkg-perl)
• #869208 src:libalgorithm-svm-perl: "libalgorithm-svm-perl FTBFS on 32bit: SVM.c: loadable library and perl binaries are mismatched (got handshake key 0x80c0080, needed 0x81c0080)"
  add patch to add ccflags (pkg-perl)
• #869360 slic3r: "slic3r: missing dependency on perlapi-*"
  propose a patch
• #869383 src:clanlib: "clanlib FTBFS with perl 5.26"
  propose a patch
• #869418 src:libtest-taint-perl: "libtest-taint-perl FTBFS on armel: not ok 2 - $ENV TEST_ACTIVE is tainted"
  add a patch (pkg-perl)
• #869429 vile: "vile: Uninstallable after BinNMU / not binNMU-safe"
  some triaging

• Debian Policy sprint
• conversations about using git for Debian packaging, especially with regard to dgit
  • writing up or following up on feature requests
• Emacs team sprint
  • talking to maintainers about transitioning their packages to use dh-elpa
• submitting and revising patch series to dgit
• writing a test suite for git-remote-gcrypt

• dh + dh_* now scales a lot better for large number of binary packages. Even more so with parallel builds.
• Most dh_* tools are now a lot faster when creating many directories or installing files.
• dh_prep and dh_clean now bulk removals.
• dh_install can now bulk some installations. For a concrete corner-case, libssl-doc went from approximately 11 seconds to less than a second. This optimization is implicitly disabled with exclude (among other).
• dh_installman now scales a lot better with many manpages. Even more so with parallel builds.
• dh_installman has restored its performance under fakeroot (regression since 10.2.2)

• avoiding fork+exec of commands for things doable natively in perl. Especially, when each fork+exec only process one file or dir.
• bulking as many files/dirs into the call as possible, where fork+exec is still used.
• caching / memorizing slow calls (e.g. in parts of pkgfile inside Dh_Lib)
• adding an internal API for dh to do bulk check for pkgfiles. This is useful for dh when checking if it should optimize out a helper.
• and, of course, doing things in parallel where trivially possible.

• If you use install_ file,prog,lib,dir , then it will come out of the box. These functions are available in Debian/stable. On a related note, if you use doit to call install (or mkdir ), then please consider migrating to these functions instead.
• If you need to reset owner+mode (chown 0:0 FILE + chmod MODE FILE), consider using reset_perm_and_owner. This is also available in Debian/stable.
  • CAVEAT: It is not recursive and YMMV if you do not need the chown call (due to fakeroot).
• If you have a lot of items to be processed by a external tool, consider using xargs(). Since 10.5.1, it is now possible to insert the items anywhere in the command rather than just in the end.
• If you need to remove files, consider using the new rm_files function. It removes files and silently ignores if a file does not exist. It is also available since 10.5.1.
• If you need to create symlinks, please consider using make_symlink (available in Debian/stable) or make_symlink_raw_target (since 10.5.1). The former creates policy compliant symlinks (e.g. fixup absolute symlinks that should have been relative). The latter is closer to a ln -s call.
• If you need to rename a file, please consider using rename_path (since 10.5). It behaves mostly like mv -f but requires dest to be a (non-existing) file.
• Have a look at whether on_pkgs_in_parallel() / on_items_in_parallel() would be suitable for enabling parallelization in your tool.
  • The emphasis for these functions is on making parallelization easy to add with minimal code changes. It pre-distributes the items which can lead to unbalanced workloads, where some processes are idle while a few keeps working.

• Helmut Grohne
• Kurt Roeckx
• Gianfranco Costamagna
• Iain Lane
• Sven Joachim
• Adrian Bunk
• Michael Stapelberg