This is an overview of the hardware and internals of the PDP-8/e replica I m building.

The front panel board

If you know the original or remember the picture from the first post it is clear that this is a functional replica not aiming to be as pretty as those of the other projects I mentioned. I have reordered the switches into two rows to make the board more compact (which also means cheaper) without sacrificing usability. There s the two rows of display lights plus one run light the 8/e provides. The upper row is the address made up of 12 bits of memory address and 3 bits of extended memory address or field. Below are the 12 bits indicator which can show one data set out of six as selected by the user. All the switches of the original are implemented as more compact buttons¹. While momentary switches are easily substituted by buttons, all buttons implementing two position switches toggle on/off with each press and they have a LED above that shows the current state. The six position rotary switch is implemented as a button cycling through all indicator displays together with six LEDs which show the active selection. Markings show the meaning of the indicator and switches as on the original, grouped in threes as the predominant numbering system for the PDPs was octal. The upper line shows the meaning for the state indicator, the middle for the status indicator and bit numbers for the rest. Note that on the PDP-8 and opposite to modern conventions, the most significant bit was numbered 0. I designed it as a pure front panel board without any PDP-8 simulation parts. The buttons and associated lights are controllable via SPI lines with a 3.3 V supply. The address and indicator lights have a separate common anode configuration with all cathodes individually available on a pin header without any resistors in the path, leaving voltage and current regulation up to the simulation board. This board is actually a few years old from a similar project where I emulated the PDP-8 in software on a microcontroller and the flexible design allowed me to reuse it unchanged.

The main board

This is where the magic happens. You can see three big ICs on the board: On the left is the STM32F405 microcontroller (with ARM Cortex-M4 core), the bigger one in the middle is the Altera² MAX 10 FPGA and finally to the right is the SRAM that is large enough to hold all the main memory of the 32 KW maximum expansion of the PDP-8/e. The two smaller chips to the right of that are just buffers that drive the front panel address LEDs, the small chip at the top left is a RS-232 level shifter. The idea behind this is that the PDP-8 and peripherals that are simple to directly implement, such as GPIO or a serial port, are fully on the FPGA. Other peripherals such as paper and magnetic tape and disks, which are after all not connected to real PDP-8 drives but disk images on a microSD, are implemented on the microcontroller interfacing with stub devices in the FPGA. Compared to implementing everything everything in the FPGA, the STM32F4 has the advantage of useful built in peripherals such as two host/device capable USB ports. 5 V tolerant I/O pins are very useful and simply not available in any FPGA. I have to admit that this board was a bit of a rush job in order to have something at all to show at the Vintage Computer Festival Europe 18.0. Given that it was my first time designing a board with a large microcontroller and the first time with an FPGA, it wasn t exactly my fastest progressing project either and I got basic functionality (front panel allows toggling in small programs and running them) working just in time. For various reasons the project hasn t progressed much since, so the following is still just plans, but plans for which the hardware was designed. Since the aim is to have a cycle accurate PDP-8/e implementation, digital I/O was always planned. Rather than defining my own header I have included Arduino R3 compatible headers (for 3.3 V compatible boards only) that have become a popular even outside the Arduino world for this purpose. The digital Arduino pins are connected directly to the FPGA and will be directly controllable by PDP-8 software. The downside of choosing the Arduino headers is that the original PDP-8 digital I/O interface is not a perfect match since it naturally has 12 lines whereas the Arduino has 15. The analog inputs are not connected to the FPGA, the documentation of the MAX10 s ADC in the EQFP package are not very encouraging. They are connected to the STM32 instead³. Another interface connected directly to the FPGA and that would be directly under PDP-8 control is a standard 9 pin RS-232 interface. RX, TX, CTS and RTS are connected and level-shifted between 3.3 V and RS-232 levels by a MAX3232. Besides the PDP-8, I also plan to implement a full video terminal on the board. The idea is that with a power supply, keyboard and monitor this board would form a complete system without the need of connecting another computer to act as a terminal. To that end, there is a VGA port attached to the FPGA with simple resistor network DACs for 9 bits color (RGB with 3 bits each). This is another spot where I left myself room to expand, for e.g. a VT220 you really only need one color in two brightness levels. Keyboards will be connected either via the PS/2 connector on the right or the USB-A host port at the top left. Last of the interface ports is the USB micro-AB port on the left, which for now I am using only for power supply. I mainly plan to use it to provide alternative or additional serial ports to the PDP-8 or to export the video terminal serial port for testing purposes. Other possible uses are access to the image files on the microSD and firmware updates. This has gotten rather long again, so I m stopping here and leave some implementation notes for another post.

---

1. They are also much cheaper. Given the large number of switches, the savings are substantial. Additionaly the buttons are nicer to operate than long rows of tiny switches. ^[return]
2. Or rather Intel now. At least Altera s web site, documentation and software have already been thoroughly rebranded, but the chips I got were produced prior to that. ^[return]
3. That s not to say that the analog conversions on the STM32 are necessarily better than those of the MAX10 when you can t follow their guidelines, I have no comparisons. Certainly following the guidelines would have been prohibitive given how many pins usage they restrict. ^[return]

TL;DR: Each of Linux 2.6.39, 3.6 and 4.0 brings notable performance improvements for the IPv4 route lookup process.

---

In a previous article, I explained how Linux implements an IPv4 routing table with compressed tries to offer excellent lookup times. The following graph shows the performance progression of Linux through history: Two scenarios are tested:

• 500,000 routes extracted from an Internet router (half of them are /24), and
• 500,000 host routes (/32) tightly packed in 4 distinct subnets.

All kernels are compiled with GCC 4.9 (from Debian Jessie). This version is able to compile older kernels¹ as well as current ones. The kernel configuration used is the default one with CONFIG_SMP and CONFIG_IP_MULTIPLE_TABLES options enabled (however, no IP rules are used). Some other unrelated options are enabled to be able to boot them in a virtual machine and run the benchmark. The measurements are done in a virtual machine with one vCPU². The host is an Intel Core i5-4670K and the CPU governor was set to performance . The benchmark is single-threaded. Implemented as a kernel module, it calls fib_lookup() with various destinations in 100,000 timed iterations and keeps the median. Timings of individual runs are computed from the TSC (and converted to nanoseconds by assuming a constant clock). The following kernel versions bring a notable performance improvement:

• In Linux 2.6.39, commit 3630b7c050d9, David Miller removes the hash-based routing table implementation to switch to the LPC-trie implementation (available since Linux 2.6.13 as a compile-time option). This brings a small regression for the scenario with many host routes but boosts the performance for the general case.
• In Linux 3.0, commit 281dc5c5ec0f, the improvement is not related to the network subsystem. Linus Torvalds disables the compiler size-optimization from the default configuration. It was believed that optimizing for size would help keeping the instruction cache efficient. However, compilers generated under-performing code on x86 when this option was enabled.
• In Linux 3.6, commit f4530fa574df, David Miller adds an optimization to not evaluate IP rules when they are left unconfigured. From this point, the use of the CONFIG_IP_MULTIPLE_TABLES option doesn t impact the performances unless some IP rules are configured. This version also removes the route cache (commit 5e9965c15ba8). However, this has no effect on the benchmark as it directly calls fib_lookup() which doesn t involve the cache.
• In Linux 4.0, notably commit 9f9e636d4f89, Alexander Duyck adds several optimizations to the trie lookup algorithm. It really pays off!
• In Linux 4.1, commit 0ddcf43d5d4a, Alexander Duyck collapses the local and main tables when no specific IP rules are configured. For non-local traffic, both those tables were looked up.

DRM leasing part three (vblank) The last couple of weeks have been consumed by getting frame sequence numbers and events handled within the leasing environment (and Vulkan) correctly. Vulkan EXT_display_control extension This little extension provides the bits necessary for applications to track the display of frames to the user.

VkResult
vkGetSwapchainCounterEXT(VkDevice           device,
             VkSwapchainKHR         swapchain,
             VkSurfaceCounterFlagBitsEXT    counter,
             uint64_t           *pCounterValue);

This function just retrieves the current frame count from the display associated with swapchain.

VkResult
vkRegisterDisplayEventEXT(VkDevice          device,
              VkDisplayKHR          display,
              const VkDisplayEventInfoEXT   *pDisplayEventInfo,
              const VkAllocationCallbacks   *pAllocator,
              VkFence           *pFence);

This function creates a fence that will be signaled when the specified event happens. Right now, the only event supported is when the first pixel of the next display refresh cycle leaves the display engine for the display. If you want something fancier (like two frames from now), you get to do that on your own using this basic function. drmWaitVBlank drmWaitVBlank is the existing interface for all things sequence related and has three modes (always nice to have one function do three things, I think). It can:

1. Query the current vblank number
2. Block until a specified vblank number
3. Queue an event to be delivered at a specific vblank number

This interface has a few issues:

• It has been kludged into supporting multiple CRTCs by taking bits from the 'type' parameter to hold a 'pipe' number, which is the index in the kernel into the array of CRTCs.
• It has a random selection of 'int' and 'long' datatypes in the interface, making it need special helpers for 32-bit apps running on a 64-bit kernel.
• Times are in microseconds, frame counts are 32 bits. Vulkan does everything in nanoseconds and wants 64-bits of frame counts.

For leases, figuring out the index into the kernel list of crtcs is pretty tricky -- our lease has a subset of those crtcs, so we can't actually compute the global crtc index. drmCrtcGetSequence

int drmCrtcGetSequence(int fd, uint32_t crtcId,
               uint64_t *sequence, uint64_t *ns);

Here's a simple new function hand it a crtc ID and it provides the current frame sequence number and the time when that frame started (in nanoseconds). drmCrtcQueueSequence

int drmCrtcQueueSequence(int fd, uint32_t crtcId,
                 uint32_t flags, uint64_t sequence,
             uint64_t user_data);
struct drm_event_crtc_sequence  
    struct drm_event    base;
    __u64           user_data;
    __u64           time_ns;
    __u64           sequence;
 ;

This will cause a CRTC_SEQUENCE event to be delivered at the start of the specified frame sequence. That event will include the frame when the event was actually generated (in case it's late), along with the time (in nanoseconds) when that frame was started. The event also includes a 64-bit user_data value, which can be used to hold a pointer to whatever data the application wants to see in the event handler. The 'flags' argument contains a combination of:

#define DRM_CRTC_SEQUENCE_RELATIVE      0x00000001  /* sequence is relative to current */
#define DRM_CRTC_SEQUENCE_NEXT_ON_MISS      0x00000002  /* Use next sequence if we've missed */
#define DRM_CRTC_SEQUENCE_FIRST_PIXEL_OUT   0x00000004  /* Signal when first pixel is displayed */

These are similar to the values provided for the drmWaitVBlank function, except I've added a selector for when the event should be delivered to align with potential future additions to Vulkan. Right now, the only time you can ask for is first-pixel-out, which says that the event should correspond to the display of the first pixel on the screen. DRM events Vulkan fences With the kernel able to deliver a suitable event at the next frame, all the Vulkan code needed was a to create a fence and hook it up to such an event. The existing fence code only deals with rendering fences, so I added window system interface (WSI) fencing infrastructure and extended the radv driver to be able to handle both kinds of fences within that code. Multiple waiting threads I've now got three places which can be waiting for a DRM event to appear:

1. Frame sequence fences.
2. Wait for an idle image. Necessary when you want an image to draw the next frame to.
3. Wait for the previous flip to complete. The kernel can only queue one flip at a time, so we have to make sure the previous flip is complete before queuing another one.

Vulkan allows these to be run from separate threads, so I needed to deal with multiple threads waiting for a specific DRM event at the same time. XCB has the same problem and goes to great lengths to manage this with a set of locking and signaling primitives so that only one thread is ever doing poll or read from the socket at time. If another thread wants to read at the same time, it will block on a condition variable which is then signaled by the original reader thread at the appropriate time. It's all very complicated, and it didn't work reliably for a number of years. I decided to punt and just create a separate thread for processing all DRM events. It blocks using poll(2) until some events are readable, processes those and then broadcasts to a condition variable to notify any waiting threads that 'something' has happened. Each waiting thread simply checks for the desired condition and if not satisfied, blocks on that condition variable. It's all very simple looking, and seems to work just fine. Code Complete, Validation Remains At this point, all of the necessary pieces are in place for the VR application to take advantage of an HMD using only existing Vulkan extensions. Those will be automatically mapped into DRM leases and DRM events as appropriate. The VR compositor application is working pretty well; tests with Dota 2 show occasional jerky behavior in complex scenes, so there's clearly more work to be done somewhere. I need to go write a pile of tests to independently verify that my code is working. I wonder if I'll need to wire up some kind of light sensor so I can actually tell when frames get displayed as it's pretty easy to get consistent-but-wrong answers in this environment. Source Code

• Linux. This is based off of a reasonably current drm-next branch from Dave Airlie. 965 commits past 4.12 RC3. git://people.freedesktop.org/~keithp/linux drm-lease-v3
• X server (which includes xf86-video-modesetting). This is pretty close to master. git://people.freedesktop.org/~keithp/xserver drm-lease
• RandR protocol changes git://people.freedesktop.org/~keithp/randrproto drm-lease
• xcb proto (no changes to libxcb sources, but it will need to be rebuilt) git://people.freedesktop.org/~keithp/xcb/proto drm-lease
• DRM library. About a dozen patches behind master. git://people.freedesktop.org/~keithp/drm drm-lease
• Mesa. Branched early this month (4 June), this is pretty far from master. git://people.freedesktop.org/~keithp/mesa drm-lease

Here is my monthly update covering what I have been doing in the free software world (previous month):

• Updated travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds:
  • Support Debian "buster". (commit)
  • Set TRAVIS=true environment variable when running autopkgtests. (#45)
• Updated the documentation in django-slack, my library to easily post messages to the Slack group-messaging utility to link to Slack's own message formatting documentation. (#66)
• Added "buster" support to local-debian-mirror, my package to easily maintain and customise a local Debian mirror via the DebConf configuration tool. (commit)

---

Weblate 2.15 has been released today. It is slightly behind schedule what was mostly caused by my vacation. As with 2.14, there are quite a lot of security improvements based on reports we got from HackerOne program and various new features. Full list of changes:

• Show more related translations in other translations.
• Add option to see translations of current unit to other languages.
• Use 4 plural forms for Lithuanian by default.
• Fixed upload for monolingual files of different format.
• Improved error messages on failed authentication.
• Keep page state when removing word from glossary.
• Added direct link to edit secondary language translation.
• Added Perl format quality check.
• Added support for rejecting reused passwords.
• Extended toolbar for editing RTL languages.

If you are upgrading from older version, please follow our upgrading instructions. You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects. Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure. Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: Debian English SUSE Weblate

Time is a big enough subject that I'd never finish writing about it. Fortunately this is a Linux technical blog, so I only have to talk about it from that perspective. In a departure from our usual article series format, we are going to start with an introductory article and follow up starting from basics, introducing new concepts in order of complexity. What is time? Time is seconds since the unix epoch. The Unix epoch is midnight, Thursday, 1st January 1970 in the UTC time zone. If you know the unix time and how many seconds in each day since then you can use the unix timestamp to work out what time it is now in UTC time. If you want to know the time in another timezone the na ve thing is to apply a timezone offset based on how far or behind the zone is compared to UTC. This is non-trivial since time is a political issue, requiring global coordination with hundreds of countries coming in and out of [daylight saving time][] and various other administrative changes, time often changes. Some times don't exist in some timezones. Samoa skipped a day and changed time zone. A side-effect of your system clock being in Unix time is that to know local time you need to know where you are. The talk linked to in https://fosdem.org/2017/schedule/event/python_datetime/ is primarily about time in python, but covers the relationship between time stamps, clock time and time zones. http://www.creativedeletion.com/2015/01/28/falsehoods-programmers-date-time-zones.html is a good summary of some of the common misconceptions of time zone handling.

---

Now that we've covered some of the concepts we can discuss in future articles how this is implemented in Linux.

With the release of stretch we are pleased to open the doors for stretch-backports and jessie-backports-sloppy. \o/ As usual with a new release we will change a few things for the backports service.

What to upload where As a reminder, uploads to a release-backports pocket are to be taken from release + 1, uploads to a release-backports-sloppy pocket are to be taken from release + 2. Which means:

Source Distribution	Backports Distribution	Sloppy Distribution
buster	stretch-backports	jessie-backports-sloppy
stretch	jessie-backports	-

Deprecation of LTS support for backports We started supporting backports as long as there is LTS support as an experiment. Unfortunately it didn t worked, most maintainers didn t wanted to support oldoldstable-backports (squeeze) for the lifetime of LTS. So things started to rot in squeeze and most packages didn t received updates. After long discussions we decided to deprecate LTS support for backports. From now on squeeze-backports(-sloppy) is closed and will not receive any updates. Expect it to get removed from the mirrors and moved to archive in the near future.

BSA handling We - the backports team - didn t scale well in processing BSA requests. To get things better in the future we decided to change the process a little bit. If you upload a package which fixes security problems please fill out the BSA template and create a ticket in the rt tracker (see https://backports.debian.org/Contribute/#index3h2 for details).

Stretching the rules From time to time its necessary to not follow the backports rules, like a package needs to be in testing or a version needs to be in Debian. If you think you have one of those cases, please talk to us on the list before upload the package.

Thanks Thanks have to go out to all people making backports possible, and that includes up front the backporters themself who do upload the packages, track and update them on a regular basis, but also the buildd team making the autobuilding possible and the ftp masters for creating the suites in the first place. We wish you a happy stretch :) Alex, on behalf of the Backports Team

There are two Qt 5.7 submodules that we could not package in time for Strech but are/will be available in their 5.7 versions in testing. This are qtdeclarative-render2d-plugin and qtvirtualkeyboard.

declarative-render2d-plugin makes use of the Raster paint engine instead of OpenGL to render the contents of a scene graph, thus making it useful when Qt Quick2 applications are run in a system without OpenGL 2 enabled hardware. Using it might require tweaking Debian's /etc/X11/Xsession.d/90qt5-opengl. On Qt 5.9 and newer this plugin is merged in Qt GUI so there should be no need to perform any action on the user's behalf.

Debian's VirtualKeyboard currently has a gotcha: we are not building it with the embedded code it ships. Upstream ships 3rd party code but lacks a way to detect and use the system versions of them. See QTBUG-59594, patches are welcomed. Please note that we prefer patches sent directly upstream to the current dev revision, we will be happy to backport patches if necessary.
Yes, this means no hunspell, openwnn, pinyin, tcime nor lipi-toolkit/t9write support.

So recently I wrote a Linux Security Module (LSM) which would deny execution of commands, unless an extended attribute existed upon the filesystem belonging to the executables. The whitelist-LSM worked well, but it soon became apparent that it was a little pointless. Most security changes are pointless unless you define what you're defending against - your "threat model". In my case it was written largely as a learning experience, but also because I figured it seemed like it could be useful. However it wasn't actually as useful because you soon realize that you have to whitelist too much:

• The redis-server binary must be executable, to the redis-user, otherwise it won't run.
• /usr/bin/git must be executable to the git user.

In short there comes a point where user alice must run executable blah. If alice can run it, then so can mallory. At which point you realize the exercise is not so useful. Taking a step back I realized that what I wanted to to prevent was the execution of unknown/unexpected, and malicious binaries How do you identify known-good binaries? Well hashes & checksums are good. So for my second attempt I figured I'd not look for a mere "flag" on a binary, instead look for a valid hash. Now my second LSM is invoked for every binary that is executed by a user:

• When a binary is executed the sha1 hash is calculated of the files contents.
• If that matches the value stored in an extended attribute the execution is permitted.
  • If the extended-attribute is missing, or the checksum doesn't match, then the execution is denied.

In practice this is the same behaviour as the previous LSM - a binary is either executable, because there is a good hash, or it is not, because it is missing or bogus. If somebody deploys a binary rootkit this will definitely stop it from executing, but of course there is a huge hole - scripting-languages:

• If /usr/bin/perl is whitelisted then /usr/bin/perl /tmp/exploit.pl will succeed.
• If /usr/bin/python is whitelisted then the same applies.

Despite that the project was worthwhile, I can clearly describe what it is designed to achieve ("Deny the execution of unknown binaries", and "Deny binaries that have been modified"), and I learned how to hash a file from kernel-space - which was surprisingly simple.

(Yes I know about IMA and EVM - this was a simple project for learning purposes. Public-key signatures will be something I'll look at next/soon/later. :)

Perhaps the only other thing to explore is the complexity in allowing/denying actions based on the user - in a human-readable fashion, not via UIDs. So www-data can execute some programs, alice can run a different set of binaries, and git can only run /usr/bin/git. Of course down that path lies apparmour, selinux, and madness..

Most routers/switches out there store their configuration in plain text, which is nice for backups. I m talking about Cisco, Juniper, HPE, etc. The configuration of our routers are being changed several times a day by the operators, and in this case we lacked some proper way of tracking these changes. Some of these routers come with their own mechanisms for doing backups, and depending on the model and version perhaps they include changes-tracking mechanisms as well. However, they mostly don t integrate well into our preferred version control system, which is git. After some internet searching, I found rancid, which is a suite for doing tasks like this. But it seemed rather complex and feature-full for what we required: simply fetch the plain text config and put it into a git repo. Worth noting that the most important drawback of not triggering the change-tracking from the router/switch is that we have to follow a polling approach: loggin into each device, get the plain text and the commit it to the repo (if changes detected). This can be hooked in cron, but as I said, we lost the sync behaviour and won t see any changes until the next cron is run. In most cases, we lost authorship information as well. But it was not important for us right now. In the future this is something that we will have to solve. Also, some routers/switches lack some basic SSH security improvements, like public-key authentication, so we end having to hard-code user/pass in our worker script. Since we have several devices of the same type, we just iterate over their names. For example, this is what we use for hp comware devices: You should create a read-only user git in the devices. And beware that each device model has the config file stored in a different place. For reference, in HP comware, the file to scp is flash:/startup.cfg. And you might try creating the user like this: In Junos/Juniper, the file you should scp is /config/juniper.conf.gz and the script should gunzip the data before committing. For the read-only user, try is something like this: The file to scp in HP procurve is /cfg/startup-config. And for the read-only user, try something like this: What would be the ideal situation? Get the device controlled directly by git (i.e. commit > git hook > device update) or at least have the device to commit the changes by itself to git. I m open to suggestions :-)

I had to extract copyright information from the git repository of the crate upstream. The need aroused as part of updating debcargo, tool to create Debian package source from the Rust crate. General idea behind taking copyright information from git is to extract starting and latest contribution year for every author/committer. This can be easily achieved using following shell snippet Now challenge was to execute these command in Rust and get the required answer. So first step was I looked at std::process, default standard library support for executing shell commands. My idea was to execute first command to extract authors into a Rust vectors or array and then have 2 remaining command to extract years in a loop. (Yes I do not need additional author_email command in Rust as I can easily get both in the first command which is used in for loop of shell snippet and use it inside another loop). So I setup to 3 commands outside the loop with input and output redirected, following is snippet should give you some idea of what I tried to do. And inside the loop I would create additional 2 git commands read their output via pipe and feed it to head command. This is where I learned that it is not straight forward as it looks :-). std::process::Command type does not implement Copy nor Clone traits which means one use of it I will give up the ownership!. And here I started fighting with borrow checker. I need to duplicate declarations to make sure I've required commands available all the time. Additionally I needed to handle error output at every point which created too many nested statements there by complicating the program and reducing its readability When all started getting out of control I gave a second thought and wondered if it would be good to write down this in shell script ship it along with debcargo and use the script Rust program. This would satisfy my need but I would need to ship additional script along with debcargo which I was not really happy with. Then a search on crates.io revealed subprocess, a crate designed to be similar with subprocess module from Python!. Though crate is not highly downloaded it still looked promising, especially the trait implements a trait called BitOr which allows use of operator to chain the commands. Additionally it allows executing full shell commands without need of additional chaining of argument which was done above snippet. End result a much simplified easy to read and correct function which does what was needed. Below is the function I wrote to extract copyright information from git repo. Of course it is not as short as the shell or probably Python code, but that is fine as Rust is system level programming language (which is intended to replace C/C++) and doing complex Shell code (complex due to need of shell pipelines) in approximately 50 lines of code in safe and secure way is very much acceptable. Besides code is as much readable as a plain shell snippet thanks to the operator implemented by subprocess crate.

Since 2003, the Debian project has been running a server called Alioth to host source code version control systems. The server will hit the end of life of the Debian LTS release (Wheezy) next year; that deadline raised some questions regarding the plans for the server over the coming years. Naturally, that led to a discussion regarding possible replacements. In response, the current Alioth maintainer, Alexander Wirt, announced a sprint to migrate to pagure, a free-software "Git-centered forge" written in Python for the Fedora project, which LWN covered last year. Alioth currently runs FusionForge, previously known as GForge, which is the free-software fork of the SourceForge code base when that service closed its source in 2001. Alioth hosts source code repositories, mainly Git and Subversion (SVN) and, like other "forge" sites, also offers forums, issue trackers, and mailing list services. While other alternatives are still being evaluated, a consensus has emerged on a migration plan from FusionForage to a more modern and minimal platform based on pagure.

Why not GitLab? While this may come as a surprise to some who would expect Debian to use the more popular GitLab project, the discussion and decision actually took place a while back. During a lengthy debate last year, Debian contributors discussed the relative merits of different code-hosting platforms, following the initiative of Debian Developer "Pirate" Praveen Arimbrathodiyil to package GitLab for Debian. At that time, Praveen also got a public GitLab instance running for Debian (gitlab.debian.net), which was sponsored by GitLab B.V. the commercial entity behind the GitLab project. The sponsorship was originally offered in 2015 by the GitLab CEO, presumably to counter a possible move to GitHub, as there was a discussion about creating a GitHub Organization for Debian at the time. The deployment of a Debian-specific GitLab instance then raised the question of the overlap with the already existing git.debian.org service, which is backed by Alioth's FusionForge deployment. It then seemed natural that the new GitLab instance would replace Alioth. But when Praveen directly proposed to move to GitLab, Wirt stepped in and explained that a migration plan was already in progress. The plan then was to migrate to a simpler gitolite-based setup, a decision that was apparently made in corridor discussions surrounding the Alioth Git replacement BoF held during Debconf 2015. The first objection raised by Wirt against GitLab was its "huge number of dependencies". Another issue Wirt identified was the "open core / enterprise model", preferring a "real open source system", an opinion which seems shared by other participants on the mailing list. Wirt backed his concerns with an hypothetical example:

Debian needs feature X but it is already in the enterprise version. We make a patch and, for commercial reasons, it never gets merged (they already sell it in the enterprise version). Which means we will have to fork the software and keep those patches forever. Been there done that. For me, that isn't acceptable.

This concern was further deepened when GitLab's Director of Strategic Partnerships, Eliran Mesika, explained the company's stewardship policy that explains how GitLab decides which features end up in the proprietary version. Praveen pointed out that:

[...] basically it boils down to features that they consider important for organizations with less than 100 developers may get accepted. I see that as a red flag for a big community like debian.

Since there are over 600 Debian Developers, the community seems to fall within the needs of "enterprise" users. The features the Debian community may need are, by definition, appropriate only to the "Enterprise Edition" (GitLab EE), the non-free version, and are therefore unlikely to end up in the "Community Edition" (GitLab CE), the free-software version. Interestingly, Mesika asked for clarification on which features were missing, explaining that GitLab is actually open to adding features to GitLab CE. The response from Debian Developer Holger Levsen was categorical: "It's not about a specific patch. Free GitLab and we can talk again." But beyond the practical and ethical concerns, some specific features Debian needs are currently only in GitLab EE. For example, debian.org systems use LDAP for authentication, which would obviously be useful in a GitLab deployment; GitLab CE supports basic LDAP authentication, but advanced features, like group or SSH-key synchronization, are only available in GitLab EE. Wirt also expressed concern about the Contributor License Agreement that GitLab B.V. requires contributors to sign when they send patches, which forces users to allow the release of their code under a non-free license. The debate then went on going through a exhaustive inventory of different free-software alternatives:

• GitLab, a Ruby-based GitHub replacement, dual-licensed MIT/Commercial
• Gogs, Go, MIT
• Gitblit, Java, Apache-licensed
• Kallithea, in Python, also supports Mercurial, GPLv3
• and finally, pagure, also written Python, GPLv2

A feature comparison between each project was created in the Debian wiki as well. In the end, however, Praveen gave up on replacing Alioth with GitLab because of the controversy and moved on to support the pagure migration, which resolved the discussion in July 2016. More recently, Wirt admitted in an IRC conversation that "on the technical side I like GitLab a lot more than pagure" and that "as a user, GitLab is much nicer than pagure and it has those nice CI [continuous integration] features". However, as he explained in his blog "GitLab is Opencore, [and] that it is not entirely opensource. I don't think we should use software licensed under such a model for one of our core services" which leaves pagure as the only stable candidate. Other candidates were excluded on technical grounds, according to Wirt: Gogs "doesn't scale well" and a quick security check didn't yield satisfactory results; "Gitblit is Java" and Kallithea doesn't have support for accessing repositories over SSH (although there is a pending pull request to add the feature). In an email interview, Sid Sijbrandij, CEO of GitLab, did say that "we want to make sure that our open source edition can be used by open source projects". He gave examples of features liberated following requests by the community, such as branded login pages for the VLC project and GitLab Pages after popular demand. He stressed that "There are no artificial limits in our open source edition and some organizations use it with more than 20.000 users." So if the concern of the Debian community is that features may be missing from GitLab CE, there is definitely an opening from GitLab to add those features. If, however, the concern is purely ethical, it's hard to see how an agreement could be reached. As Sijbrandij put it:

On the mailinglist it seemed that some Debian maintainers do not agree with our open core business model and demand that there is no proprietary version. We respect that position but we don't think we can compete with the purely proprietary software like GitHub with this model.

Working toward a pagure migration The issue of Alioth maintenance came up again last month when Boyuan Yang asked what would happen to Alioth when support for Debian LTS (Wheezy) ends next year. Wirt brought up the pagure migration proposal and the community tried to make a plan for the migration. One of the issues raised was the question of the non-Git repositories hosted on Alioth, as pagure, like GitLab, only supports Git. Indeed, Ben Hutchings calculated that while 90% (\~19,000) of the repositories currently on Alioth are Git, there are 2,400 SVN repositories and a handful of Mercurial, Bazaar (bzr), Darcs, Arch, and even CVS repositories. As part of an informal survey, however, most packaging teams explained they either had already migrated away from SVN to Git or were in the process of doing so. The largest CVS user, the web site team, also explained it was progressively migrating to Git. Mattia Rizzolo then proposed that older repository services like SVN could continue running even if FusionForge goes down, as FusionForge is, after all, just a web interface to manage those back-end services. Repository creation would be disabled, but older repositories would stay operational until they migrate to Git. This would, effectively, mean the end of non-Git repository support for new projects in the Debian community, at least officially. Another issue is the creation of a Debian package for pagure. Ironically, while Praveen and other Debian maintainers have been working for 5 years to package GitLab for Debian, pagure isn't packaged yet. Antonio Terceiro, another Debian Developer, explained this isn't actually a large problem for debian.org services: "note that DSA [Debian System Administrator team] does not need/want the service software itself packaged, only its dependencies". Indeed, for Debian-specific code bases like ci.debian.net or tracker.debian.org, it may not make sense to have the overhead of maintaining Debian packages since those tools have limited use outside of the Debian project directly. While Debian derivatives and other distributions could reuse them, what usually happens is that other distributions roll their own software, like Ubuntu did with the Launchpad project. Still, Paul Wise, a member of the DSA team, reasoned that it was better, in the long term, to have Debian packages for debian.org services:

Personally I'm leaning towards the feeling that all configuration, code and dependencies for Debian services should be packaged and subjected to the usual Debian QA activities but I acknowledge that the current archive setup (testing migration plus backporting etc) doesn't necessarily make this easy.

Wise did say that "DSA doesn't have any hard rules/policy written down, just evaluation on a case-by-case basis" which probably means that pagure packaging will not be a blocker for deployment. The last pending issue is the question of the mailing lists hosted on Alioth, as pagure doesn't offer mailing list management (nor does GitLab). In fact, there are three different mailing list services for the Debian project:

• the main service, lists.debian.org, running ~~Mailman 2~~ SmartList and managed by hand
• the Alioth service, lists.alioth.debian.org, running Mailman 2 and managed by FusionForge
• the Debconf service, lists.debconf.org, also running Mailman 2

Wirt, with his "list-master hat" on, explained that the main mailing list service is "not really suited as a self-service" and expressed concern at the idea of migrating the large number mailing lists hosted on Alioth. Indeed, there are around 1,400 lists on Alioth while the main service has a set of 300 lists selected by the list masters. No solution for those mailing lists was found at the time of this writing. In the end, it seems like the Debian project has chosen pagure, the simpler, less featureful, but also less controversial, solution and will use the same hosting software as their fellow Linux distribution, Fedora. Wirt is also considering using FreeIPA for account management on top of pagure. The plan is to migrate away from FusionForge one bit at a time, and pagure is the solution for the first step: the Git repositories. Lists, other repositories, and additional features of FusionForge will be dealt with later on, but Wirt expects a plan to come out of the upcoming sprint. It will also be interesting to see how the interoperability promises of pagure will play out in the Debian world. Even though the federation features of pagure are still at the early stages, one can already clone issues and pull requests as Git repositories, which allows for a crude federation mechanism. In any case, given the long history and the wide variety of workflows in the Debian project, it is unlikely that a single tool will solve all problems. Alioth itself has significant overlap with other Debian services; not only does it handle mailing lists and forums, but it also has its own issue tracker that overlaps with the Debian bug tracking system (BTS). This is just the way things are in Debian: it is an old project with lots of moving part. As Jonathan Dowland put it: "The nature of the project is loosely-coupled, some redundancy, lots of legacy cruft, and sadly more than one way to do it." Hopefully, pagure will not become part of that "legacy redundant cruft". But at this point, the focus is on keeping the services running in a simpler, more maintainable way. The discussions between Debian and GitLab are still going on as we speak, but given how controversial the "open core" model used by GitLab is for the Debian community, pagure does seem like a more logical alternative.

Note: this article first appeared in the Linux Weekly News.

In the near future various upstream projects related to the Ubuntu desktop experience as we have known it so far may become only sporadically maintained or even fully unmaintained. Ubuntu will switch to the Gnome desktop environment with 18.04 LTS as its default desktop, maybe even earlier. The Application Indicators [1] brought into being by Canonical Ltd. will not be needed in Gnome (AFAIK) any more. We can expect the Application Indicator related projects become unmaintained upstream. (In fact I have recently been offered continuation of upstream maintenance of libdbusmenu). Historical Background This all started at Ubuntu Developer Summit 2012 when Canonical Ltd. announced Ubuntu to become the successor of Windows XP in business offices. The Unity Greeter received an Remote Login Service enhancement: since then it supports Remote Login to Windows Terminal Servers. The question came up, why Remote Login to Linux servers--maybe even Ubuntu machines--is not on the agenda. It turned out, that it wasn't even a discussable topic. At that time, I started looking into the Unity Greeter code, adding support for X2Go Logon into Unity Greeter. I never really stopped looking at the greeter code from time to time. Since then, it turned into some sort of a hobby... While looking into the Unity Greeter code over the past years and actually forking Unity Greeter as Arctica Greeter [2] in September 2015, I also started looking into the Application Indicators concept just recently. And I must say, the more I have been looking into it, the more I have started liking the concept behind Application Indicators. The basic idea is awesome. However, lately all indicators became more and more Ubuntu-centric and IMHO too polluted by code related to the just declared dead Ubuntu phablet project. Forking Application Indicators Saying all this, I recently forked Application Indicators as Ayatana Indicators. At the moment I represent upstream and Debian package maintainer in one person. Ideally, this is only temporary and more people join in. (I heard some Unity 7 maintainers think about switching to Ayatana Indicators for the now community maintained Unity 7). The goal is to provide Ayatana Indicators to all desktop environments generically, all that want to use them, either as default or optionally. Release-wise, the idea is to strictly differentiate between upstream and Debian downstream in the release cycles of the various related components. I hope, noone is too concerned about the choice of name, as the "Ayatana" word actually was first used for upstream efforts inside Ubuntu [3]. Using the Ayatana term for the indicator forks is meant as honouring the previously undertaken efforts. I have seen very good work, so far, while going through the indicators' code. The upstream code must not be distro-specific, but, of course, can be distro-aware. Contributions Welcome The Ayatana Indicators upstream project components are currently hosted on Github under the umbrella of the Arctica Project. Regarding Debian, first uploads have recently been accepted to Debian experimental. The Debian packages are maintained under the umbrella of the revived Ayatana Packagers team [4]. Meet you at the Ayatana Indicators BoF at DebConf 17 (hopefully) For DebConf 17 (yeah, I am going there, if all plans work out well!!!!) I have submitted a BoF on this topic (let's hope, it gets accepted...). I'd like to give a quick overview on the current status of above named efforts and reasonings behind my commitment to the work. Most of the time during that BoF I would like to get into discussion with desktop maintainers, possibly upstream developers, Ubuntu developers, etc. Anyone who sees an asset in the Indicators approach is welcome to share and contribute. References

• [1] https://wiki.ubuntu.com/DesktopExperienceTeam/ApplicationIndicators
• [2] https://github.com/ArcticaProject/arctica-greeter
• [3] https://launchpad.net/ayatana
• [4] https://qa.debian.org/developer.php?login=pkg-ayatana-devel%40lists.alio...

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In May, about 182 work hours have been dispatched among 11 paid contributors. Their reports are available:

• Ben Hutchings did 13 hours (out of 15h allocated + 3 extra hours, thus keeping 5 extra hours for June).
• Brian May did 10 hours.
• Chris Lamb did 18 hours.
• Emilio Pozuelo Monfort did 23 hours (out of 24 hours allocated + 2 hours remaining, thus keeping 3 hours for June).
• Guido G nther did 8 hours.
• Hugo Lefeuvre did 15 hours.
• Jonas Meurer gave back his remaining hours from last month.
• Markus Koschany did 27.25 hours.
• Ola Lundqvist did 12 hours (out of 6h allocated + 6 remaining hours).
• Rapha l Hertzog did 12 hours.
• Roberto C. Sanchez did 22 hours (out of 20 hours allocated + 4.5 hour remaining, thus keeping 2.5 extra hours for June).
• Thorsten Alteholz did 27.25 hours.

Evolution of the situation The number of sponsored hours did not change and we are thus still a little behind our objective. The security tracker currently lists 44 packages with a known CVE and the dla-needed.txt file 42. The number of open issues is close to last month. Thanks to our sponsors New sponsors are in bold (none this month unfortunately).

• Platinum sponsors:
• TOSHIBA (for 20 months)
• GitHub (for 11 months)
• Gold sponsors:
• The Positive Internet (for 36 months)
• Blablacar (for 35 months)
• Linode (for 25 months)
• Babiel GmbH (for 14 months)
• Plat Home (for 14 months)
• Silver sponsors:
• Domeneshop AS (for 35 months)
• Universit Lille 3 (for 35 months)
• Trollweb Solutions (for 33 months)
• Nantes M tropole (for 29 months)
• Dalenys (for 26 months)
• Univention GmbH (for 21 months)
• Universit Jean Monnet de St Etienne (for 21 months)
• Sonus Networks (for 15 months)
• UR Communications BV (for 9 months)
• maxcluster GmbH (for 9 months)
• Exonet B.V. (for 5 months)
• Bronze sponsors:
• David Ayers IntarS Austria (for 36 months)
• Evolix (for 36 months)
• Offensive Security (for 36 months)
• Seznam.cz, a.s. (for 36 months)
• Freeside Internet Service (for 35 months)
• MyTux (for 35 months)
• Linuxhotel GmbH (for 33 months)
• Intevation GmbH (for 32 months)
• Daevel SARL (for 31 months)
• Bitfolk LTD (for 30 months)
• Megaspace Internet Services GmbH (for 30 months)
• Greenbone Networks GmbH (for 29 months)
• NUMLOG (for 29 months)
• WinGo AG (for 28 months)
• Ecole Centrale de Nantes LHEEA (for 25 months)
• Sig-I/O (for 22 months)
• Entr ouvert (for 20 months)
• Adfinis SyGroup AG (for 17 months)
• Laboratoire LEGI UMR 5519 / CNRS (for 12 months)
• Quarantainenet BV (for 12 months)
• GNI MEDIA (for 11 months)
• RHX Srl (for 9 months)
• Bearstech (for 3 months)
• LiHAS (for 3 months)

No comment Liked this article? Click here. My blog is Flattr-enabled.

Some days ago, I blogged asking for pointers to get started with the translation of Made with Creative Commons. Thank you all for your pointers and ideas! To the people that answered via private mail, via IRC, via comments on the blog. We have made quite a bit of progress so far; I want to test some things before actually sending a call for help. What do we have?

Git repository set up

I had already set up a repository at GitLab; right now, the contents are far from useful, they merely document what I have done so far. I have started talking with my Costa Rican friend Leo Arias, who is also interested in putting some muscle behind this translation, and we are both the admins to this project.

Talked with the authors

Sarah is quite enthusiastic about us making this! I asked her to hold a little bit before officially announcing there is work ongoing... I want to get bits of infrastructure ironed out first. Important Talking with her, she discussed the tools they used for authoring the book. It made me less of a purist :) Instead of starting from something "pristine", our master source will be the PDF export of the Google Docs document.

Markdown conversion

Given that translation tools work over the bits of plaintext, we want to work with the "plainest" rendition of the document, which is Markdown. I found that Pandoc does a very good approximation to what we need (that is, introduces very little "ugly" markup elements). Converting the ODT into Markdown is as easy as:
$ pandoc -f odt MadewithCreativeCommonsmostup-to-dateversion.odt -t markdown > MadewithCreativeCommonsmostup-to-dateversion.md
Of course, I want to fine-tune this as much as possible.

Producing a translatable .po file

I have used Gettext to translate user interfaces; it is a tool very well crafted for that task. Translating a book is quite different: How and where does it break and join? How are paragraphs "strung" together into chapters, parts, a book? That's a task for PO 4 Anything (po4a). As simple as this:
po4a-gettextize -f text -m MadewithCreativeCommonsmostup-to-dateversion.md -p MadewithCreativeCommonsmostup-to-dateversion.po -M utf-8
I tested the resulting file with my good ol' trusty poedit, and it works... Very nicely!

What is left to do?

• I made an account and asked for hosting at Weblate. I have not discussed this with Leo, so I hope he will agree ;-) Weblate is a Web-based infrastructure for collaborative text translation, provided by Debian's Michal iha . It integrates nicely with version control systems, preserves credit for each translated string (and I understand, but might be mistaken, that it understands the role of "editors", so that Leo and I will be able to do QA on the translation done by whoever joins us, trying to have a homogeneous-sounding result. I hope the project is approved for Weblate soon!
• Work on reconstructing the book. One thing is to deconstruct, find paragraphs, turn them into translatable strings... And a very different one is to build a book again from there! I have talked with some people to help me get this in shape. It is basically just configuring Pandoc But as I have never done that, any help will be most, most welcome!
• Setting translation policies. What kind of language will we use? How will we refer to English names and terms? All that important stuff to give proper quality to our work
• Of course, the long work itself: Performing the translations

I recently finished a paper that presents a novel social computing system called the Wikipedia Adventure. The system was a gamified tutorial for new Wikipedia editors. Working with the tutorial creators, we conducted both a survey of its users and a randomized field experiment testing its effectiveness in encouraging subsequent contributions. We found that although users loved it, it did not affect subsequent participation rates.

Start screen for the Wikipedia Adventure.

A major concern that many online communities face is how to attract and retain new contributors. Despite it s success, Wikipedia is no different. In fact, researchers have shown that after experiencing a massive initial surge in activity, the number of active editors on Wikipedia has been in slow decline since 2007.

The number of active, registered editors ( 5 edits per month) to Wikipedia over time. From Halfaker, Geiger, and Morgan 2012.

Research has attributed a large part of this decline to the hostile environment that newcomers experience when begin contributing. New editors often attempt to make contributions which are subsequently reverted by more experienced editors for not following Wikipedia s increasingly long list of rules and guidelines for effective participation. This problem has led many researchers and Wikipedians to wonder how to more effectively onboard newcomers to the community. How do you ensure that new editors Wikipedia quickly gain the knowledge they need in order to make contributions that are in line with community norms? To this end, Jake Orlowitz and Jonathan Morgan from the Wikimedia Foundation worked with a team of Wikipedians to create a structured, interactive tutorial called The Wikipedia Adventure. The idea behind this system was that new editors would be invited to use it shortly after creating a new account on Wikipedia, and it would provide a step-by-step overview of the basics of editing.

The Wikipedia Adventure was designed to address issues that new editors frequently encountered while learning how to contribute to Wikipedia. It is structured into different missions that guide users through various aspects of participation on Wikipedia, including how to communicate with other editors, how to cite sources, and how to ensure that edits present a neutral point of view. The sequence of the missions gives newbies an overview of what they need to know instead of having to figure everything out themselves. Additionally, the theme and tone of the tutorial sought to engage new users, rather than just redirecting them to the troves of policy pages. Those who play the tutorial receive automated badges on their user page for every mission they complete. This signals to veteran editors that the user is acting in good-faith by attempting to learn the norms of Wikipedia.

An example of a badge that a user receives after demonstrating the skills to communicate with other users on Wikipedia.

Once the system was built, we were interested in knowing whether people enjoyed using it and found it helpful. So we conducted a survey asking editors who played the Wikipedia Adventure a number of questions about its design and educational effectiveness. Overall, we found that users had a very favorable opinion of the system and found it useful.

Survey responses about how users felt about TWA.

Survey responses about what users learned through TWA.

We were heartened by these results. We d sought to build an orientation system that was engaging and educational, and our survey responses suggested that we succeeded on that front. This led us to ask the question could an intervention like the Wikipedia Adventure help reverse the trend of a declining editor base on Wikipedia? In particular, would exposing new editors to the Wikipedia Adventure lead them to make more contributions to the community? To find out, we conducted a field experiment on a population of new editors on Wikipedia. We identified 1,967 newly created accounts that passed a basic test of making good-faith edits. We then randomly invited 1,751 of these users via their talk page to play the Wikipedia Adventure. The rest were sent no invitation. Out of those who were invited, 386 completed at least some portion of the tutorial. We were interested in knowing whether those we invited to play the tutorial (our treatment group) and those we didn t (our control group) contributed differently in the first six months after they created accounts on Wikipedia. Specifically, we wanted to know whether there was a difference in the total number of edits they made to Wikipedia, the number of edits they made to talk pages, and the average quality of their edits as measured by content persistence. We conducted two kinds of analyses on our dataset. First, we estimated the effect of inviting users to play the Wikipedia Adventure on our three outcomes of interest. Second, we estimated the effect of playing the Wikipedia Adventure, conditional on having been invited to do so, on those same outcomes. To our surprise, we found that in both cases there were no significant effects on any of the outcomes of interest. Being invited to play the Wikipedia Adventure therefore had no effect on new users volume of participation either on Wikipedia in general, or on talk pages specifically, nor did it have any effect on the average quality of edits made by the users in our study. Despite the very positive feedback that the system received in the survey evaluation stage, it did not produce a significant change in newcomer contribution behavior. We concluded that the system by itself could not reverse the trend of newcomer attrition on Wikipedia. Why would a system that was received so positively ultimately produce no aggregate effect on newcomer participation? We ve identified a few possible reasons. One is that perhaps a tutorial by itself would not be sufficient to counter hostile behavior that newcomers might experience from experienced editors. Indeed, the friendly, welcoming tone of the Wikipedia Adventure might contrast with strongly worded messages that new editors receive from veteran editors or bots. Another explanation might be that users enjoyed playing the Wikipedia Adventure, but did not enjoy editing Wikipedia. After all, the two activities draw on different kinds of motivations. Finally, the system required new users to choose to play the tutorial. Maybe people who chose to play would have gone on to edit in similar ways without the tutorial. Ultimately, this work shows us the importance of testing systems outside of lab studies. The Wikipedia Adventure was built by community members to address known gaps in the onboarding process, and our survey showed that users responded well to its design. While it would have been easy to declare victory at that stage, the field deployment study painted a different picture. Systems like the Wikipedia Adventure may inform the design of future orientation systems. That said, more profound changes to the interface or modes of interaction between editors might also be needed to increase contributions from newcomers.

This blog post, and the open access paper that it describes, is a collaborative project with Sneha Narayan, Jake Orlowitz, Jonathan Morgan, and Aaron Shaw. Financial support came from the US National Science Foundation (grants IIS-1617129 and IIS-1617468), Northwestern University, and the University of Washington. We also published all the data and code necessary to reproduce our analysis in a repository in the Harvard Dataverse. Sneha posted the material in this blog post over on the Community Data Science Collective Blog.

I stumbled across some information about pathological behaviour of Western Digital Green (and some Red) hard drives relating to drive-head parking when the device is idle. In some circumstances, a particular pattern of drive activity can result in the drive head being repeatedly parked and un-parked in short intervals, possibly* resulting in excess wear on the drive. Apparently* the drive head parking is recorded in the S.M.A.R.T. "Load Cycle Count" attribute. I have two WD Red drives in my NAS, one for live data and one for backup. The latter drive is basically unused most of the day until scheduled backup jobs kick in and those jobs are all clustered together. I already unmount the backup filesystems when the jobs are not active (I wrote about this in mount-on-demand backups). Inspecting the S.M.A.R.T. attributes was surprising:

drive	power on hours	load cycle count
regular	12143	348
backup	12191	13043

It certainly looks like my backup drive has a much higher load cycle count than you might expect for a mostly-idle drive. I checked the attributes again 24 hours later and the regular drive had incremented by a single cycle, whilst the backup drive went up by 56. There are some official tools from Western Digital that makes an adjustment to the idle timeouts for head parking on the drives. There's also an unofficial tool idle3tcl to do the same, which is packaged in Debian. The unofficial tool let you set and fetch a particular value from the drive firmware. I don't know for sure* that the official tool does exactly the same thing, and nothing else. One advantage of the unofficial tool is it lets you read the value as well as write it. I tried the unofficial tool to get the drive's default value, which was 0x8a, and bump it to the maximum of 0xff. I then tried the official tool then fetched the value again: interestingly the official tool had reset the value back to 0x8a. I haven't managed to assess the impact of these changes on the attrition rate yet because I need to perform a cold boot for the change to take effect and that isn't convenient just now. My plan is to try and disable the feature completely via the unofficial tool. If that rectifies the issue I will then investigate changing the power management settings by hand at backup start/end time, perhaps via hdparm. ( The problem with these kind of issues is there is precious little in the way of reliable documentation as to the real issue, real drive behaviour, etc. I've marked a few sections of this blog post with * asterisks to indicate where we are having to make informed guesses. )

A new version of the anytime package is now on CRAN. It marks the eleventh release since the inaugural version late last summer. anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to either POSIXct or Date objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples. This release brings a little more consistency to how numeric or integer arguments are handled. Previously, we were overly eager in accepting something such as 20150605 (i.e. today) as a (numerical or integer) input to both anytime() and anydate(). That is well-intentioned, but ultimately foolish. We relied on heuristic cutoffs to determine whether input was "meant to be" a date or time offset. There lies madness. We now differentiate whether we were called via anytime() (in which case numerical data is second offset to the epoch, just as.POSICct()) or anytime() (in which case it is days offset to the (date) epoch, just like as.Date()). The previous behaviour can be restored via a options, both function-local as well as global are supported. And of course, there is no change for all other (and more common) input formats, notably character or factor. A full list of changes follows.

Changes in anytime version 0.3.0 (2017-06-05)

• Numeric input is now always an offset to epoch, with anytime() using seconds, and anydate() using dates. (#65 fixing #63).
• Old behaviour can be re-enabled with an option also supporting a global setting getOption("anytimeOldHeuristic")
• RStudio versions 1.1.129 or later can run all functions without fear of crashing due to a change in their use of Boost.
• Replaced init.c with registration code inside of RcppExports.cpp thanks to Rcpp 0.12.11.

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

TeX Live 2017 has been released! CTAN mirrors are busy updating. Get out bottles of good wine, and enjoy a good long download
Besides the huge amount of package updates due to the 2 month testing hiatus, I want to pick a few changes before copying the complete changelog entry: Additional TEXMF trees tlmgr got a new functionality to easily add and remove additional TEXMF trees to the search path. MikTeX had this features since long, and it was often requested. As it turned out, it is a rather trivial thing to achieve by some texmf.cnf lines. The rest is just front end in tlmgr. Here a few invocations (not very intelligent usage, though):

$ tlmgr conf auxtrees show
tlmgr: no auxiliary texmf trees defined.
$ tlmgr conf auxtrees add /projects/book-abc
$ tlmgr conf auxtrees
List of auxiliary texmf trees:
  /projects/book-abc
$ tlmgr conf auxtrees remove /projects/book-abc
$ tlmgr conf auxtrees show
tlmgr: no auxiliary texmf trees defined.

TeX Live Manager interactive shell tlmgr also got an interactive shell now, that can also be used for scripting. Available commands are all the usual command line actions, plus a few more (see documentation). Again, here a simple session:

$ tlmgr shell
protocol 1
tlmgr> load local
OK
tlmgr> load remote
tlmgr: package repositories
	main = /home/norbert/public_html/tlnet (verified)
	koma = http://www.komascript.de/repository/texlive/2017 (verified)
OK
tlmgr> update --list
tlmgr: package repositories
	main = /home/norbert/public_html/tlnet (verified)
	koma = http://www.komascript.de/repository/texlive/2017 (verified)
tlmgr: saving backups to /home/norbert/tl/2017/tlpkg/backups
tlmgr: no updates available
OK
tlmgr> byebye
$

User versus System mode for updmap and fmtutil I have reported on this during BachoTeX/TUG 2017, here are the slides, that the two central configuration programs updmap and fmtutil will change their operation mode slightly by disallowing invocations without mde specification. That is, one either needs to call updmap -sys (or updmap-sys, nothing changed here from previous years) or updmap -user (or updmap-user, new in TeX Live 2017). We hope by this and the accompanying web page of recommendations to help users not to shoot themselves to often by calling updmap without knowing the consequences. The upcoming proceedings of the BachoTeX will also comtain an article fully documenting both updmap and fmtutil including the most recent changes.

---

These were only a few changes where I had my fingers in the development. For the full list, read on below. Now get ready for partying, and also for the end of the peace, because daily package updates will restart in the next days. Enjoy Changes since TeX Live 2016 The following list of changes is directly from the TeX Live documentation.

• LuaTEX: More callbacks, more typesetting control, more access to internals; ffi library for dynamic code loading added on some platforms.
• pdfTEX: Environment variable SOURCE_DATE_EPOCH_TEX_PRIMITIVES from last year renamed to FORCE_SOURCE_DATE, with no changes in functionality; if the \pdfpageattr token list contains the string /MediaBox, omit output of the default /MediaBox.
• XeTEX: Unicode/OpenType math now based on HarfBuzz s MATH table support; some bug fixes.
• Dvips: Make the last papersize special win, for consistency with dvipdfmx and package expectations; the -L0 option (L0 config setting) restores the previous behavior of the first special winning.
• epTEX, eupTEX: New primitives \pdfuniformdeviate, \pdfnormaldeviate, \pdfrandomseed, \pdfsetrandomseed, \pdfelapsedtime, \pdfresettimer, from pdfTEX.
• MacTEX: As of this year, only Mac OS X releases for which Apple still releases security patches will be supported in MacTEX, under the platform name x86_64-darwin; currently this means Yosemite, El Capitan, and Sierra (10.10 and newer). Binaries for older Mac OS X versions are not included in MacTEX, but are still available in TEX Live (x86_64-darwinlegacy, i386-darwin, powerpc-darwin).
• Infrastructure: The TEXMFLOCAL tree is now searched before TEXMFSYSCONFIG and TEXMFSYSVAR (by default); the hope is that this will better match expectations of local files overriding system files. Also, tlmgr has a new mode shell for interactive and scripted use, and a new action conf auxtrees to easily add and remove extra trees.
• updmap and fmtutil: These scripts now give a warning when invoked without explicitly specifying either so-called system mode (updmap-sys, fmtutil-sys, or option -sys), or user mode (updmap-user, fmtutil-user, or option -user). The hope is that this will reduce the perennial problem of invoking user mode by accident and thus losing future system updates. See http://tug.org/texlive/scripts-sys-user.html and an upcoming TUGboat article for details.
• install-tl: Personal paths are now set to MacTEX values (~/Library/...) by default on Macs. New option -init-from-profile to start an installation with the values from a given profile; new command P to explicitly save a profile; new profile variable names (but previous ones are still accepted).
• SyncTEX: the temporary file name now looks like foo.synctex(busy), instead of foo.synctex.gz(busy) (no .gz). Front ends and build systems that want to remove temp files may need adjusting.
• Other utilities: texosquery-jre8 is a new cross-platform program for retrieving locale and other OS information from a TEX document; it is included in shell_escape_commands by default for restricted shell execution. (Older JRE versions are supported by texosquery, but cannot be enabled in restricted mode, as they are no longer supported by Oracle, even for security issues.)
• Platforms: See MacTEX entry above; no other changes.

Here is my monthly update covering what I have been doing in the free software world (previous month):

• Wrote and released installation-birthday. Installing this package will celebrate the anniversary of installing your system by sending you an email via cron(8).
• Fixed an issue in the Django web development framework where you couldn't run the testsuite against a read-only copy of the source code. This was found by the Debian Continuous Integration service. (#26755)
• Provided a pull request for the "wammu" mobile phone manager to ensure the build is reproducible. (#49)

---