Search Results: "ema"

4 March 2016

Daniel Pocock: Machine Learning for Hackers with Debian and Ubuntu

Data Science and Machine Learning are hot topics at the moment. Many people are considering how to extend their skills into these areas and many solutions have appeared, including full online degrees, free online courses combined with free software and for those who prefer hard copy, a staggering choice of books on the topic. One of those books is O'Reilly's Machine Learning for Hackers by John Myles White and Drew Conway. The book uses R to demonstrate a series of techniques for analysis and prediction. The book offers a great opportunity to simultaneously get an introduction to basic machine learning techniques and also an introduction to the increasingly popular R platform. On page 11 they list all the major R packages needed to run their examples (available on Github). I had a look over this list to see how many could be installed on a Debian system using apt-get and found that about half of them were already present. Five of them and one dependency, however, were not already available so I've whipped up packages for them and they are now in jessie-backports for all users of the current stable release. If you are following the exercises in this book, you can get all the software you need with one convenient command: 
$ sudo apt-get install -t jessie-backports \
  r-cran-ggplot2 r-cran-lme4 r-cran-rcurl \
  r-cran-reshape r-cran-xml r-cran-arm \
  r-cran-glmnet r-cran-igraph r-cran-lubridate \
  r-cran-rjsonio r-cran-tm
Thanks to all those who already packaged other parts of R and backported the relevant packages. Note that the RJSONIO package's authors have not provided a valid free software license so it is in non-free. It is there to support people using the book but I would encourage people to use RJSON for any new projects as it does have a valid license.

Enrico Zini: Praise of component reuse

I farm bits and pieces out to the guys who are much more brilliant than I am. I say, "build me a laser", this. "Design me a molecular analyzer", that. They do, and I just stick 'em together. (Seth Brundle, "The Fly")
When I decided to try and turn siterefactor into staticsite, I decided that I would go ahead only for as long as it could be done with minimal work, writing code in the most straightforward way on top of existing and stable components. I am pleased by how far that went. Python-Markdown It works fast enough, already comes with extensions for most of what I needed, and can be extended in several ways. One of the extension methods is a hook for manipulating the ElementTree of the rendered document before serializing it to HTML, which made it really easy to go and process internal links in all <a href= and <img src= attributes. To tell an internal link from an external link I just use the standard python urlparse and see if the link has a scheme or a netloc component. If it does not, and if it has a path, then it is an internal link. This also means that I do not need to invent new Markdown syntax for internal references, avoiding the need for remembering things like [text]( < relref "blog/post.md" > ) or [text]( filename /blog/post.md). In staticsite, it's just [text](/blog/post.md) or [text](post.md) if the post is nearby. This feels nicely clean to me: if I wanted to implement fancy markdown features, I could do it as Python-Markdown extensions and submit them upstream. If I wanted to implement fancy interlinking features, I could do it with a special url scheme in links. For example, it would be straigtforward to implement a ssite: url scheme that expanded the url with elements from staticsite's settings using a call to python's string.format (ssite: SETTING_NAME /bar maybe?), except I do not currently see any use cases for extending internal linking from what it is now. Jinja2 Jina2 is a template engine that I already knew, it is widely used, powerful and pleasant to use, both on the templating side and on the API's side. It is not HTML specific, so I can also use it to generate Atom, RSS2, "dynamic" site content, and even new site Markdown pages. Implementing RSS and Atom feeds was just a matter of writing and testing these Jinja2 macros and then reusing them anywhere. toml, yaml, json No need to implement my own front matter parsing. Also, reusing the same syntax as Hugo allows me to just link to its documentation. python-slugify I found python-slugify so I did not bother writing a slug-generating function. As a side effect, now things works better than I would even have thought to implement, including transliteration of non-ascii characters: 
$ ./ssite new example --noedit --title "Cos  parl  Enrico"
/enrico-dev/staticsite/example/site/blog/2016/cosi-parlo-enrico.md
(I just filed an RFP) python-livereload Implementing ssite serve which monitors the file system and autoreloads when content changes and renders everything on the fly, took about an hour. Most of that hour went into implementing rendering pages on demand. Then I discovered that it autoreloads even when I edit staticsite's source code. Then I discovered that it communicates with the browser and even automatically triggers a page refresh. I can keep vim on half my screen and a browser in the other half, and I get live preview for free every time I save, without ever leaving the editor. Bootstrap I already use Bootstrap at work, so creating the default theme templates with it took about 10 minutes. This morning I tried looking at my website using my mobile phone, and I pleasantly saw it automatically turning into a working mobile version of itself. Pygments Python-Markdown uses Pygments for syntax highlighting, and it can be themed just by loading a .css. So, without me really doing anything, even staticsite's syntax highligthing is themable, and there's even a nice page with a list of themes to choose from. Everything else... Command line parsing? Straight argparse. Logging? python's logging support. Copying static resource files? shutil.copy2. Parsing dates? dateutil.parser. Timing execution? time.perf_counter. Timezone handling? pytz. Building the command to run an editor? string.format. Matching site pages? fnmatch.translate. ...and then some. If I ever decide to implement incremental rendering, how do I implement tracking which source files have changed? Well, for example, how about just asking git?

Arturo Borrero Gonz lez: How are we training to climb Mulhacen

Alcazaba, Punta de la Cornisa and Mulhac n 3.479 m

I have been in some adventures with a group of friends for a couple of years now. We call ourselves Los Extraviaos (a quick translation to english is perhaps The Wanderers).

We like nature adventures. During summer we use to do water adventures (like rivers trekking, waterfall climbing, snorkeling, canyoning, rafting, kayaking and so on). During winter and spring, we use to do trekking, hiking and mountaineering (low altitudes, keep reading).

For this sport season, one of our main milestones is to climb the second most prominent mountain in Spain: Mulhacen, with 3.479 m over sea level, during summer (June).
From an alpinist point of view, this isn't a very big deal, since most of the mountain's snow and ice is completely melted and the altitude is not that high.
However, from our amateur mountaineering point of view we will be taking a lot of kilometres to walk and a lot of elevation gain uphill (almost 1.600 m), which is in fact a thing to take into account.
Anyway, Mulhacen is considered 'high mountaineering' and all cautions should be taken both during preparation/training and during the climbing of the mountain itself.

To train our bodies and minds we are climbing lower mountains. We have climbed 3 or 4 already, and will continue to climb more in upcoming moths until we go to Mulhacen.
Also, it's important to climb mountains in different weather conditions. We need to train how to overcome wind, snow, storm, cold, night situations. The way we learn most is facing new, challenging situations.

We started by going to the summit of 'El Aljibe', 1.094 m over sea level, with about 500 m elevation gain uphill during 13 km.
This time, the weather was very windy. A video of the summit moment:


A pic of the group:


Next was the 'Torrecilla' peak. The summit is at 1.919 m over sea level, with about 900 m elevation gain uphill during 15 km.
This time we faced a snowy trekking.
Here a music video of that day:


The Torrecilla peak is in fact challenging with snow, so we went a second time, with even more snow and wind:


This second time we faced winds of 70km/h which, due to wind-chill, resulted in a very cold day.

We also went to Sierra de Grazalema Natural Park, in a hard trekking under snowfall:


This time we walked about 15 km with about 500m elevation gain uphill, while enjoying this kind of environment:


What's next: more mountains! We plan to reach 2.069 m over sea level next week in 'La Maroma' and keep doing steep trekking in upcoming months until we go in June to Mulhacen.

It should be very funny to do some FLOSS contribution from the summit of a mountain. Do you know of someone who did that?

3 March 2016

Simon Richter: Why sysvinit?

As most distributions have switched to systemd as default "init", some people have asked why we actually keep sysvinit around, as it's old and crusty, and systemd can do so much more. My answer is: systemd and sysvinit solve entirely different problems, and neither can ever replace the other fully. The big argument in favour of systemd is integration: when you keep everything in one process, it is easy to make sure that components talk to each other. That is nice, but kind of misses the point: integration could be done via IPC as well, and systemd uses a lot of it to separate privileges anyway. The tighter integration comes from something else: the configuration format. Where sysvinit uses a lot of shell scripts ("imperative" format), systemd uses explicit keywords ("descriptive" format), and this is the important design decision here: descriptive formats cannot do anything unanticipated, neither good nor bad. This is important for UI programmers: if there is a button that says "if I close the lid switch, suspend the laptop", then this needs to be connected to functionality that behaves this way, and this connection needs to work in both directions, so the user will find the active setting there when opening the dialog. So, we need to limit ourselves to a closed set of choices, so the UI people can prepare translations. This is the design tradeoff systemd makes: better integration in exchange for less flexibility. Of course, it is simple to allow "custom" actions everywhere, but that starts to break the "well-integrated" behaviour. There is an old blog post from Joey Hess about how to implement a simple alarm clock, using systemd, and this shows both how powerful and how limited the system is. On one hand, the single line WakeSystem=true takes care of the entire dependency chain attached to making sure the system is actually on -- when the system is going to suspend mode, some program needs to determine that there are scheduled jobs that should wake up the system, determined which of these is the next one, program the wakeup time into the hardware and only then allow the suspend to continue. On the other hand, the descriptive framework already breaks down in this simple example, because the default behaviour of the system is to go to sleep if the lid switch is closed, and the easiest way to fix this is to tell systemd to ignore the lid switch while the program is running. The IPC permissions disallow scheduled user jobs from setting this, so a "custom" action is taken that sets up the "ignore the lid switch" policy, changes to an unprivileged user, and runs the actual job. So, it is possible to get back flexibility, but this is traded for integration: while the job is running, any functionality that the "suspend laptop when the lid is closed" button had, is broken. For something simple as a switch on a personal laptop that breaks if the user configures something that is non-standard, that is acceptable. However, the keywords in the service unit files, the timer files, etc. are also part of a descriptive framework, and the services using them expect that the things they promise are kept by the framework, so there is a limit to how many exceptions you can grant. SystemV init really comes from the other direction here. Nothing is promised to init scripts but a vague notion that they will be run with sufficient privileges. There is no dependency tracking that ensures that certain services will always start before others, just priorities. You can add a dependency system like insserv if you like and make sure that all the requirements for it to work (i.e. services declaring their dependencies) are given, and there is a safe fallback of not parallelizing so we can always play it safe. Because there is so little promised to scripts, writing them becomes a bit tedious -- all of them have a large case statement where they parse the command line, all of them need to reimplement dropping privileges and finding the PID of a running process. There are lots of helpers for the more common things, like the LSB standard functions for colorful and matching console output, and Debian's start-stop-daemon for privilege and PID file handling, but in the end it remains each script's responsibility. Systemd will never be able to provide the same flexibility to admins while at the same time keeping the promises made in the descriptive language, and sysvinit will never reach the same amount of easy integration. I think it is fairly safe to predict both of these things, and I'd even go a step further: if any one of them tried, I'd ask the project managers what they were thinking. The only way to keep systems as complex as these running is to limit the scope of the project. With systemd, the complexity is kept in the internals, and it is important for manageability that all possible interactions are well-defined. That is a huge issue when writing adventure games, where you need to define interaction rules for each player action in combination with each object, and each pair of objects that can be combined with the "Use" action. Where adventure games would default to "This doesn't seem to work." or "I cannot use this.", this would not be a helpful error message when we're trying to boot, so we really need to cut down on actions and object types here. Sysvinit, on the other hand, is both more extreme in limiting the scope to very few actions (restart, wait, once) and only one object type (process that can be started and may occasionally terminate) in the main program, and a lot more open in what scripts outside the main program are allowed to do. The prime example for this is that the majority of configuration takes place outside the inittab even -- a script is responsible for the interface that services are registered by dropping init scripts into specific directories, and scripts can also create additional interfaces, but none of this is intrinsic to the init system itself. Which system is the right choice for your system is just that: your choice. You would choose a starting point, and customize what is necessary for your computer to do what you want it to do. Here's the thing: most users will be entirely happy with fully uncustomized systemd. It will suspend your laptop if you close the lid, and even give your download manager veto power. I fully support Debian's decision to use systemd as the default init for new installs, because it makes sense to use the default that is good for the vast majority of users. However, my opening statement still stands: systemd can never support all use cases, because it is constrained by its design. Trying to change that would turn it into an unmaintainable mess, and the authors are right in rejecting this. There are use cases that define the project, and use cases that are out of scope. This is what we need sysvinit for: the bits that the systemd project managers rightfully reject, but that are someone's use case. Telling these users to get lost would be extremely bad style for the "universal operating system." For example, if someone needs a service that asks a database server for a list of virtual machines to start, runs each in its private network namespace that is named after replacing part of the virtual machine name with the last two octets of the host machine's IP address, then binds a virtual Ethernet device into the network namespace and sets up a NAT rule that allows devices to talk to the public Internet and a firewall rule to stop VMs from talking to each other. Such a beast would live outside of systemd's world view. You can easily start it, but systemd would not know how to monitor it (as long as there is some process still running, is that a good sign), not know how to shut down one instance, not know how to map processes to instances and so on. SystemV init has the advantage here that none of these things are defined by the system, and as long as my init script has some code to handle "status", I can easily check up on my processes with the same interface I'd use for everything else. This requires me to have more knowledge of how the init system works, that is true, however I'd also venture that the learning curve for sysvinit is shallower. If you know shell scripting, you can understand what init scripts do, without memorizing special rules that govern how certain keywords interact, and keeping track of changes to these rules as the feature set is expanded. That said, I'm looking at becoming the new maintainer of the sysvinit package so we can continue to ship it and give our users a choice, but I'd definitely need help, as I only have a few systems left that won't run properly with systemd because of its nasty habit to keep lots of code in memory where sysvinit would unload it after use (by letting the shell process end), and these don't make good test systems. The overall plan is to set up a CI system that Ideally, I'd like to reach a state where we can actually ensure that the default configuration is somewhat similar regardless of which init system we use, and that it is possible to install with either (this is a nasty regression in jessie: bootstrapping a foreign-architecture system broke). This will take time, and step one is probably looking at bugs first -- I'm grateful for any help I can get here, primarily I'd like to hear about regressions via bug reports. tl;dr: as a distribution, we need both systemd and sysvinit. If you suggest that systemd can replace sysvinit, I will probably understand that you think that the systemd maintainers don't know project management. I'm reluctantly looking at becoming the official maintainer because I think it is important to keep around.

Simon Richter: UI Design is Hard, Let's Go Shopping

As I have a few hours of downtime on the train, I'm working on one of my favourite KiCad features again: the pin table. This silently appeared in the last KiCad release -- when you have a schematic component open in the editor, there is a new icon, currently on the right of the toolbar, that brings up a table of all pins. Currently, I have the pin number, name and type in the table (as well as the coordinates on the grid, but that's probably not that useful). For the release, this table is read-only, but can at least be useful for proofreading: left mouse click on a column header sorts (as would be expected), and right mouse click on a column header creates a grouping (i.e. all pins that have the same value in this column are collapsed into a single row). The latter is entirely unintuitive, but I have no idea how to show the user that this option exists at all. The good old DirectoryOpus on Amiga had a convention that buttons that had right-click functionality had an earmark to indicate that a "backside" exists, but that is a fairly obscure UI pattern, and would be hard to reproduce across all platforms. I could make the right click drop down a menu instead of immediately selecting this column for grouping -- at least this way it would be explained what is happening, but advanced users may get annoyed at the extra step. I could add a status bar with a hint, but that will always cost me screen space, and this only papers over the issue. I could add a menu with sorting/grouping options, which would allow novice and intermediate users to find the function, but that still does not hint at the shortcut, and the functions to set the sorting would have to be disabled on Linux because wxGTK cannot set the sort column from a program. For editing, another problem turns up in the collapsed rows: what should editing do when we are basically editing multiple pins at once. The most useful mapping would be: However, I'd like to later on introduce multiple-function pins, which would add multiple (partially) exclusive pin names per pin number, which would change the semantics: At the moment, I'm mostly inclined to have separate modes for "grouped by pin number" and "grouped by pin name" -- these modes could be switched by the current grouping column, but that is neither intuitive, nor does it provide a full solution (because grouping by pin type is also supported). Time to look at the use cases: At 32C3, I've implemented a quick completeness check, consisting of a summary line that collapses the pin number column into a comma-separated list of ranges (where a range is defined as consecutive numbers in the last group of digits). I wonder if there is a group of developers with an UI focus who are happy about solving problems like these...

Antonio Terceiro: Debian Ruby Sprint 2016 - day 3: Ruby 2.3 in unstable, Reproducible Builds, and Data Structures for Dinner Booths

Day 3 was again a full of useful work. Since the beginning of the sprint, we were able to fix more than 50 FTBFS bugs, alongside general quality improvements in the packages.
in the Debian jargon, FTBFS means that a package fails to build from source , which in Debian is a critical bug because users need to be able to produce binary packages from their source code to fully exercise the free software principles.
An important milestone that was also achieved on day 3 was the upload of ruby-defaults 1:2.3.0+1, making ruby2.3 the new default version of Ruby. That is the version that will shipped in the next Debian release, codenamed stretch. This is the culmination of a joint effort between the Ruby team and Debian Release Team that involves rebuilding a little more than 130 packages that use the Ruby C API to make sure everything will just work on upgrades, both from the previous stable release, and from earlier snapshots of the current development release. Another small change that will have a big impact for Debian and for free software was an improvement to gem2deb that fixes a reproducibility issue in Ruby packages and will help currently more than 100 Ruby packages become reproducible. The full list of items that have been worked on is this: The day ended at Outback, where we had an amount of beer that led us to formulate what we will now call the One-Sided Dinner Booth Problem. In a party arranged like above, when the people closest to wall need to go alleviate themselves of some beer, you basically have to perform a removal from the bottom of a stack, which requires popping all the elements at the top. When they come back, you have to options: The One-Sided Dinner Booth Problem is finding the optimal data structure and algorithm for this situation. It is postulated that this is an NP-complete problem, and that only probabilistic solutions are cost-effective.

Ben Hutchings: Last week 'flu by

Last Monday, I began a new project at Codethink - working on kernel support for the next generation of an interactive system fitted in some high-performance cars. I travelled up to Manchester the night before, and on Monday I met Codethink's team (most of whom I was familiar with). My first chore was to set up VPN access to the development resources (source control, wiki, etc.). I sandboxed the proprietary VPN client in a VM with a systemd unit to run it at boot, so I can control it by starting and stopping that VM. I then set to work on unpacking and exploring the SoC vendor's evaluation module (EVM), starting by looking at serial output - of which there was none. Nothing on the LCD panel or network port either. A frustrating day. On Tuesday, the architect from our immediate customer, already a friend of mine, joined us in Manchester. As for the EVM, without any obvious change in cabling it was now reliably spitting out the expected u-boot and kernel boot messages, unfortunately ending with errors from the touchscreen driver intermingled with messages from various graphics drivers. I looked at the u-boot environment to see how the kernel parameters were controlled and started tweaking them with the aim of working around the errors. None of this worked. I looked for more documentation on the EVM, but found none beyond the quick start guide. Had I been given the wrong kernel or device tree? They matched the available downloads though the device tree looked very different from the upstream version. I started trying to edit the vendor's device tree to remove the troublesome devices before realising that the decompiled tree was full of absolute cross-references that would be invalid after recompiling. So, a second day of frustration. In the evening I started to feel a cold coming on. I slept poorly, and just dragged myself out of bed in time for 11:00 checkout. In the office, I remembered that u-boot had commands for manipulating device trees in memory. I tried stepping through the boot script and then removing device tree nodes for the touchscreen and graphics hardware before booting the kernel. Success! I got a login prompt where of course username:root, password:[empty] worked. (It doesn't run sshd, but does run telnetd.) Then it was time for a team lunch at the excellent Samsi, though I hesitated to go as I was feeling quite poorly. Shortly after lunch I found myself falling asleep at my desk in the middle of reading documentation. I had to take a nap on the sofa. I got a little more work done that day, but I got progressively feebler. The train journey home was thankfully uneventful. I arrived home at midnight and soon went to sleep. I spent most of Thursday in a high fever, dreaming of abstract entities and their relationships that I struggled to name, but in retrospect might correspond to various kinds of software component. It was incredibly tedious. I had no appetite and little ability to process the real world. Nattie tended to me as she could. I called my mum to cancel her visit planned for the next day. On Friday my fever started to recede and I drank some soup. I was able to look after myself to some extent, though I sorely missed Nattie when she went out. I spent most of the day watching comedy and music docs on the BBC iPlayer, something far too passive for me to do normally. On Saturday morning I felt well enough to tackle the meze platter that Nattie brought back from her previous evening's outing. It tasted pretty damn good. I felt better hour by hour, and in the evening we both went out to a group dinner in a pub. Hot food smelled and tasted even more amazing! I was able to start catching up on email and some real work, preparing kernel stable updates. I even migrated DNS service for decadent.org.uk late that night, driven by anger at the incompetence of my previous provider that had prevented my mails reaching bugs.debian.org. Despite the awfulness of Thursday and the symptoms I still have (tickly throat and runny nose), this bout of the 'flu has been a positive experience for me, but I'll explain why that is in a subsequent entry.

1 March 2016

Thorsten Alteholz: My Debian Activities in February 2016

FTP assistant This month I marked 364 package for accept and rejected 66. Due to the help of lamby, the length of the NEW queue dropped mostly below 50, so there is no need for complaints anymore :-) . I also sent 22 emails to maintainers asking questions. Squeeze LTS This was my twentieth month that I did some work for the Squeeze LTS initiative, started by Raphael Hertzog at Freexian. This month more people started to contribute and my workload dropped down to 11.25h. Altogether I uploaded those DLAs: This month I was also involved in embargoed uploads and could do an upload on my own (DLA 433-1). Now Squeeze LTS is officially done. I leave it with mixed feelings. On the one hand it became more and more difficult to backport patches for the latest version to the old software. On the other hand I could learn a lot of stuff about the methods other maintainers used some years ago. Yes, although not always visible at first sight, over the years there are lots of improvements on how packages can be handled in Debian. So, let us start with Wheezy now Other stuff On the way to pump.io, grunt and some other cool stuff, I uploaded: Yes, sometimes this npm2deb makes it really easy to create a package. In order to fix FTBFSs, errors from DebCI or whatever might fail these days, I also uploaded new versions of: Today I could see the first fruits of my labor. Some packages, I did not touch, migrated to testing because some of their dependencies were finally able to migrate as well.

Vincent Sanders: Hope is tomorrow's veneer over today's disappointment.

Recently I have been very hopeful about the 96boards Hikey SBC and as Evan Esar predicted I have therefore been very disappointed. I was given a Hikey after a Linaro connect event some time ago by another developer who could not get the system working usefully and this is the tale of what followed.
The Standard Design
This board design was presented as Linaro creating a standard for the 64bit Single Board Computer (SBC) market. So I had expected that a project with such lofty goals would have considered many factors and provided at least as good a solution as the existing 32bit boards.

The lamentable hubris of creating a completely new form factor unfortunately sets a pattern for the whole enterprise. Given the aim towards "makers" I would have accepted that the system would not be a ATX PC size motherboard, but mini/micro/nano and pico ITX have been available for several years.

If opting for a smaller "credit card" form factor why not use one of the common ones that have already been defined by systems such as the Raspberry Pi B+? Instead now every 96Board requires special cases and different expansion boards.

Not content with defining their own form factor the design also uses a 8-18V supply, this is the only SBC I own that is not fed from a 5V supply. I understand that a system might require more current than a micro USB connector can provide, but for example the Banana Pi manages with a DC barrel jack readily capable of delivering 25W which would seem more than enough

The new form factor forced the I/O connectors to be placed differently to other SBC, given the opportunity to concentrate all connectors on one edge (like ATX designs) and avoid the issues where two or three sides are used. The 96board design instead puts connectors on two edges removing any possible benefit this might have given.

The actual I/O connectors specified are rather strange. There is a mandate for HDMI removing the possibility of future display technology changes. The odd USB arrangement of two single sockets instead of a stacked seems to be an attempt to keep height down but the expansion headers and CPU heatsink mean this is largely moot.

The biggest issue though is mandating WIFI but not Ethernet (even as an option), everything else in the design I could logically understand but this makes no sense. It means the design is not useful for many applications without adding USB devices.

Expansion is presented as a 2mm pitch DIL socket for "low speed" signals and a high density connector for "high speed" signals. The positioning and arrangement of these connectors proffered an opportunity to improve upon previous SBC designs which was not taken. The use of 2mm pitch and low voltage signals instead of the more traditional 2.54mm pitch 3.3v signals means that most maker type applications will need adapting from the popular Raspberry Pi and Arduino style designs.

In summary the design appears to have been a Linaro project to favour one of their members which took a Hisilicon Android phone reference design and put it onto a board with no actual thought beyond getting it done. Then afterwards attempted to turn that into a specification, this has simply not worked as an approach.

My personal opinion is that this specification is fatally flawed and, is a direct cause of, the bizarre situation where the "consumer" specification exists alongside the "enterprise" edition which itself has an option of microATX form factor anyhow!
The ImplementationIf we ignore the specification appearing to be nothing more than a codification of the original HiKey design we can look at the HiKey as an implementation.

Initially the board required modifying to add headers to attach a USB to 1.8V LVTTL serial adaptor on the UART0 serial port. Once Andy Simpkins had made this change for me I was able to work through the instructions and attempt to install a bootloader and OS image.

The initial software was essentially HiSilicon vendor code using the Android fastboot system to configure booting. There was no source and the Ubuntu OS images were an obvious afterthought to the Android images. Just getting these images installed required a great deal of effort, repetition and debugging. It was such a dreadful experience this signalled the commencement one of the repeated hiatuses throughout this project, the allure of 64 bit ARM computing has its limits even for me.

When I returned to the project I attempted to use the system from the on-board eMMC but the pre-built binary only kernel and OS image were very limited. Building a replacement kernel , or even modules for the existing one proved fruitless and the system was dreadfully unstable.

I wanted to use the system as a builder for some Open Source projects but the system instability ruled this out. I considered attempting to use virtualisation which would also give better system isolation for builder systems. By using KVM running a modern host kernel and OS as a guest this would also avoid issues with the host systems limitations. At which point I discovered the system had no virtualisation enabled apparently because the bootloader lacked support.

In addition to these software issues there were hardware problems, despite forcing the use of USB for all additional connectivity the USB implementation was dreadful. For a start all USB peripherals have to run at the same speed! One cannot mix full (12Mbit) and high speed (480Mbit) devices which makes adding a USB Ethernet and SATA device challenging when you cannot use a keyboard.

And because I needed more challenges only one of the USB root hubs was functional. In effect this made the console serial port critical as it was the only reliable way to reconfigure the system without a keyboard or network link (and WIFI was not reliable either)

After another long pause in proceedings I decided that I should house all the components together and that perhaps being out on my bench might be the cause of some instability. I purchased a powered Amazon basics USB 2 hub, an Ethernet adaptor and a USB 2 SATA interface in the hope of accessing some larger mass storage.

The USB hub power supply was 12V DC which matched the Hikey requirements so I worked out I could use a single 4A capable supply and run a 3.5inch SATA hard drive too. I designed a laser cut enclosure and mounted all the components. As it turned out I only had a 2.5inch hard drive handy so the enclosure is a little over size. If I were redoing this design I would attempt to make it fit in 1U of height and be mountable in a 19inch rack instead it is an 83mm high (under 2U) box.

A new software release had also become available which purported to use an UEFI bootloader after struggling to install this version unsuccessfully, made somewhat more problematic by the undocumented change from UART0 (unpopulated header) to UART3 on the low speed 2mm pitch header. The system seemed to start the kernel which panicked and hung either booting from eMMC or SD card. Once again the project went on hold after spending tens of hours trying to make progress.
Third time's a charmAs the year rolled to a close I once again was persuaded to look at the hikey, I followed the much improved instructions and installed the shiny new November software release which appears to have been made for the re-release of the Hikey through LeMaker. This time I obtained a Debian "jessie" system that booted from the eMMC.

Having a booted system meant I could finally try and use it. I had decided to try and use the system to host virtual machines used as builders within the NetSurf CI system.

The basic OS uses a mixture of normal Debian packages with some replacements from Linaro repositories. I would have prefered to see more use of Debain packages even if they were from the backports repositories but on the positive side it is good to see the use of Debian instead of Ubuntu.

The kernel is a heavily patched 3.18 built in a predominantly monolithic (without modules) manner with the usual exceptions such as the mali and wifi drivers (both of which appear to be binary blobs). The use of a non-mainline capable SoC means the standard generic distribution kernels cannot be used and unless Linaro choose to distribute a kernel with the feature built in it is necessary to compile your own from sources.

The default install has a linaro user which I renamed to my user and ensured all the ssh keys and passwords on the system were changed. This is an important step when using this pre-supplied images as often a booted system is identical to every other copy.

To access mass storage my only option was via USB, indeed to add any additional expansion that is the only choice. The first issue here is that the USB host support is compiled in so when the host ports are initialised it is not possible to select a speed other than 12MBit. The speed is changed to 480Mbit by using a user space application found in the users home directory (why this is not a tool provided by a package and held in sbin I do not know).

When the usb_speed tool is run there is a chance that the previously enumerated devices will be rescanned and what was /dev/sda has become /dev/sdb if this happens there is a high probability that the system must be rebooted to prevent random crashes due to the "zombie" device.

Because the speed change operation is unreliable it cannot be reliably placed in the boot sequence so this must be executed by hand on each boot to get access to the mass storage.

NetSurf project already uses a x86_64 virtual host system which runs an LLVM physical volume from which we allocate logical volumes for each VM. I initially hoped to do this with the hikey but as soon as I tried to use the logical volume with a VM the system locked up with nothing shown on console. I did not really try very hard to discover why and instead simply used files on disc for virtual drives which seemed to work.

To provide reliable network access I used a USB attached Ethernet device, this like the mass storage suffered from unreliable enumeration and for similar reasons could not be automated requiring manually using the serial console to start the system.

Once the system was started I needed to install the guest VM. I had hoped I might be able to install locally from Debian install media as I do for x86 using the libvirt tools. After a great deal of trial and error I finally was forced to abandon this approach when I discovered the Linaro kernel is lacking iso9660 support so installing from standard media was not possible.

Instead I used the instructions provided by Leif Lindholm to create a virtual machine image on my PC and copied the result across. These instructions are great except I used version 2.5 of Qemu instead of 2.2 which had no negative effect. I also installed the Debian backports for Jessie to get an up to date 4.3 kernel.

After copying the image to the Hikey I started it by hand from the command line as a four core virtual machine and was successfully able to log in. The guest would operate for up to a day before stopping with output such as

$
Message from syslogd@ciworker13 at Jan 29 07:45:28 ...
 kernel:[68903.702501] BUG: soft lockup - CPU#0 stuck for 27s! [mv:24089]

Message from syslogd@ciworker13 at Jan 29 07:45:28 ...

 kernel:[68976.958028] BUG: soft lockup - CPU#2 stuck for 74s! [swapper/2:0]

Message from syslogd@ciworker13 at Jan 29 07:47:39 ...
 kernel:[69103.199724] BUG: soft lockup - CPU#3 stuck for 99s! [swapper/3:0]

Message from syslogd@ciworker13 at Jan 29 07:53:21 ...
 kernel:[69140.321145] BUG: soft lockup - CPU#3 stuck for 30s! [rs:main Q:Reg:505]

Message from syslogd@ciworker13 at Jan 29 07:53:21 ...
 kernel:[69192.880804] BUG: soft lockup - CPU#0 stuck for 21s! [jbd2/vda3-8:107]

Message from syslogd@ciworker13 at Jan 29 07:53:21 ...
 kernel:[69444.805235] BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0]

Message from syslogd@ciworker13 at Jan 29 07:55:21 ...
 kernel:[69570.177600] BUG: soft lockup - CPU#1 stuck for 112s! [systemd:1]

Timeout, server 192.168.7.211 not responding.

After this output the host system would not respond and had to be power cycled never mind the guest!

Once I changed to single core operation the system would run for some time until the host suffered from the dreaded kernel OOM killer. I was at a loss as to why the oom killer was running as the VM was only allocated half the physical memory (512MB) allowing the host what I presumed to be an adequate amount.

By adding a 512MB swapfile the system was able to push the few hundred kilobytes it wanted to swap and the system was now stable! The swapfile of course has to be started by hand as the external storage is unreliable and unavailable at boot.

I converted the qemu command line to a libvirt config using the virsh tool
virsh domxml-from-native qemu-argv cmdln.args
The converted configuration required manual editing to get a working system but now I have a libvirt based VM guest I can control along with all my other VM using the virt-manager tool.

This system is now stable and has been in production use for a month at time of writing. The one guest VM is a single core 512MB aarch64 system which takes over 1100 seconds (19 minutes) to do what a Banana Pi 2 dual core 1GB memory 32bit native ARM system manages in 300 seconds.

It seems the single core limited memory system with USB SATA attached storage is very, very slow.

I briefly attempted to run the CI system job natively within the host system but within minutes it crashed hard and required a power cycle to retrieve, it had also broken the UEFI boot. I must thank Leif for walking me through recovering the system otherwise I would have needed to start over.
ConclusionsI must stress these conclusions and observations are my own and do not represent anyone else.

My main conclusions are:

This project has taken almost a year to get to the final state and has been one of the least enjoyable within that time. The only reason I have a running system at the end of it is sheer bloody mindedness because after spending hundreds of hours of my free time I was not prepared to see it all go to waste.

To be fair, the road I travelled is now much smoother and if the application is suited to having a mobile phone without a screen then the Hikey probably works as a solution. For me, however, the Hikey product with the current hardware and software limitations is not something I would recommend in preference to other options.
 
 





 
Ritesh Raj Sarraf: Linux Power Savings 2016

 

     
Having moved to a new place, now at times, I also have to deal with power outages. As heat increases, the power outages will be much longer and more frequent. So much, that UPS and Power Inverters run out. Such are ideal times to measure idle power consumption for my laptop.
Here's what my default (and idle) OS looks like. It should be standard to most "typical" users. Some minor odds could be apport, dnsmasq, and maybe, tor. Also, thanks to systemd, there are some native services which are now converted to socket activation based ones. There's also Laptop Mode Tools for userspace power savings. But huge thanks to Intel's work in making the kernel more power efficient. With things like Runtime PM and Intel P-State, we can now define, default and aggressive power-savings modes, helping conserve more power.
We still have challenges with things like binary-only drivers; and also stock GPL drivers but with an external firmware blob. There have been many instances when my previous machines (which had both) ran into severe power drain. The challenge with power savings is that it is not just the job of the kernel. All components, kernel and userspace, need to be made aware of, and work in unison. Otherwise, one ill behaving web browser tab can be the cause of power drain. P-State driver seems to already be doing some scale down of CPU resources. And with CGroups maturing moe I think we'll eventually have all Desktop Environments making use of it.

systemd-+-ModemManager-+- gdbus 
                        - gmain 
         -NetworkManager-+- gdbus 
                          - gmain 
         -accounts-daemon-+- gdbus 
                           - gmain 
         -agetty
         -apport-notifyd
         -atd
         -avahi-daemon---avahi-daemon
         -bluetoothd
         -colord-+- gdbus 
                  - gmain 
         -cron
         -dbus-daemon
         -dnsmasq
         -evolution-+- book-client-dbu 
                     - dconf worker 
                     -7*[ evolution ]
                     - gdbus 
                     - gmain 
         -gdm3-+-gdm-session-wor-+-gdm-x-session-+-Xorg
                                                  -gnome-session-b-+-gnome-settings--+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                                      - pool 
                                                                                      - threaded-ml 
                                                                    -gnome-shell-+-ibus-daemon-+-ibus-dconf-+- dconf worker 
                                                                                                             - gdbus 
                                                                                                             - gmain 
                                                                                                -ibus-engine-sim-+- gdbus 
                                                                                                                  - gmain 
                                                                                                - gdbus 
                                                                                                - gmain 
                                                                                  - JS GC Helper 
                                                                                  - JS Sour~ Thread 
                                                                                  - dconf worker 
                                                                                  - gdbus 
                                                                                  - gmain 
                                                                                  - threaded-ml 
                                                                    - dconf worker 
                                                                    - gdbus 
                                                                    - gmain 
                                                  - gdbus 
                                                  - gmain 
                                  - gdbus 
                                  - gmain 
                -gdm-session-wor-+-gdm-x-session-+-Xorg
                                                  -gnome-session-b-+-evolution-alarm-+- cal-client-dbus 
                                                                                      - dconf worker 
                                                                                      - evolution-alarm 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                    -gnome-settings--+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                                      - pool 
                                                                                      - threaded-ml 
                                                                    -gnome-shell-+-2*[redshift-+- gdbus ]
                                                                                                - gmain ]
                                                                                  - JS GC Helper 
                                                                                  - JS Sour~ Thread 
                                                                                  - dconf worker 
                                                                                  - gdbus 
                                                                                  - gmain 
                                                                                  - threaded-ml 
                                                                    -parcimonie
                                                                    -parcimonie-appl-+- gdbus 
                                                                                      - gmain 
                                                                    -python3---libinput-debug-
                                                                    -ssh-agent
                                                                    -tracker-extract-+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                                      -10*[ pool ]
                                                                    -tracker-miner-a-+- gdbus 
                                                                                      - gmain 
                                                                    -tracker-miner-f-+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                    -tracker-miner-u-+- gdbus 
                                                                                      - gmain 
                                                                    - dconf worker 
                                                                    - gdbus 
                                                                    - gmain 
                                                  - gdbus 
                                                  - gmain 
                                  - gdbus 
                                  - gmain 
                - gdbus 
                - gmain 
         -geoclue-+- gdbus 
                   - gmain 
         -gnome-keyring-d-+- gdbus 
                           - gmain 
                           - timer 
         -gpg-agent
         -gsd-printer-+- gdbus 
                       - gmain 
         -ibus-daemon-+-ibus-dconf-+- dconf worker 
                                    - gdbus 
                                    - gmain 
                       -ibus-engine-sim-+- gdbus 
                                         - gmain 
                       - gdbus 
                       - gmain 
         -ibus-x11-+- gdbus 
                    - gmain 
         -iio-sensor-prox-+- gdbus 
                           - gmain 
         -inetd---leafnode
         -libvirtd---15*[ libvirtd ]
         -lvmetad
         -master-+-pickup
                  -qmgr
                  -tlsmgr
         -mcelog
         -minissdpd
         -mount.ntfs
         -onboard-+- dconf worker 
                   - gdbus 
                   - gmain 
         -packagekitd-+- gdbus 
                       - gmain 
         -polkitd-+- gdbus 
                   - gmain 
         -2*[pulseaudio-+-gconf-helper]
                         - alsa-sink-ALC23 ]
                         - alsa-sink-HDMI  ]
                         - alsa-sink-pcspe ]
                         - alsa-source-ALC ]
         -rtkit-daemon---2*[ rtkit-daemon ]
         -systemd-+-(sd-pam)
                   -at-spi-bus-laun-+-dbus-daemon
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -at-spi2-registr-+- gdbus 
                                     - gmain 
                   -dbus-daemon
                   -dconf-service-+- gdbus 
                                   - gmain 
                   -goa-daemon-+- dconf worker 
                                - gdbus 
                                - gmain 
                   -goa-identity-se-+- gdbus 
                                     - gmain 
                                     - pool 
                   -gvfs-afc-volume-+- gdbus 
                                     - gmain 
                                     - gvfs-afc-volume 
                   -gvfs-goa-volume-+- gdbus 
                                     - gmain 
                   -gvfs-gphoto2-vo-+- gdbus 
                                     - gmain 
                   -gvfs-mtp-volume-+- gdbus 
                                     - gmain 
                   -gvfs-udisks2-vo-+- gdbus 
                                     - gmain 
                   -gvfsd-+- gdbus 
                           - gmain 
                   -gvfsd-fuse-+- gdbus 
                                - gmain 
                                - gvfs-fuse-sub 
                                -2*[ gvfsd-fuse ]
                   -gvfsd-metadata-+- gdbus 
                                    - gmain 
                   -mission-control-+- dconf worker 
                                     - gdbus 
                                     - gmain 
         -systemd-+-(sd-pam)
                   -at-spi-bus-laun-+-dbus-daemon
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -at-spi2-registr-+- gdbus 
                                     - gmain 
                   -dbus-daemon
                   -gconfd-2
         -systemd-+-(sd-pam)
                   -at-spi-bus-laun-+-dbus-daemon
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -at-spi2-registr-+- gdbus 
                                     - gmain 
                   -dbus-daemon
                   -dconf-service-+- gdbus 
                                   - gmain 
                   -evolution-addre-+-2*[evolution-addre-+- dconf worker ]
                                                          - evolution-addre ]
                                                          - gdbus ]
                                                          - gmain ]
                                     - dconf worker 
                                     - evolution-addre 
                                     - gdbus 
                                     - gmain 
                   -evolution-calen-+-evolution-calen-+- dconf worker 
                                                       - evolution-calen 
                                                       - gdbus 
                                                       - gmain 
                                                       -6*[ pool ]
                                     -evolution-calen-+- book-client-dbu 
                                                       - dconf worker 
                                                       - evolution-calen 
                                                       - gdbus 
                                                       - gmain 
                                                       - pool 
                                     -2*[evolution-calen-+- dconf worker ]
                                                          - evolution-calen ]
                                                          - gdbus ]
                                                          - gmain ]
                                     - dconf worker 
                                     - evolution-calen 
                                     - gdbus 
                                     - gmain 
                   -evolution-sourc-+- dconf worker 
                                     - gdbus 
                                     - gmain 
                   -gconfd-2
                   -gnome-shell-cal-+- cal-client-dbus 
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                                     - gnome-shell-cal 
                   -gnome-terminal--+-bash---pstree
                                     -bash
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -goa-daemon-+- dconf worker 
                                - gdbus 
                                - gmain 
                   -goa-identity-se-+- gdbus 
                                     - gmain 
                                     - pool 
                   -gpaste-daemon-+- dconf worker 
                                   - gdbus 
                                   - gmain 
                   -gvfs-afc-volume-+- gdbus 
                                     - gmain 
                                     - gvfs-afc-volume 
                   -gvfs-goa-volume-+- gdbus 
                                     - gmain 
                   -gvfs-gphoto2-vo-+- gdbus 
                                     - gmain 
                   -gvfs-mtp-volume-+- gdbus 
                                     - gmain 
                   -gvfs-udisks2-vo-+- gdbus 
                                     - gmain 
                   -gvfsd-+- gdbus 
                           - gmain 
                   -gvfsd-burn-+- gdbus 
                                - gmain 
                   -gvfsd-dnssd-+- gdbus 
                                 - gmain 
                   -gvfsd-fuse-+- gdbus 
                                - gmain 
                                - gvfs-fuse-sub 
                                -2*[ gvfsd-fuse ]
                   -gvfsd-metadata-+- gdbus 
                                    - gmain 
                   -gvfsd-network-+- dconf worker 
                                   - gdbus 
                                   - gmain 
                   -gvfsd-trash-+- gdbus 
                                 - gmain 
                   -mission-control-+- dconf worker 
                                     - gdbus 
                                     - gmain 
                   -nautilus-+-gedit
                              - dconf worker 
                              - gdbus 
                              - gmain 
                   -tracker-store-+- dconf worker 
                                   - gdbus 
                                   - gmain 
                                   -4*[ pool ]
         -systemd-journal
         -systemd-logind
         -systemd-network
         -systemd-timesyn--- sd-resolve 
         -systemd-udevd
         -thermald--- thermald 
         -tor
         -udisksd-+- cleanup 
                   - gdbus 
                   - gmain 
                   - probing-thread 
         -upowerd-+- gdbus 
                   - gmain 
         -wpa_supplicant

So I used the standard power measurement tool, powertop. Below is a screenshot of PowerTop claiming 9 - 10 hours of possible battery backup. This is idle state. "Idle" would mean when the user is not interacting with the machine. When under basic usage (email, web), the actual battery backup I've sensed is somewhere around 6-7 hrs, which is still good from what we got a couple years ago.
There's one oddity in the screenshot though. The estimated power consumption reported is 4.55W, where as, the power consumption of just the wifi card is reported 6.17W. That too, when the wifi was disconnected.
This reminded me of the bug report against PowerTop, which is now more than 2 years old. We've still not concluded in that bug report, so if anyone can shed some light on PowerTop's reporting, please do share it in the bug report.

Note: Please ignore the terminal's title in the screenshot. That is wrong and I've not bothered to figure out why it is displaying the title name constructed from an outdated session, which is already terminated.

Categories: 
Keywords: 
Like: 
 
 





 
Enrico Zini: Static site generators

 

   I decided to rethink the state of my personal site, and try out some of the new
static site generators that are available now.
To do that, I jotted down a series of things that I want in a static site
generator, then wrote a tool to
convert my ikiwiki site to other formats, and set out
to evaluate things.
As a benchmark I did a full rebuild of my site, which currently contains
1164 static files and 458 markdown pages.
My requirements
Free layout for my site
My / is mostly a high-level index to the site contents.
Blog posts are at /blog.
My talk archive is organised like a separate blog at /talks.
I want the freedom to create other sections of the site, each with its own rss
feed, located wherever I want in the site hierarchy.
Assets next to posts
I occasionally blog just a photo with a little
comment, and I would like the .md page with the
comment to live next to the image in the file system.
I did not even know that I had this as a requirement until I found static site
generators that mandated a completely different directory structure for
markdown contents and for static assets.
Multiple RSS/Atom feeds
I want at least one RSS/Atom feed per tag, because I use tags for marking which
articles go to http://planet.debian.org.
I also want RSS/Atom feeds for specific parts of the site, like the
blog and talks.
Arbitrary contents in /index.html
I want total control over the contents of the main home page of the site.
Quick preview while editing contents
I do not like to wait several seconds for the site to be rebuilt at every
review iteration of the pages I write.
This makes me feel like the task of editing is harder than it should, and makes
me lose motivation to post.
Reasonable time for a full site rebuild
I want to be able to run a full rebuild of the site in a reasonable time.
I could define "reasonable" in this case as how long I can stare at the screen
without getting bored, starting to do something else, and forgetting what it
was that I was doing with the site.
It is ok if a rebuild takes something like 10 or 30 seconds. It is not ok if it
takes minutes.
Code and dependency ecosystems that I can work with
I can deal with Python and Go.
I cannot deal with Ruby or JavaScript.
I forgot all about Perl.
Also, if it isn't in Debian it does not exist.
Decent themes out of the box
One of my hopes in switching to a more mainstream generator is to pick and
choose themes and easily give my site a more modern look.
Hugo
Hugo is written in Go and is in Debian testing.
Full rebuild time for my site is acceptable, and it can even parallelize:

  $ time hugo
  real      0m5.285s
  user      0m9.556s
  sys       0m1.052s


Free layout for my site was hard to get.
I could replace /index.html by editing the template page for it, but then I
did not find out how to create another similar index in an arbitrary place.
Also, archetypes are applied only on the first path component of new posts, but
I would like them instead to be matched on the last path component first, and
failing that traveling up to the path until the top. This should be easy to fix
by reorganizing the content a bit around
here
For example, a path for a new blog post of mine could be blog/2016/debian/
and I would like it to match the debian archetype first, and failing that the
blog archetype.
Assets next to posts almost work.
Hugo automatically generates one feed per taxonomy element, and one feed per
section. This would be currently
sufficient for me, although I don't like the idea that sections map 1 to 1 to
toplevel directories in the site structure.
Hugo has a server that watches the file system and rerenders pages as they are
modified, so the quick preview while editing works fine.
About themes, it took me several tries to find a theme that would render
navigation elements for both sections and tags, and most themes would render by
pages with white components all around, and expect me to somehow dig in and
tweak them. That frustrated me, because for quite a while I could not tell if I
had misconfigured Hugo's taxonomies or if the theme was just somehow
incomplete.
Nikola
Nikola is written in Python and is in Debian testing.
Full rebuild time for my site is almost two orders of magnitude more than Hugo,
and I am miffed to find the phrases "Nikola is fast." or "Fast building
process" in its front page and package description:

  $ time nikola build
  real      3m31.667s
  user      3m4.016s
  sys       0m24.684s


Free layout could be achieved fiddling with the site configuration to tell it
where to read sources.
Assets next to post work after tweaking the configuration, but they require to
write inconsistent links in the markdown source:
https://github.com/getnikola/nikola/issues/2266
I have a hard time accepting that that, because I want to author content with
consistent semantic interlinking, because I want to be able 10 years from now
to parse it and convert it to something else if a new technology comes out.
Nikola generates one RSS/Atom feed per tag just fine. I have not tried
generating feeds for different sections of the site.
Incremental generation inside its built in server works fine.
Pelican
Pelican is written in Python and is in Debian
testing.
Full rebuild time for my site is acceptable:

  $ time pelican -d
  real      0m18.207s
  user      0m16.680s
  sys       0m1.448s


By default, pelican seems to put generate a single flat directory of html files
regardless of the directory hierarchy of the sources. To have free layout,
pelican needs some convincing in the configuration:

  PATH_METADATA = r"(?P<relpath>.+)\.md"
  ARTICLE_SAVE_AS = " relpath /index.html"


but even if I do that, the urls that it generates still point to just
 slug /index.html and I have not trivially found a configuration option to
fix that accordingly. I got quite uncomfortable at the idea of needing to
configure content generation and linking to match, instead of having one
automatically being in sync with the other.
Having assets next to posts seems to be possible
(also setting STATIC_PATHS = ["."]), but I do not recall making progress on
this front.
I did not manage to generate a feed for each tag out of the box, and probably
there is some knob in the configuration for it.
I gave up with Pelican as trying it out felt like a constant process of
hacking the configuration from defaults that do not make any sense for me,
withouth even knowing if a configuration exists that would do what I need
Ikiwiki
Ikiwiki is written in Perl and is in Debian. Although
I am not anymore proficient with Perl, I was already using it, so it was worth
considering.
Full rebuild time feels a bit on the slow side but is still acceptable:

  $ time ikiwiki --setup site.setup
  real      0m37.356s
  user      0m34.488s
  sys       0m1.536s


In terms of free site structure, all feeds for all or part of the site, ikiwiki
just excels.
I even considered writing a python web server that monitors the file system and
calls ikiwiki --refresh when anything changes, and calling it a day.
However, when I tried to re-theme my website around a simple bootstrap
boilerplate, I found that to be hard, as a some of the HTML structure is
hardcoded in Perl
(and it's also my fault) and there is only so much that can be done by tweaking
the (rather unreadable) templates.
siterefactor
During all these experiments I had built siterefactor
to generate contents for all those static site engines, and it was going
through all the contents quite fast:

  $ time ./siterefactor src dst -t hugo
  real  0m1.222s
  user  0m0.904s
  sys   0m0.308s


So I wondered how slow it would become if, instead of making it write markdown,
I made it write HTML via python markdown
and Jinja2:

  $ time ./siterefactor ~/zz/ikiwiki/pub/ ~/zz/ikiwiki/web -t web
  real  0m6.739s
  user  0m5.952s
  sys   0m0.416s


I then started wondering how slower it would become if I implemented
postprocessing of all local URLs generated by Markdown to make sure they are
kept consistent even if the path of a generated page is different than the path
of its source. Not much slower, really.
I then added taxonomies. And arbitrary Jinja2 templates in the input, able to
generate page lists and RSS/Atom feeds.
And theming.
And realised that reading all the sources and cross-linking them took 0.2
seconds, and the rest was generation time. And that after cross-linking, each
page can be generated independently from all the others.
staticsite
So my site is now generated with staticsite:

  $ time ssite build
  real  0m6.833s
  user  0m5.804s
  sys   0m0.500s


It's comparable with Hugo, and on a single process. 
 





 
Erich Schubert: Stop abusing lambda expressions - this is not functional programming

 

     
I know, all the Scala fanboys are going to hate me now. But:


Stop overusing lambda expressions.



Most of the time when you are using lambdas, you are not even doing
functional programming, because you often are violating one key rule of
functional
programming: no side effects.


For example:

collection.forEach(System.out::println);


is of course very cute to use, and is (wow) 10 characters shorter than:

for (Object o : collection) System.out.println(o);


but this is not functional programming because it has side effects.


What you are doing are anonymous methods/objects, using a
shorthand notion. It's sometimes convenient, it is usually short, and
unfortunately often unreadable, once you start cramming complex problems into
this framework.


It does not offer efficiency improvements, unless you have the
propery of side-effect freeness (and a language compiler that can exploit
this, or parallelism that can then call the function concurrently in arbitrary
order and still yield the same result).


Here is an examples of how to not use lambdas:

DZone Java 8
Factorial (with boilerplate such as the Pair class omitted): 


Stream<Pair> allFactorials = Stream.iterate(
  new Pair(BigInteger.ONE, BigInteger.ONE),
  x -> new Pair(
    x.num.add(BigInteger.ONE),
    x.value.multiply(x.num.add(BigInteger.ONE))));
return allFactorials.filter(
  (x) -> x.num.equals(num)).findAny().get().value;


When you are fresh out of the functional programming class, this may seem
like a good idea to you... (and in contrast to the examples mentioned above,
this is really a functional program).

But such code is a pain to read, and will not scale well either.
Rewriting this to classic Java yields:

BigInteger cur = BigInteger.ONE, acc = BigInteger.ONE;
while(cur.compareTo(num) <= 0)  
  cur = cur.add(BigInteger.ONE); // Unfortunately, BigInteger is immutable!
  acc = acc.multiply(cur);
 
return acc;






Sorry, but the traditional loop is much more readable. It will still
not perform very well (because of BigInteger not being designed for efficiency
- it does not even make sense to allow BigInteger for num - the
factorial of 2**63-1, the maximum of a Java long, needs
1020 bytes to store, i.e. about 500 exabyte.



For some, I did some benchmarking. One hundred random values
num (of course the same for all methods) from the range 1 to 1000.


I also included this even more traditional version:

BigInteger acc = BigInteger.ONE;
for(long i = 2; i <=x; i++)  
  acc = acc.multiply(BigInteger.valueOf(i));
 
return acc;





Here are the results (Microbenchmark, using JMH, 10 warum iterations,
20 measurement iterations of 1 second each):

functional    1000     100  avgt   20  9748276,035   222981,283  ns/op
biginteger    1000     100  avgt   20  7920254,491   247454,534  ns/op
traditional   1000     100  avgt   20  6360620,309   135236,735  ns/op


As you can see, this "functional" approach above is about 50% slower than the
classic for-loop. This will be mostly due to the Pair and additional
BigInteger objects created and garbage collected.



Apart from being substantially faster, the iterative approach is also
much simpler to follow. (To some extend it is faster because it is also
easier for the compiler!)


There was a recent blog post by Robert Br utigam that
discussed exception throwing in Java
lambdas and the pitfalls associated with this. The discussed approach
involves abusing generics for throwing unknown checked exceptions in the
lambdas, ouch.




Don't get me wrong. There are cases where the use of lambdas is
perfectly reasonable. There are also cases where it adheres to the
"functional programming" principle. For example, a
stream.filter(x -> x.name.equals("John Doe")) can be a readable
shorthand when selecting or preprocessing data. If it is really functional
(side-effect free), then it can safely be run in parallel and give you some
speedup.


Also, Java lambdas were carefully designed, and the hotspot VM tries hard
to optimize them. That is why Java lambdas are not closures - that would be
much less performant. Also, the stack traces of Java lambdas remain
somewhat readable (although still much worse than those of traditional code).
This blog post by Takipi showcases how bad the stacktraces become (in the
Java example, the stream function is more to blame than the
actual lambda - nevertheless, the actual lambda application shows up as
the cryptic LmbdaMain$$Lambda$1/821270929.apply(Unknown Source)
without line number information). Java 8 added new bytecodes to be able to
optimize Lambdas better - earlier JVM-based languages may not yet make good
use of this.


But you really should use lambdas only for one-liners. If it is a more
complex method, you should give it a name to encourage reuse and
improve debugging.


Beware of the cost of .boxed() streams!


And do not overuse lambdas. Most often, non-Lambda code is
just as compact, and much more readable. Similar to foreach-loops, you do
lose some flexibility compared to the "raw" APIs such as Iterators:

for(Iterator<Something>> it = collection.iterator(); it.hasNext(); )  
  Something s = it.next();
  if (someTest(s)) continue; // Skip
  if (otherTest(s)) it.remove(); // Remove
  if (thirdTest(s)) process(s); // Call-out to a complex function
  if (fourthTest(s)) break; // Stop early
 


In many cases, this code is preferrable to the lambda hacks we see pop up
everywhere these days. Above code is efficient, and readable.

If you can solve it with a for loop, use a for loop!



Code quality is not measured by how much functionality you can do
without typing a semicolon or a newline!


On the contrary: the key ingredient to writing high-performance code
is the memory layout (usually) - something you need to do low-level.


Instead of going crazy about Lambdas, I'm more looking forward to real
value types
(similar to a struct in C, reference-free objects) maybe in Java 9
(Project Valhalla),
as they will allow reducing the memory impact for many scenarios considerably.
I'd prefer a mutable design, however - I understand why this is proposed,
but the uses cases I have in mind become much less elegant when having to
overwrite instead of modify all the time.
 
 





 
Russ Allbery: Review: The Lost Steersman

 

   Review: The Lost Steersman, by Rosemary Kirstein


  

    Series:
    Steerswomen #3
  

  

    Publisher:
    Rosemary Kirstein
  

  

    Copyright:
    2003, 2014
  

  

    Printing:
    2014
  

  

    ISBN:
    0-9913546-2-1
  

  

    Format:
    Kindle
  

  

    Pages:
    432
  


This is the third book in the Steerswomen series and a direct
follow-up to the events of The
Outskirter's Secret.  It does, marvel of marvels, feature an
in-character summary of the events of the series to date!  I do love when
authors do this; it helps immensely if you come back to a series after a
bit of a break between books.  But this whole series is so good, and the
emotional tone and development of Rowan as a character is so strong, that
I recommend against starting here.


After the events of the last book, Rowan has returned to the Inner Lands.
She's sent her report back to the Archives, but stopped at the Annex in
Alemeth.  This is an auxiliary library that should have copies of the
journals and other research that Rowan wants to search, and stopping there
saves substantial travel time.  However, she finds the steerswoman who was
custodian of the Annex is deceased and the Annex is, from Rowan's
perspective, a mess.  Nothing is organized, the books aren't properly
cared-for, and Mira's interactions with the townsfolk were far different
than Rowan's natural attitude.


The start of this book was a surprising shift.  After the large-scale
revelations at the end of The Outskirter's Secret, and the sense of
escalating danger, Rowan's return to small-town life in the Inner Lands
comes as a shock.  That's true for both the reader and for Rowan, and the
parallels make it a remarkably effective bit of writing.


At the beginning of the story, the reader is already familiar with Rowan
(at least if you've read the previous books) and how she thinks of being a
steerswoman.  Rowan is very much on edge and in a hurry given what's going
on in the broader world.  But the town is used to Mira: a gregarious
socializer who cared far more about town gossip and her role as
coordinator of it than she cared about most of her steerswoman duties.
(At least as seen from Rowan's perspective.  By the end of the book, we
have a few hints that something else might be going on, but the damage to
the books at least feels unforgivable.)  Rowan is resented and even
disliked at first, particularly by Steffie and Gwen who did most of the
chores at the Annex and were closest to Mira.


One of the reasons why I love this series so much is that Kirstein has a
gift for characterization.  Rowan (and Bel, who largely doesn't appear in
this book) are brilliant characters, but it's not just them.  At the start
of this book, the reader tends to share Rowan's opinion of the town: a
sort of half-bemused, half-exasperated indifference.  Even as the
characters start to grow on one, it seems like a backwater and a diversion
from the larger story.  But it becomes clear that Rowan is very on-edge
from her experience in the Outskirts, that she's underestimated the
relevance of Mira to at least the town's happiness, and she's greatly
underestimated the ability of the townsfolk to help her.  Steffie, in
particular, is a wonderful character; by the end of the book, he had
become one of my favorite people in the series so far.  He doesn't think
he's particularly smart, and his life before Rowan is very simple, but
there are depths to him that no one, including him, expected.


There is a plot here, apart from small town politics and Rowan's slow
relaxation.  (Although those were so compelling that I'm not sure I would
have minded if that were the entire book.)  The lost steersman of the
title is an old student friend of Rowan that she unexpectedly meets in
town, a former steersman who quit the order and refused to explain why.
Rowan, of course, cannot resist trying to fix this situation.  The second
plot driver is a dangerous invasion of Outskirts monsters into the town.
Those who have read the previous books will have some immediate guesses as
to why this might be, and Rowan does as well.  But there's more going on
than it might first appear.


This book is not entirely a diversion.  It returns to the main plot of the
series by the end of the book, and we learn much more about Rowan's world.
But, somewhat surprisingly, that was my least favorite part of the book.
It has some nice bits of exploration and puzzle-solving, and Rowan is
always a delight to spend time with.  But the last section of the book is
similar to many other genre books I've read before   well-written, to be
sure, but not as unique.  It also features a rather long section following
a character who is severely physically ill, which is something I always
find very hard to read (a personal quirk).  But there's a lot of meat here
for the broader plot, and I have no idea what will happen in the next
book.


The part of The Lost Steersman that I'm going to remember, though,
are the town bits, up through the arrival of Zenna (another delightful
character who adds even more variety to Kirstein's presentation of
steerswomen).  Kirstein is remarkably good at mixing small-town characters
with the scientific investigation of the steerswomen and letting them
bounce off of each other to reveal more about the character of both.  If
it weren't for the end of the book, which bothered me for partly
idiosyncratic reasons, I think this would have been my favorite book of
the series.


Followed by The Language of Power, and be warned that this book
ends on something close to a cliffhanger.

Rating: 8 out of 10 
 





 
Stein Magnus Jodal: February contributions

 

   The following is a short summary of my open source work in February, just like
in
previous
months.

Debian


Mopidy

    
  
  • 
    Released Mopidy-Spotify 2.3.1:
Works around search being broken in libspotify by using the Spotify Web API
for the search functionality, and nothing else.
  
    • 
  
  • 
    Wrapped up and released Mopidy 2.0
release. This was a quite
large release that ports to GStreamer 1.x and adds support for gapless
playback. It was uploaded to Debian in time to be part of Ubuntu 16.04 LTS.
  
    • 
  
  • 
    Released Mopidy-Spotify 3.0.0: A
couple of small compatibility fixes to work with Mopidy 2.x and GStreamer
1.x.
  
    • 
  
  • 
    Updates to packages not in Debian, only at apt.mopidy.com:
    
      
      
    • Uploaded mopidy-spotify-tunigo 1.0.0-0mopidy1. Far down the chain, it
depends on libspotify, which is not in Debian.
      • 
      
    • Uploaded mopidy-spotify 2.0.0-0mopidy1.
      • 
    
    
  
    • 
  
  • 
    Updated homebrew-mopidy with
lots of new releases:
    
      
      
    • Mopidy 2.0.0
      • 
      
    • Mopidy-Dirble 1.3.0
      • 
      
    • Mopidy-SoundCloud 2.0.2
      • 
      
    • Mopidy-Spotify 2.3.0, 2.3.1 and 3.0.0
      • 
      
    • python-backports-abc 0.4
      • 
      
    • python-backports-ssl-match-hostname 3.5.0.1
      • 
      
    • python-certifi 2015.11.20.1
      • 
      
    • python-cffi 1.5.0 and 1.5.2
      • 
      
    • python-requests 2.9.1
      • 
      
    • python-singledispatch 3.4.0.3
      • 
      
    • python-six 1.10.0
      • 
      
    • python-tornado 4.3
      • 
    
    
  
    • 
  
  • 
    Rebased my
feature/py3-compat
branch on top of Mopidy 2.0.
  
    • 
  
  • 
    Planned breaking changes for Mopidy 3.0:
unbundling of Mopidy.js,
removal of deprecated
core APIs,
model APIs,
audio APIs and removal of the
http/static_dir config.
  
    • 
  
  • 
    Planned breaking changes to modernize Mopidy.js:
change default calling convention
to by-position-or-by-name,
replace When.js with ES6 Promise,
replace Bane.js with EventEmitter,
and
replace Buster.js with Karma+Mocha.
  
    • 



Comics

    
  
  • Started working on the upgrade to Django 1.8. Most deps are
upgraded.
The major remaining pieces is to upgrade django-registration from 0.8 to 2.0
and to replace the years-old vendored copy of django-invitation (singular)
with django-invitations
(plural).
    • 

 
 




29 February 2016



 
Chris Lamb: Free software activities in February 2016

 

     Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

    

  • Various improvements to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
      

    • Added explicit support to send messages asynchronously via the Celery distributed task queue. (#29)
      • 

    • Worked with Patrick Clope to add an escapeslack template filter to ensure the correct characters for Slack's API are escaped   using Django's built-in safe is too invasive. (#36)
      • 

    • Corrected an issue where custom endpoints and channel names were incompatible. (#27)
      • 

    • Overhaul of the field/option/block parsing. (49ff3c4)
      • 

    • Moved away from using a django.template.Context instance, preventing a deprecation warning. (#31)
      • 

    

    • 

  • Added a create-folders subcommand to my tickle-me-email Getting Things Done (GTD) email toolbox to create numbered folders for the rotate "tickler" functionality. (#3)
    • 

  • Wrote and released a Django template tag to collapse multiple whitespace/newline characters in order to correctly format, for example, plaintext emails that make extensive use of for-loops. These would normally require careful and fragile placement of the  % for .. %  and  % endfor %  tags. (Repo)
    • 

  • Pushed a number of updates to my Strava Enhancement Suite, a Chrome extension that improves and fixes annoyances in the web interface of the Strava cycling and running tracker:
      

    • Fix unit conversion where the element was wrapped in an another HTML element. (#48)
      • 

    • Hide premium indicator from top-level navigation. (#45)
      • 

    • Correct matching of the "shop" link in top-level navigation and footer module. (#47 & #46)
      • 

    • Remove "premium" pages on Club pages. (#44)
      • 

    

    • 

  • Added a security-oriented warning to Ansible's documentation regarding the behaviour if an UFW firewall application profile is added and subsequently removed. (#1740)
    • 

  • Added support to my django-staticfiles-dotd "staticfiles" library   which concatenates Javascript and CSS files from .d-style directories   to support an arbitrary file rendering method. This allows the use of media pre-processors such as SASS within such directories. (#1)
    • 

  • Corrected my Chrome extension for the FastMail web interface to correctly hook into the internal "send" mechanism. (#2)
    • 

  • Moved my stravabot IRC bot to use dh-virtualenv and systemd, improving security and reliability. (1c48709)
    • 

  • Updated django-pedantic-http-methods   a tool to raise an exception during development when attempting to perform side effects in GET and HEAD HTTP methods   to support the latest version of Django. (#1)
    • 






Debian

    

  • Updated travis.debian.net   a hosted script to easily test and build Debian packages on the Travis CI continuous integration platform   to support:
      

    • Automatic bumping of the version number in debian/changelog based on TRAVIS_BUILD_NUMBER. (#14)
      • 

    • Security repositories. Thanks to Stefan Jenkner for the initial pull request. These are additionally now enabled by default. (#15)
      • 

    • The backports repositories. (#13)
      • 

    

    • 

  • Applied #812830 and #812830 from James Clark to the Debian Archive Kit to improve the interface of various webpages it generates.
    • 

  • Updated the SSL certificate for try.diffoscope.org, a hosted version of the diffoscope in-depth and content-aware diff utility. Thanks to Bytemark for sponsoring the hardware.
    • 

  • Worked on my slides for Reproducible Builds - fulfilling the original promise of free software, to be presented at FOSSASIA '16.
    • 


My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#40, #41,  #42, #43)




LTS







This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:

    

  • "Frontdesk" duty for the week of 22nd 28th, triaging CVEs, etc.
    • 

  • Proofread announcements, etc. for the upcoming migration to wheezy-lts.
    • 

  • Issued DLA 417-1 for xdelta3 to fix a buffer overflow that allowed arbitrary code execution from input files.
    • 

  • Issued DLA 420-1 for libmatroska, correcting a heap information leak.
    • 

  • Issued DLA 428-1 for websvn fixing a cross-site scripting vulnerability.
    • 

  • Issued DLA 429-1 for pixman fixing a buffer overflow issue.
    • 

  • Issued DLA 430-1 & DLA 431-1 for libfcgi and libfcgi-perl respectfully, fixing a remote denial-of-service (DoS) vulnerability.
    • 








Uploads

    

  • redis (2:3.0.7-2)   Correcting my SOURCE_DATE_EPOCH reproducibility patch as the conditional was accidentally inverted. Thanks to Reiner Herrmann (deki).
    • 

  • disque (1.0~rc1-5)   Making the parallel SOURCE_DATE_EPOCH patch change and additionally tidying the packaging after introducing procps as a build-dependency.
    • 











RC bugs











I also filed 137 FTBFS bugs against aac-tactics, angular.js, astyle, bcftools, blacs-mpi, bogofilter, boxes, caldav-tester, ccdproc, ckeditor, coq-float, cqrlog, dasher, django-recurrence, dspdfviewer, eclipse-egit, ess, etcd, felix-latin, fio, flexml, funny-manpages, gap-atlasrep, garmin-plugin, gitlab, gnome-mines, graphicsmagick, haskell-nettle, healpy, hg-git, hunspell, hwloc, ijs, ipset, janest-core-extended, jpathwatch, kcompletion, kcompletion, keyrings.alt, kodi-pvr-hts, kodi-pvr-vdr-vnsi, libcommons-compress-java, libgnome2-wnck-perl, libkate, liblrdf, libm4ri, libnet-server-mail-perl, libsis-jhdf5-java, libspectre, libteam, libwnck, libwnckmm, libxkbcommon, lombok, lombok-patcher, mako, maven-dependency-analyzer, mopidy-mpris, mricron, multcomp, netty-3.9, numexpr, ocaml-textutils, openimageio, openttd-openmsx, osmcoastline, osmium-tool, php-guzzle, php-net-smartirc, plexus-component-metadata, polari, profitbricks-client, pyentropy, pynn, pyorbital, pypuppetdb, python-aioeventlet, python-certifi, python-hglib, python-kdcproxy, python-matplotlib-venn, python-mne, python-mpop, python-multipletau, python-pbh5tools, python-positional, python-pydot-ng, python-pysam, python-snuggs, python-tasklib, r-cran-arm, r-cran-httpuv, r-cran-tm, rjava, ros-geometry-experimental, ros-image-common, ros-pluginlib, ros-ros-comm, rows, rr, ruby-albino, ruby-awesome-print, ruby-default-value-for, ruby-fast-gettext, ruby-github-linguist, ruby-gruff, ruby-hipchat, ruby-omniauth-crowd, ruby-packetfu, ruby-termios, ruby-thinking-sphinx, ruby-tinder, ruby-versionomy, ruby-zentest, sbsigntool, scikit-learn, scolasync, sdl-image1.2, signon-ui, sisu-guice, sofa-framework, spykeutils, ssreflect, sunpy, tomcat-maven-plugin, topmenu-gtk, trocla, trocla, tzdata, verbiste, wcsaxes, whitedune, wikidiff2, wmaker, xmlbeans, xserver-xorg-input-aiptek & zeroc-icee-java.






FTP Team







As a Debian FTP assistant I ACCEPTed 107 packages: androguard, android-platform-dalvik, android-platform-development, android-platform-frameworks-base, android-platform-frameworks-native, android-platform-libnativehelper, android-platform-system-core, android-platform-system-extras, android-platform-tools-base, android-sdk-meta, apktool, armci-mpi, assertj-core, bart, bind9, caja, caldav-tester, clamav, class.js, diamond, diffoscope, django-webpack-loader, djangocms-admin-style, dnsvi, esptool, fuel-astute, gcc-6-cross, gcc-6-cross-ports, gdal, giella-core, gnupg, golang-github-go-ini-ini, golang-github-tarm-serial, gplaycli, gradle-jflex-plugin, haskell-mountpoints, haskell-simple, hurd, iceweasel, insubstantial, intellij-annotations, jetty9, juce, keyrings.alt, leptonlib, libclamunrar, libdate-pregnancy-perl, libgpg-error, libhtml5parser-java, libica, libvoikko, linux, llvm-toolchain-3.8, lombok-patcher, mate-dock-applet, mate-polkit, mono-reference-assemblies, mxt-app, node-abab, node-array-equal, node-array-flatten, node-array-unique, node-bufferjs, node-cors, node-deep-extend, node-original, node-setimmediate, node-simplesmtp, node-uglify-save-license, node-unpipe, oar, openjdk-8, openjdk-9, pg8000, phantomjs, php-defaults, php-random-compat, php-symfony-polyfill, pnetcdf, postgresql-debversion, pulseaudio-dlna, pyconfigure, pyomo, pysatellites, python-fuelclient, python-m3u8, python-pbh5tools, python-qtpy, python-shellescape, python-tunigo, pyutilib, qhull, r-cran-rjsonio, r-cran-tm, reapr, ruby-fog-dynect, scummvm-tools, symfony, talloc, tesseract, twextpy, unattended-upgrades, uwsgi, vim-command-t, win-iconv, xkcdpass & xserver-xorg-video-ast.
I additionally REJECTed 4 packages.



 
 





 
Tom Marble: is-slfc-shooting-open-source-in-the-foot

 

   Is SFLC Shooting Open Source in the Foot?
The academic article by SFLC about ZFS is troubling
and may unintentionally shoot free software licensing in the foot.
When I was at Sun (as part of the team that released the Java
Programming Language by starting the OpenJDK project) I often
heard community concerns about the CDDL license. At the time
the big complaint was about the "Choice of Venue" clause.
I got involved because Sun had developed many essential
Java libraries and distributed them under CDDL. The community
requested a more permissive license and I was able to
convince internal project leaders (and Sun's lawyers) to make
a licensing change for a handful of these projects.
And there was much rejoicing.
Based on my experience in helping Java to become open source
I came to appreciate the legal hacks on copyright which
make open source possible. It's the free software license
which uses copyright to enable sharing (vs. the default
of disabling sharing).





Open Source Licenses



And so I have appreciated many of the writings and speeches
from SFLC on the mechanisms of software freedom. I was
particularly moved by the talks about the "Freedom Box"
concept.
That's why this SFLC post on ZFS sounds so off key:
if open source works because of free software licenses
it seems weird to weaken that foundation by prioritizing
the "equity" (or intended spirit) of the license.
Allow me to mention that as I do most of my computing these days on
GNU/Linux I miss the super cool features of ZFS from Solaris. I did
try an early version of btrfs and was quite disappointed (but that's
another story).
In this happy case the source code for ZFS is available, but what
about the future, when we aren't so lucky and someone asserts in
court that the "you know, the software license was really
about the spirit of sharing and that means we are allowed
to use it -- and not be held to the pesky details as written in the license".
A lawyer I respect called this out: "Equity" has no place in US law. The point is that for lawyers software licenses work because
they have clear, written rules to guarantee the spirit is upheld; but spirit
doesn't work in front of a judge -- clear rules do.
Free and open source software has made so much progress in
all facets of life why on earth would we second guess
the licensing tools that made it possible? And why would SFLC
try to shift the spotlight (and in this case the legal burden)
to "a good-faith belief that the conduct falls within the equity
of the license". Especially given the earlier comment which clearly states
"[the combination] is inconsistent with the literal meaning of
GPLv2 section 2(b)."


Wat?




The entire raison d' tre for open source software licenses
was so that developers (and users) would have clarity and
wouldn't have to ask permission to use the software!!!
As stated elsewhere (and like I did with those Java libraries)
the easy solution is to have the ZFS copyright holder (now Oracle)
reclicense (or dual license) the code under a compatible license
(permissive or copyleft). If OpenSolaris was still a thing I might
understand some hesitancy, but why not liberate ZFS now?
So we have to wonder what could possibly be motivating this
odd "spirit of the license" position on the part of SFLC?
Fortunately charities that enjoy non-profit status are required
to make public filings of their income in something called a "Form 990".
The latest SFLC 990 I could find shows SFLC getting 78%
(or just over $5 million) from "non public support" (see page 14).
A number with "two commas" would even be interesting to
for-profit companies. Just whom is making these "donations"
and what exactly do they get in return? Apparently I'm not the only one
wondering about this question.
On one hand it's important to know if SFLC as a non-profit is, indeed,
acting in the public interest (as the IRS requires). Yet the even
bigger issue here is would "asking for a consensus about the spirit"
trump the written copyright license and set a scary precedent for open
source software in general? 
 





 
Daniel Silverstone: Kicad hacking - Intra-sheet links and ERC

 

     

  This is a bit of an odd posting since it's about something I've done but is
  also here to help me explain why I did it and thus perhaps encourage some
  discussion around the topic within the Kicad community...


Recently (as you will know if you follow this blog anywhere it is syndicated) I
have started playing with Kicad for the development of some hardware
projects I've had a desire for.  In addition, some of you may be aware that
I used to work for a hardware/software consultancy called Simtec, and there
I got to play for a while with an EDA tool called Mentor Designview.  Mentor
was an expensive, slow, clunky, old-school EDA tool, but I grew to understand
and like the workflow.
I spent time looking at gEDA and Eagle when I wanted to get back into
hardware hacking for my own ends; but neither did I really click with.  On
the other hand, a mere 10 minutes with Kicad and I knew I had found the
tool I wanted to work with long-term.
I designed the beer'o'meter project (a flow meter for the pub we are somehow
intimately involved with) and then started on my first personal surface-mount
project -- SamDAC which is a DAC designed to work with our HiFi in our
study at home.
As I worked on the SamDAC project, I realised that I was missing a very
particular thing from Mentor, something which I had low-level been annoyed by
while looking at other EDA tools -- Kicad lacks a mechanism to mark a wire as
being linked to somewhere else on the same sheet.  Almost all of the EDA tools
I've looked at seem to lack this nicety, and honestly I miss it greatly, so
I figured it was time to see if I could successfully hack on Kicad.
Kicad is written in C++, and it has been mumble mumble years since I last did
any C++, either for personal hacking or professionally, so it took a little
while for that part of my brain to kick back in enough for me to grok the
codebase.  Kicad is not a small project, taking around ten minutes to build on
my not-inconsiderable computer.  And while it beavered away building, I spent
time looking around the source code, particularly the schematic editor
eeschema.
To skip ahead a bit, after a couple of days of hacking around, I had a
proof-of-concept for the intra-sheet links which I had been missing from my
days with Mentor, and some ERC (electrical rules checking) to go alongside that
to help produce schematics without unwanted "sharp corners".
In total, I added:

    

  • an "intra-sheet link" schematic element, including:
Intra-sheet linking tool

      

    • the schematic element itself
      • 

    • loading and saving of the element (which required a bump in schematic
file format versioning)
      • 

    • UI for adding intra-sheet links to your schematic
      • 

    • netlist build support to ensure that the links found their way into
the internal netlist content.
      • 

    • 



    

  • an ERC tickybox "Strict wiring checks" and behind that I hid:
Strict wiring checks

      

    • checks for wires with dangling ends (dangles are bad)
      • 

    • junctions which have no point in being present (pointless junctions are
confusing to readers)
      • 

    • checks that intra-sheet links exist in groups of two or more on any
given net, since individual links are pointless.
      • 

    • 


I forked the Kicad mirror on Github and pushed my own branch with this
work to my Kicad fork.
All of this is meant to allow schematic capture engineers to more clearly
state their intentions regarding what they are drawing.  The intra-sheet link
could be thought of like a no-connect element, except instead of saying "this
explicitly goes nowhere" we're saying "this explicitly goes somewhere else on
this sheet, you can go look for it".
Obviously, people who dislike (or simply don't want to use) such intra-sheet
link elements can just disable that ERC tickybox and not be bothered by them
in the least (well except for the toolbar button and menu item I suppose).
Whether this work gets accepted into Kicad, or festers and dies on the
vine, it was good fun developing it and I'd like to illustrate how it could
help you, and why I wrote it in the first place:

A contrived story


  Note, while this story is meant to be taken seriously, it is somewhat
  contrived, the examples are likely electrical madness, but please just think
  about the purpose of the checks etc.


To help to illustrate the feature and why it exists, I'd like to tell you a
somewhat contrived story about Fred.  Fred is a schematic capture engineer and
his main job is to review schematics generated by his colleagues.  Fred and his
colleagues work with Kicad (hurrah) but of late they've been having a few
issues with being able to cleanly review schematics.
Fred's colleagues are not the neatest of engineers.  In particular they tend to
be quite lazy when it comes to running busses, which are not (for example)
address and data busses, around their designs and they tend to simply have
wires which end in mid-space and pick up somewhere else on the sheet.  All this
is perfectly reasonable of course, and Kicad handles it with aplomb.  Sadly
it seems quite error prone for Fred's workplace.
As an example, Fred's colleague Ben has been designing the power supply for a
particular board.  As with most power supplies, plenty of capacitors are needed
to stabilise the regulators and smooth the output.  In the example below,
the intent is that all of the capacitors are on the FOO net.
Contrived problem example 1
Sadly there's a missing junction and/or slightly misplaced label in the upper
section which means that C2 and C3 simply don't join to the FOO net.  This
could easily be missed, but the ERC can't spot it at all since there's more
than one thing on each net, so the pins of the capacitors are connected to
something.
Fred is very sad that this kind of problem can sometimes escape notice by the
schematic designer Ben, Fred himself, and the layout engineer, resulting in
boards which simply do not work.  Fred takes it upon himself to request that
the strict wiring checks ERC is made mandatory for all designs, and that the
design engineers be required to use intra-sheet link symbols when they have
signals which wander off to other parts of the sheet like FOO does in the
example.  Without any further schematic changes, strict wiring checks enabled
gives the following points of ERC concern for Ben to think about:
Contrived problem example 2
As you can see, the ERC is pointing at the wire ends and the warnings are
simply that the wires are dangling and that this is not acceptable.  This
warning is very like the pin-not-connected warnings which can be silenced with
an explicit no-connect schematic element.  Ben, being a well behaved and gentle
soul, obeys the design edicts from Fred and seeks out the intra-sheet link
symbols, clearing off the ERC markers and then adding intra-sheet links to his
design:
Contrived problem example 3
This silences the dangling end ERC check, which is good, however it results
in another ERC warning:
Contrived problem example 4
This time, the warning for Ben to deal with is that the intra-sheet links
are pointless.  Each exists without a companion to link to because
of the net name hiccough in the top section.  It takes Ben a moment to realise
that the mistake which has been made is that a junction is missing in the
top section.  He adds the junction and bingo the ERC is clean once more:
Contrived problem example 5
Now, this might not seem like much gain for so much effort, but Ben can now be
more confident that the FOO net is properly linked across his design and Fred
can know, when he looks at the top part of the design, that Ben intended for
the FOO net to go somewhere else on the sheet and he can look for it.

Why do this at all?
Okay, dropping out of our story now, let's discuss why these ERC checks are
worthwhile and why the intra-sheet link schematic element is needed.


  Note: This bit is here to remind me of why I did the work, and to hopefully
  explain a little more about why I think it's worth adding to Kicad...


Designers are (one assumes) human beings.  As humans we (and I count myself
here too) are prone to mistakes.  Sadly mistakes are often subtle and could
easily be thought of as deliberate if the right thought processes are not
followed carefully when reviewing.  Anyone who has ever done code review,
proofread a document, or performed any such activity, will be quite familiar
with the problems which can be introduced by a syntactically and semantically
valid construct which simply turns out to be wrong in the greater context.
When drawing designs, I often end up with bits of wire sticking out of
schematic sections which are not yet complete.  Sadly if I sleep between design
sessions, I often lose track of whether such a dangling wire is meant to be
attached to more stuff, or is simply left because the net is picked up
elsewhere on the sheet.  With intra-sheet link elements available, if I had
intended the latter, I'd have just dropped such an element on the end of the
wire before I stopped for the day.
Also, when drawing designs, I sometimes forget to label a wire, especially if
it has just passed through a filter or current-limiting resistor or similar.
As such, even with intra-sheet link elements to show me when I mean for a net
to go bimbling off across the sheet, I can sometimes end up with unnamed nets
whose capacitors end up not used for anything useful.  This is where the ERC
comes in.
By having the ERC complain if a wire dangles -- the design engineer won't
forget to add links (or check more explicitly if the wire is meant to be
attached to something else).  By having junctions which don't actually link
anything warned about, the engineer can't just slap a junction blob down on the
end of a wire to silence that warning, since that doesn't mean anything to a
reviewer later down the line.  By having the ERC warn if a net has exactly one
intra-sheet link attached to it, missing net names and errors such as that
shown in my contrived example above can be spotted and corrected.
Ultimately this entire piece of work is about ensuring that the intent of
the design engineer is captured clearly in the schematic.  If the design
engineer meant to leave that wire dangling because it's joining to another
bit of wire elsewhere on the sheet, they can put the intra-sheet links in to
show this.  The associated ERC checks are there purely to ensure that the
validation of this intent is not bypassed accidentally, or deliberately, in
order to make the use of this more worthwhile and to increase the usefulness
of the ERC on designs where signals jump around on sheets where wiring them up
directly would just create a mess. 
 




28 February 2016



 
Ian Campbell: Hotswapping a failed RAID device

 

   Recently I started getting SMART warnings from on of the disks in my home NAS
(a QNAP TS-419P II armel/kirkwood device running Debian Jessie):

Device: /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 [SAT], Self-Test Log error count increased from 0 to 1


Meaning it was now time to switch out that disk from the RAID5 array.
Since everytime this happens I have to go and lookup again what to do I've
decided to write it down this time.
I configure SMART to talk about devices by-id (giving me their name and model
number) so first I needed to figure out what the kernel was calling this device
(although mdadm is happy with the by-id path, various other bits are not):

# readlink /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 
../../sdd


Next I needed to mark the device as failed in the array:

# mdadm --detail /dev/md0 
/dev/md0:
[...]
          State : clean 
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
[...]
    Number   Major   Minor   RaidDevice State
       5       8       48        0      active sync   /dev/sdd
       1       8       32        1      active sync   /dev/sdc
       6       8       16        2      active sync   /dev/sdb
       4       8        0        3      active sync   /dev/sda
# mdadm --fail /dev/md0 /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 
mdadm: set /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 faulty in /dev/md0
# mdadm --detail /dev/md0 
/dev/md0:
[...]
 Active Devices : 3
Working Devices : 3
 Failed Devices : 1
  Spare Devices : 0
[...]
Number   Major   Minor   RaidDevice State
   0       0        0        0      removed
   1       8       32        1      active sync   /dev/sdc
   6       8       16        2      active sync   /dev/sdb
   4       8        0        3      active sync   /dev/sda
   5       8       48        -      faulty   /dev/sdd


If it had been the RAID subsystem rather than SMART monitoring which had first
spotted the issue then this would have happened already (and I would had
received a different mail from the RAID checks instead of SMART).
Once the disk is marked as failed then actually remove it from the array:

# mdadm --remove /dev/md0 /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 
mdadm: hot removed /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 from /dev/md0


And finally tell the kernel to delete the device:

# echo 1 > /sys/block/sdd/device/delete 


At this point I can physically swap the disks.
At this point I noticed there were some interesting messages in dmesg, either
from the echo to the delete node in sysfs or from the physical switch of the
disks:

[1779238.656459] md: unbind<sdd>
[1779238.659455] md: export_rdev(sdd)
[1779258.686720] sd 3:0:0:0: [sdd] Synchronizing SCSI cache
[1779258.700507] sd 3:0:0:0: [sdd] Stopping disk
[1779259.377589] ata4.00: disabled
[1779371.126202] ata4: exception Emask 0x10 SAct 0x0 SErr 0x180000 action 0x6 frozen
[1779371.133740] ata4: edma_err_cause=00000020 pp_flags=00000000, SError=00180000
[1779371.141003] ata4: SError:   10B8B Dispar  
[1779371.145309] ata4: hard resetting link
[1779371.468708] ata4: SATA link down (SStatus 0 SControl 300)
[1779371.474340] ata4: EH complete
[1779557.416735] ata4: exception Emask 0x10 SAct 0x0 SErr 0x4010000 action 0xe frozen
[1779557.424356] ata4: edma_err_cause=00000010 pp_flags=00000000, dev connect
[1779557.431264] ata4: SError:   PHYRdyChg DevExch  
[1779557.436008] ata4: hard resetting link
[1779563.357089] ata4: link is slow to respond, please be patient (ready=0)
[1779567.449096] ata4: SRST failed (errno=-16)
[1779567.453316] ata4: hard resetting link


I wonder if I should have used another method to detach the disk, perhaps
poking the controller rather than the disk (which rang a vague bell in my
memory from last time this happened) but in the end the disk is broken and the
kernel seems to have coped so I'm not too worried about it.
It looked like the new disk had already been recognised:

[1779572.593471] scsi 3:0:0:0: Direct-Access     ATA      HGST HDN724040AL A5E0 PQ: 0 ANSI: 5
[1779572.604187] sd 3:0:0:0: [sdd] 7814037168 512-byte logical blocks: (4.00 TB/3.63 TiB)
[1779572.612171] sd 3:0:0:0: [sdd] 4096-byte physical blocks
[1779572.618252] sd 3:0:0:0: Attached scsi generic sg3 type 0
[1779572.626754] sd 3:0:0:0: [sdd] Write Protect is off
[1779572.631771] sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[1779572.632588] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[1779572.665609]  sdd: unknown partition table
[1779572.671522] sd 3:0:0:0: [sdd] Attached SCSI disk
[1779855.362331]  sdd: unknown partition table


So I skipped trying to figure out how to perform a SCSI rescan and went
straight to identifying that the new disk was called:

/dev/disk/by-id/ata-HGST_HDN724040ALE640_PK1338P4GY8ENB


and then tried to do a SMART conveyancing self-test with:

# smartctl -t conveyance /dev/disk/by-id/ata-HGST_HDN724040ALE640_PK1338P4GY8ENB


But this particular drive seems to not support that, so I went straight to
editing /etc/smartd.conf to replace the old disk with the new one and:

# service smartmontools reload


With all that I was ready to add the new disk to the array:

# mdadm --add /dev/md0 /dev/disk/by-id/ata-HGST_HDN724040ALE640_PK1338P4GY8ENB
mdadm: added /dev/disk/by-id/ata-HGST_HDN724040ALE640_PK1338P4GY8ENB
# mdadm --detail /dev/md0 
/dev/md0:
[...] 
      State : clean, degraded, recovering 
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1
[...]
 Rebuild Status : 0% complete
[...]
Number   Major   Minor   RaidDevice State
   5       8       48        0      spare rebuilding   /dev/sdd
   1       8       32        1      active sync   /dev/sdc
   6       8       16        2      active sync   /dev/sdb
   4       8        0        3      active sync   /dev/sda
# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdd[5] sda[4] sdb[6] sdc[1]
      5860538880 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
      [>....................]  recovery =  0.0% (364032/1953512960) finish=1162.4min speed=28002K/sec


So now all that was left was to wait about 20 hours (with fingers crossed a
second disk didn't die! spoiler: it didn't) 
 




24 February 2016



 
Arturo Borrero Gonz lez: Playing with the HA stack in Debian strecth

 

   

Given the HA stack is now in very good shape in Debian stretch [testing by now] we can start playing a bit with these tools.

There are 2 main points in doing so now:
  1. reporting bugs
  2. systemd integration
The last HA stack was in Debian wheezy, which didn't include systemd. Good to take that into account.

My hardware: 2 virtual machines with 1 NIC each. I called them node01 and node02.
Packages
Using aptitude. install debian packages corosync, pacemaker, pacemaker-cli-utils and crmsh.

The installation of the packages is enough for corosync to start working.

$ sudo systemctl status corosync
  corosync.service - Corosync Cluster Engine
 Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
 Active: active (running) since mar 2016-02-09 11:01:24 CET; 3h 7min ago
 Main PID: 14610 (corosync)
 CGroup: /system.slice/corosync.service
  14610 /usr/sbin/corosync -f
$ sudo crm status
Last updated: Tue Feb  9 14:11:33 2016  Last change: Tue Feb  9 11:01:48 2016 by hacluster via crmd on node01
Stack: corosync
Current DC: node01 (version 1.1.14-70404b0) - partition WITHOUT quorum
1 node and 0 resources configured
Online: [ node01 ]

Full list of resources:

Basic configurationYou will need to edit the /etc/corosync/corosync.conf file:

totem  
 version: 2
 cluster_name: lbcluster
 transport: udpu
 interface  
 ringnumber: 0
 bindnetaddr: 10.0.0.0
 mcastaddr: 239.255.1.1
 mcastport: 5405
 ttl: 1
  
 
quorum  
 provider: corosync_votequorum
 two_node: 1
 
nodelist  
 node  
 ring0_addr: 10.0.0.1
 name: node01
 nodeid: 1
  
 node  
 ring0_addr: 10.0.0.2
 name: node02
 nodeid: 2
  
 
logging  
 to_logfile: yes
 logfile: /var/log/corosync/corosync.log
 to_syslog: yes
 timestamp: on
 

Note that if you choose the proper bindnetaddr directive, you can use the exact same file for all the nodes (see corosync.conf(5))

System, daemons, servicesYou may want to check that the corosync and pacemaker services are auto-started at boot time by systemd.
Do a reboot and check it:

$ sudo systemctl status corosync   grep active
 Active: active (running) since mi  2016-02-24 18:54:37 CET; 3min 11s ago
$ sudo systemctl status pacemaker   grep active
 Active: active (running) since mi  2016-02-24 18:54:38 CET; 3min 10s ago

Also, check that no firewall blocks the communication between both nodes.

Now, the status of the cluster should show both nodes working together (with no resources):

$ sudo crm status
Last updated: Wed Feb 24 19:01:13 2016  Last change: Wed Feb 24 18:43:32 2016 by hacluster via crmd on node01
Stack: corosync
Current DC: node02 (version 1.1.14-70404b0) - partition with quorum
2 nodes and 0 resources configured
Online: [ node01 node02 ]
Full list of resources:

ResourcesNow, you can start playing with the cluster resources. Below is an example of a virtual IPv6 address:

$ sudo crm configure primitive test ocf:heartbeat:IPv6addr params ipv6addr="fe00::200" cidr_netmask="64" nic="eth0" 

Which should give this cluster status:

$ sudo crm status
Last updated: Wed Feb 24 19:13:46 2016  Last change: Wed Feb 24 19:11:45 2016 by root via cibadmin on node02
Stack: corosync
Current DC: node02 (version 1.1.14-70404b0) - partition with quorum
2 nodes and 1 resource configured
Online: [ node01 node02 ]
Full list of resources:
 test  (ocf::heartbeat:IPv6addr):  Started node01

Congratulations, now you have a working HA cluster with a virtual IP address, using an active/backup approach.

Please, report any bug you may find. 
 





 
Norbert Preining: Switching from iOS to Android

 

     This article starts a series of blogs on my ventures into Android land after having used iOS devices for 6 years. I have never been a strong believer in the Apple ecosystem, and my trials to convince myself to MacBooks and MacPros always ended with installing Linux on them, so I guess I wasn t really the ideal target for iOS. Thus, my iOS devices were also permanently in jailbroken state, otherwise I would feel amputated. But as much as I disliked the lock-in and closed environment of the iOS world, it was from the user perspective surprisingly well done and smooth. So it was with a certain level of tension when I finally switched to Linux^WAndroid.
android-ios
If you don t want to read on, here the preliminary conclusion: Why didn t I do it earlier!  But before we go into details, let me start with my background:
History of my devices
Originally I was a big opponent of smart phones and preferred the Unix-way, one device for one thing. So I had a normal phone and (various) Palm devices (Tungsten X, Tungsten C, and above all my beloved Handera TRGpro). I loved the Palm world and considered it superior to the then smart phone world, until I came to Japan, where the challenge of a proper input method for Japanese and proper Japanese support posed a big hurdle. The Palm devices had a stick and written input fields, but Japanese input was practically impossible and a huge pain. Searching for a word in Japanese was more hurdle then looking it up in a printed dictionary.
In addition, I needed a phone in Japan, so I plunged into the smart phone world and got myself a iPhone (3g). What a world did open for me: easy typing of Japanese, dictionaries, on-the-fly translation, woooow! And above all, I discovered my most beloved and till now one of my most important programs: Flashcards Deluxe. Thanks, and I have to say to 80% thanks to this program my Japanese learning speed has accelerated considerably. There is nothing more important for me than getting drilled in a systematic way.
But I derail, anyway, having Flashcard Deluxe on the iPhone within rather short time I had about 10000 or more flashcards created, and moving on to a different architecture (Android) was for quite some time practically impossible without loosing years of statistics and learning, so I renewed my contract after 2 years together with a iPhone 4s. Another two years passed, and these years brought an Android version of Flashcards Deluxe, as well as Dropbox syncing, so I had no excuse anymore to remain in iOS land, wouldn t it be for an iPhone 5s that was passed to me near the end of my forth year, so I again extended the contract for two years.
Finally, after 6 years of iPhone devices, this January I finally decided it is time to switch to Android. After lots of thinking, comparing, and requesting advice from good friends with more experience in the smart phone market I went for a Google Nexus 6p.
Google Nexus 6p
I will not repeat the specs of this phone as they are widely available on the net. My original plan was a Samsung S6, but after consultation with an expert I decided for an original Google phone for better security support. That left me with the option between a Nexus 5x and 6p, and due to prize differences (prizes of mobiles are ridiculously strange in Japan) I went for the 6p instead of the 5x. One point that made the decision for this slight too big device easy was the fact that it uses a great AMOLED display.
Moving the data
Since I was using Google Calendar and Google Contacts already on the iPhone, moving to the Android phone was far less a hassle than I thought. My contacts and events showed up without a hiccup. Most of the usual apps are nowadays available on both iOS and Android, so the most difficult thing was remembering all the passwords to log into the applications again (SNS like G+, FB, Twitter etc). The same is more or less true for messengers of all kinds (Line, WhatsApp, Threema, etc), but here one is advised to check with the respective web site first to make sure one does not loose all of the important data. Line for example is a stupid ***** that deletes all previous chats on the old phone and does not make them available on the new one. WhatsApp can be converted with a special conversion program. Threema, too allows for transfer of ids.
Move of applications
After that came the hunt for replacement applications for those that are not available as is on Android:
Mail
At first like probably everyone I used the shipped GMail program. It might be good for Google Mail accounts, but for anything else it is just a real pain. Thus, I have searched a bit and finally settled (for now) for K-9 Mail: it is open source, open development, feature rich, and more a hackers  type email program, perfectly suited to me.
There is a commercial variant called K@-Mail that says that it improves the user interface and some usability items as well as features, but I didn t see much of an advantage over the original version (which is completely free) and in fact some of my accounts didn t work at all. So I remain with K9 Mail and I think this is a good decision.
Calendar
Managing Calendars is one of the most important task for me. I have been a fervent supporter of DateBk4, DateBk 5, and DateBk 6 on the original Palm series, and when I left the Palm World it was with great pain that I had to loose DateBk. Not only because it was a simply fantastic calendar program that allowed me to keep track of all my climbing routes, festivities, in a much more advanced way than any other Calendering application, but also because the programmer of the DateBk series is running the Dewar Wildlife Trust, a Gorilla rescue group and a lot of the money he makes from the app sales is going to rescue Gorillas.
With the switch to iOS this option was gone, and I first used the built-in calendar application (which is so weak) and later and for long time Pocket Informant Pro. This is a very good program and probably the only one that can compete with DateBk with respect to functionality and usefulness.
During the time of me being locked in in iOS I realized that the world has moved on and a new version of DateBk series for Java was developed, called Pimlical. First only available on Windows, it became later available also on Android and Linux, too.
The following screen shot puts Pimlical on the left, and Pocket Informant on the right. I will write a more detailed comparison in future, in short: PInformant is more streamlined and polished, Pimlical has more configuration option. Practically everything can be adjusted to one s need, and in addition there is also a Desktop application that sync either with Google and the phone, or only with the phone if you want to live off the grid.
pimlical-pinformant
So nowadays on Android I have both Pocket Informant Pro as well as Pimlical, but after a short time I have now switched practically exclusively to Pimlical.
Notes
Here there is pain   HUGE PAIN!!! iOS has an excellent applications for notes, called simply Notebooks. This little pearl was my work horse for everything (more or less) memorable. From poems and song texts to bus time tables, from PDF to GIFs, from MarkDown to HTML, everything could be saved into Notebooks, displayed, edited, ordered. And above all it had automatic background sync with Dropbox. So I could drop new files into the respective sub-folder of my Dropbox folder and could be sure I have the files available on my phone when I leave for a trip. And there is a huge bag of features that I haven t even tapped into!
Android is unfortunately not on the list of supported architectures of Notebooks. So I searched far and wide, and without any success. There are all kind of notes, flash colors, overly simple, fast and slow, stylish and plain, but none of them did even provide half of the features of Notebooks. None, not even half.
I still hope I might find the ultimate notes application, or even better would be an Android version of the original Notebooks application (but this is not high on the developers todo list), but for now I am in despair  
The Rest
As I said, most apps are nowadays available on both platforms, so there is not much more to do than download the respective Android app and log in again. That worked very nice across practically all apps.
Things I don t like (i.e., which are broken!) on Android
Although a very convenient system and perfectly made to fit my taste, there are some things that are a huge pain (and a big shame on Google to not being able to fix that for long time!):
Japanese fonts when the device is in English interface language
In case you are a foreigner living in Japan and want your Android phone in English, but still read emails, news, etc in Japanese, then Android provides you with the worst, namely Chinese fonts:
chinese-japanese-mix
This is a well known problem and I have blogged about fixing the very same problem on Linux (Debian), and the solution is a simple reshuffling in the fontconfig configuration files. There is even an  application for it in the Google store, Kanji Fix, but it needs a rooted device (which I haven t done till now   my failure!). I can only hope that Google fixes this completely stupid problem in a future version.
The  Me  problem
Another of these beasty problems: The Android Contacts application has an entry for  Me , which unfortunately, no idea why, cannot be linked with my normal  me  in the list of contacts. There are reports all over the Internet, strange suggestions, and no real solution. Again, a simple thing that should work   but doesn t.
Invisible Images folder in MTP mode
A more annoying problem is that the camera folder under Photos does not show up when connecting the device in MTP mode to my computer, and as consequence me being unable to copy photos from the device to my computer.
The solution I am using at the moment is moving the photos with a file manager to a new folder which is visible during MTP communcation, and copy the photos from there.
But this, too, should be something trivial, but alas, despite a lot of posts on the internet I couldn t find a proper solution.
Google Music
As written somewhere else, Google Music has switched from 5 star system to up/down system, which is a huge pain and PITA.
Things I do like (or I discovered) on Android
There are some things I haven t been used/tried on iOS   they might be possible   which I really like:
Yubikey Neo support
I will write about this in a different blog, but nowadays I have my GPG keys on an hardware token (Yubikey Neo) and the application OpenKeychain on Android works nicely with both K9 Mail and via NFC with my Yubikey. That is a great tool!
Bluetooth streaming
Bluetooth on iOS devices was always a bit broken for me, so connecting my phone to my old car radio I needed radio transmitter that was connected to the cable port of the iPhone. With Android I use a Bluetooth Radio device (receives data via bluetooth, and sends music out via radio waves for a car stereo to receive them). Now if my monthly data limit wouldn t be that low  
Debian on Android
Yes, you can have a full Debian system running in your terminal on Android. There are several applications providing this feature, and I am rather surprised how smooth it works.
Conclusion
My preliminary conclusion is that the switch to Android at this time was perfectly timed, and from the technological side I should have done much earlier. In future blogs I will discuss particular instances of this transition in more details.
If you have any suggestion for me, in particular for a good notes taking application, please let me know! 
 




Next.


Previous.