A new version, now at 0.2.0, of RcppCCTZ is now on CRAN. And it brings a significant change: windows builds! Thanks to Dan Dillon who dug deep enough into the libc++ sources from LLVM to port the std::get_time() function that is missing from the 4.* series of g++. And with Rtools being fixed at g++-4.9.3 this was missing for us here. Now we can parse dates for use by RcppCCTZ on Windows as well. That is important not only for RcppCCTZ but also particularly for the one package (so far) depending on it: nanotime. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. It requires only a proper C++11 compiler and the standard IANA time zone data base which standard Unix, Linux, OS X, ... computers tend to have in /usr/share/zoneinfo -- and for which R on Windows ships its own copy we can use. RcppCCTZ connects this library to R by relying on Rcpp. The RcppCCTZ page has a few usage examples, as does the post announcing the previous release. The changes in this version are summarized here:

Changes in version 0.2.0 (2017-01-08)

• Windows compilation was enabled by defining OFFSET() and ABBR() for MinGW (#10 partially addressing #9)
• Windows use completed with backport of std::get_time from LLVM's libc++ to enable strptime semantics (Dan Dillon in #11 completing #9)
• Timezone information on Windows is supplied via R's own copy of zoneinfo with TZDIR set (also #10)
• The interface to formatDouble was cleaned up

We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Finding a Libc for tiny embedded ARM systems You'd think this problem would have been solved a long time ago. All I wanted was a C library to use in small embedded systems -- those with a few kB of flash and even fewer kB of RAM. Small system requirements A small embedded system has a different balance of needs:

• Stack space is limited. Each thread needs a separate stack, and it's pretty hard to move them around. I'd like to be able to reliably run with less than 512 bytes of stack.
• Dynamic memory allocation should be optional. I don't like using malloc on a small device because failure is likely and usually hard to recover from. Just make the linker tell me if the program is going to fit or not.
• Stdio doesn't have to be awesomely fast. Most of our devices communicate over full-speed USB, which maxes out at about 1MB/sec. A stdio setup designed to write to the page cache at memory speeds is over-designed, and likely involves lots of buffering and fancy code.
• Everything else should be fast. A small CPU may run at only 20-100MHz, so it's reasonable to ask for optimized code. They also have very fast RAM, so cycle counts through the library matter.

Available small C libraries I've looked at:

• Clibc. This targets embedded Linux systems, and also appears dead at this time.
• musl libc. A more lively project; still, definitely targets systems with a real Linux kernel.
• dietlibc. Hasn't seen any activity for the last three years, and it isn't really targeting tiny machines.
• newlib. This seems like the 'normal' embedded C library, but it expects a fairly complete "kernel" API and the stdio bits use malloc.
• avr-libc. This has lots of Atmel assembly language, but is otherwise ideal for tiny systems.
• pdclib. This one focuses on small source size and portability.

Current AltOS C library We've been using pdclib for a couple of years. It was easy to get running, but it really doesn't match what we need. In particular, it uses a lot of stack space in the stdio implementation as there's an additional layer of abstraction that isn't necessary. In addition, pdclib doesn't include a math library, so I've had to 'borrow' code from other places where necessary. I've wanted to switch for a while, but there didn't seem to be a great alternative. What's wrong with newlib? The "obvious" embedded C library is newlib. Designed for embedded systems with a nice way to avoid needing a 'real' kernel underneath, newlib has a lot going for it. Most of the functions have a good balance between speed and size, and many of them even offer two implementations depending on what trade-off you need. Plus, the build system 'just works' on multi-lib targets like the family of cortex-m parts. The big problem with newlib is the stdio code. It absolutely requires dynamic memory allocation and the amount of code necessary for 'printf' is larger than the flash space on many of our devices. I was able to get a cortex-m3 application compiled in 41kB of code, and that used a smattering of string/memory functions and printf. How about avr libc? The Atmel world has it pretty good -- avr-libc is small and highly optimized for atmel's 8-bit avr processors. I've used this library with success in a number of projects, although nothing we've ever sold through Altus Metrum. In particular, the stdio implementation is quite nice -- a 'FILE' is effectively a struct containing pointers to putc/getc functions. The library does no buffering at all. And it's tiny -- the printf code lacks a lot of the fancy new stuff, which saves a pile of space. However, much of the places where performance is critical are written in assembly language, making it pretty darn hard to port to another processor. Mixing code together for fun and profit! Today, I decided to try an experiment to see what would happen if I used the avr-libc stdio bits within the newlib environment. There were only three functions written in assembly language, two of them were just stubs while the third was a simple ultoa function with a weird interface. With those coded up in C, I managed to get them wedged into newlib. Figuring out the newlib build system was the only real challenge; it's pretty awful having generated files in the repository and a mix of autoconf 2.64 and 2.68 version dependencies. The result is pretty usable though; my STM 32L discovery board demo application is only 14kB of flash while the original newlib stdio bits needed 42kB and that was still missing all of the 'syscalls', like read, write and sbrk. Here's gitweb pointing at the top of the tiny-stdio tree: gitweb And, of course you can check out the whole thing

git clone git://keithp.com/git/newlib

'master' remains a plain upstream tree, although I do have a fix on that branch. The new code is all on the tiny-stdio branch. I'll post a note on the newlib mailing list once I've managed to subscribe and see if there is interest in making this option available in the upstream newlib releases. If so, I'll see what might make sense for the Debian libnewlib-arm-none-eabi packages.

So, once again I had a box that had been installed with the kind-of-wrong Debian architecture, in this case, powerpc (32 bit, bigendian), while I wanted ppc64 (64 bit, bigendian). So, crossgrade time. If you want to follow this, be aware that I use sysvinit. I doubt this can be done this way with systemd installed, because systemd has a lot more dependencies for PID 1, and there is also a dbus daemon involved that cannot be upgraded without a reboot. To make this a bit more complicated, ppc64 is an unofficial port, so it is even less synchronized across architectures than sid normally is (I would have used jessie, but there is no jessie for ppc64). Step 1: Be Prepared To work around the archive synchronisation issues, I installed pbuilder and created 32 and 64 bit base.tgz archives:

pbuilder --create --basetgz /var/cache/pbuilder/powerpc.tgz
pbuilder --create --basetgz /var/cache/pbuilder/ppc64.tgz \
    --architecture ppc64 \
    --mirror http://ftp.ports.debian.org/debian-ports \
    --debootstrapopts --keyring=/usr/share/keyrings/debian-ports-archive-keyring.gpg \
    --debootstrapopts --include=debian-ports-archive-keyring

Step 2: Gradually Heat the Water so the Frog Doesn't Notice Then, I added the sources to sources.list, and added the architecture to dpkg:

deb [arch=powerpc] http://ftp.debian.org/debian sid main
deb [arch=ppc64] http://ftp.ports.debian.org/debian-ports sid main
deb-src http://ftp.debian.org/debian sid main
dpkg --add-architecture ppc64
apt update

Step 3: Time to Go Wild

apt install dpkg:ppc64

Obviously, that didn't work, in my case because libattr1 and libacl1 weren't in sync, so there was no valid way to install powerpc and ppc64 versions in parallel, so I used pbuilder to compile the current version from sid for the architecture that wasn't up to date (IIRC, one for powerpc, and one for ppc64). Manually installed the libraries, then tried again:

apt install dpkg:ppc64

Woo, it actually wants to do that. Now, that only half works, because apt calls dpkg twice, once to remove the old version, and once to install the new one. Your options at this point are

apt-get download dpkg:ppc64
dpkg -i dpkg_*_ppc64.deb

or if you didn't think far enough ahead, cursing followed by

cd /tmp
ar x /var/cache/apt/archives/dpkg_*_ppc64.deb
cd /
tar -xJf /tmp/data.tar.xz
dpkg -i /var/cache/apt/archives/dpkg_*_ppc64.deb

Step 4: Automate That Now, I'd like to get this a bit more convenient, so I had to repeat the same dance with apt and aptitude and their dependencies. Thanks to pbuilder, this wasn't too bad. With the aptitude resolver, it was then simple to upgrade a test package

aptitude install coreutils:ppc64 coreutils:powerpc-

The resolver did its thing, and asked whether I really wanted to remove an Essential package. I did, and it replaced the package just fine. So I asked dpkg for a list of all powerpc packages installed (since it's a ppc64 dpkg, it will report powerpc as foreign), massage that into shape with grep and sed, and give the result to aptitude as a command line. Some time later, aptitude finished, and I had a shiny 64 bit system. Crossgrade through an ssh session that remained open all the time, and without a reboot. After closing the ssh session, the last 32 bit binary was deleted as it was no longer in use. There were a few minor hiccups during the process where dpkg refused to overwrite "shared" files with different versions, but these could be solved easily by manually installing the offending package with

dpkg --force-overwrite -i ...

and then resuming what aptitude was doing, using

aptitude install

So, in summary, this still works fairly well.

When I saw this video or this video I thought of this article. When I feel part of a tightly coordinated and synchronized team I feel proud for the achievements of the team as a whole, which I see as bigger than what I could have achieved alone. I also don't feel at risk of taking bad decisions. I feel less responsible. If I do what I'm told, I can't be blamed for doing the wrong things. I find it relaxing, every once in a while, to not have to be in charge. I guess this could be part of the allure of a totalitarian regime: being freed from the burden of growing up Thinking about this, reading those articles about romantic relationships, I see quite a bit of parallels also with organising cooperation and teamwork. It looks like I ended up making parallels between Polyamory, Anarchism, and Free Software again. If you think there should traditionally be also a mention of BDSM, go back to "I find it relaxing, every once in a while, to not have to be in charge".

Every year since 2012, I typeset the problems and solutions for the German math event Tag der Mathematik, which is organized by the Zentrum f r Mathematik and reaches 1600 students from various parts of Germany. For that, I often reach to the LaTeX drawing package TikZ, and I really like the sober aesthetics of a nicely done TikZ drawing. So mostly for my own enjoyment, I collect the prettiest here. On a global scale they are still rather mundane, and for really impressive and educating examples, I recommend the TikZ Gallery.

Big news: RcppTOML now works on Windows too! This package had an uneventful 2016 without a single update. Release 0.0.5 had come out in late 2015 and we had no bugs or issues to fix. We use the package daily in production: a key part of our parameterisation is in TOML files In the summer, I took one brief stab at building on Windows now that R sports itself a proper C++11 compiler on Windows too. I got stuck over the not-uncommon problem of incomplete POSIX and/or C++11 support with MinGW and g++-4.9. And sadly ... I appears I wasn't quite awake enough to realize that the missing functionality was right there exposed by Rcpp! Having updated that date / datetime functionality very recently, I was in a better position to realize this when Devin Pastoor asked two days ago. I was able to make a quick suggestion which he tested, which I then refined ... here we are: RcppTOML on Windows too! (For the impatient: CRAN has reported that it has built the Windows binaries, they should hit mirrors such as this CRAN package for RcppTOML shortly.) So what is this TOML thing, you ask? A file format, very suitable for configurations, meant to be edited by humans but read by computers. It emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. On small typos you get parse errors, rather than silently corrupted garbage. Much preferable to any and all of XML, JSON or YAML -- though sadly these may be too ubiquitous now. But TOML is making good inroads with newer and more flexible projects. The Hugo static blog compiler is one example; the Cargo system of Crates (aka "packages") for the Rust language is another example. The new release updates the included cpptoml template header by Chase Geigle, brings the aforementioned Windows support and updates the Travis configuration. We also added a NEWS file for the first time so here are all changes so far:

Changes in version 0.1.0 (2017-01-05)

• Added Windows support by relying on Rcpp::mktime00() (#6 and #8 closing #5 and #3)
• Synchronized with cpptoml upstream (#9)
• Updated Travis CI support via newer run.sh

Changes in version 0.0.5 (2015-12-19)

• Synchronized with cpptoml upstream (#4)
• Improved and extended examples

Changes in version 0.0.4 (2015-07-16)

• Minor update of upstream cpptoml.h
• More explicit call of utils::str()
• Properly cope with empty lists (#2)

Changes in version 0.0.3 (2015-04-27)

• First CRAN release after four weeks of initial development

Courtesy of CRANberries, there is a diffstat report for this release. More information and examples are on the RcppTOML page. Issues and bugreports should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I apologize to all potential readers of this blog for not writing a comprehensive Introduction post with details of the project I am taking part in during my internship, as well as some story about how I ended up there. Let me just say that I was a Debian user for years when I discovered it is taking part in Outreachy as one of organisations. Their Reproducible Builds effort has a noble goal and a bunch of great people behind it - I had no chances not to get excited by it. Looking for a place where my skills could be of any use, I discovered diffoscope - the tool for in-depth comparassion of files, archives etc. My mentor, Mattia Rizzolo, supported my decision to work on it, so now I am concentrating my efforts on improving diffoscope. As my first steps, I am doing small (but hopefully still somewhat important) job of fixing existing bugs. It helps me to better understand how diffoscope works, as well as introduces me to the workflow of opensource development. During December, I have done several small contributions, mostly fixing bugs.

Test data and jessie-backports First of them could be somewhat called cleaning up after my own mistake, although that mistake wasn t trivial. During the application period, I have fixed a bug with diffoscope failing while comparing symlinks to directory. That was a small change, but I included some tests for that case anyway. And that actually caused problems. With these tests, I included test data: two folders with symlinks. All was good in unstable version of Debian, but in jessie-backports, that commit caused build to fail. After some digging, I discovered the problem was caused by build process including copying that data. That was done using shutils Python module, and older version of that module, included in jessie, could not handle copying symlinks to directory properly. Thanks to my mentor for giving me a hint on how to resolve this: using temporary folders and creating these symlinks at runtime. That way, we ensured tests run without problems during build process on jessie. What have I learned: A great deal, actually. I spent too much time on that one, but I learned how to build packages, what happens during dpkg-buildpackage run and what debhelper tools are for. I also learned a bit about what chroot is and how to use it for testing.

ICC profile files and file type recognizing regexp Another one was also about failing tests and, therefore, failing build. Failing tests were all due to ICC files were not recognized by diffoscope. Turned out libmagic got an update which changed the description of ICC profile files. Diffoscope was relying on regexp applied to file type description to recognize the file, so I changed regexp to reflect the changes in libmagic. What have I learned: How diffoscope recognizes file types. Got me thinking: maybe there is a better way? That regexp-based approach is doomed to cause problems with every file type description change. I have this question still lingering in my mind - maybe I will come up with an idea later.

Order-like difference in text files Next, I decided to do something a bit bigger and fullfilled a feature request. That request was for detecting order-like difference in text files (when files has the same lines, but in different order). I did it by collecting added and removed lines in diff output in lists, sorting and then comparing them. Sadly, I forgot about one particular case - when one of the files is missing the newline at the end of file. I was kindly reminded of that quite soon in comments on the bug-tracker (thanks danielsh!) and have already fixed that. I also recieved feedback on how better implement it deeper in the diffoscope - not using the results of diff, but rather comparing sum of hashes of the lines directly in the difference module. I am yet to try that. What have I learned: That a call to diff is actually the slowest part of the diffoscope run when done on two big text files. Could it help somehow in speeding it up? I don t know yet. I also learned to comment on bugs in Debian bugtracker and was surprised by how much feedback I got. Thanks to my mentor for pushing me to do that - I definetely need to overcome my fear of communications to be more effective!

Random FTBFS There was also a very nasty bug that caused diffoscope to fail to be built from source randomly, failing with non-informative Fatal Python error: deallocated None. It already seemed strange when it was first reported; It got only more strange when suddenly that bug ceased to be reproducible. We hoped that would mean that bug was caused by some external tool, and was fixed there. Turns out it was not that easy. I tested this on two separate computers and on virtual machine; I used different versions of diffoscope. Well. Seems like that bug is still somehow tied to diffoscope version and not some external tool version - I still can do git checkout 64 and be able to reproduce the bug (still randomly, though). Although I spent quite a lot of time on that one, the only result was the information about connection between bug apperances and diffoscope version. I still wasn t able to get to the root of the problem - hopefully, someone else will be able to, given the information I found. What have I learned: git-bisect! Thanks to my friend for pointing me to it, that tool came handy in that situation. Also, got some experience in catching nasty bugs like that (pity that no experience in squashing them). I had some extra time commitements in December, one of them (Reproducible Builds Summit II) connected to my internship and one (my exam session in university) not. In January, I should be able to allocate more time to that work - I hope it will help me achieve more significant results. Many thanks to Mattia Rizzolo, Chris Lamb, Holger Levsen and all other folks of Reproducible Builds project - I cannot stress enough how important your support is to me. Wish you all a great 2017!

Review: Secrets of Productive People, by Mark Forster

Publisher:	Teach Yourself
Copyright:	2015
ISBN:	1-4736-0885-6
Format:	Kindle
Pages:	289

Regular readers of my reviews will know that Mark Forster is my favorite writer on time management and productivity. That's mostly because of his flexible toolkit approach that talks about theory and overall goals and then describes multiple ways to get there, rather than presenting a single system that will solve all your problems. There are a lot of writers who explain productivity tips and tricks or describe systems that work for them. There are fewer who can explain why those tricks work (and why they sometimes don't work), and even fewer who can put them into a meaningful analytical framework for thinking about productivity. Forster has several books, but they're a mixed bag. His clearest and most coherent book prior to this one was Do It Tomorrow. Secrets of Productive People is organized differently, chopped up in to small bite-sized chunks with synopses to an extent that it felt a bit choppy to me, but apart from that it's the closest I've seen to an updating of Do It Tomorrow. His other books can be slight (The Pathway to Awesomeness) or downright weird (How to Make Your Dreams Come True). I still have a soft spot for Do It Tomorrow, and I like how it was organized a bit better than this one, but I think Secrets of Productive People has become my new recommendation for where to start with Forster. Secrets of Productive People is divided into five sections: The basics of productivity, the productive attitude, productive projects, aids to productivity, and productivity in action. Each part is divided into several small chapters, which open with a generous helping of quotes about some productivity topic (and I'm going to go back and save some of those), present some easily-digestible related set of thoughts (usually with an exercise), and conclude with a summary. Each chapter is about the length of a long blog post. I think this structure interferes with developing an idea at greater length, but it does make for good reading material in an environment where you're regularly interrupted or only have five or ten minutes. I think I've mentioned in every review that Forster won me over by being willing to talk about the problems with attempting to do too much, not just presenting a system to allow one to accomplish more. Some other books, such as David Allen's famous Getting Things Done, seem to assume you already know what you need to get done, or will easily be able to figure that out when you think for a while, and just need a system to manage all the things you've decided to do. Forster takes the opposite approach, and this book is the clearest yet on this point: most productivity problems are not from being insufficiently efficient, but from doing the wrong things and too many things. You don't need more time; everyone gets the same amount of time. You need to do the right things with the time you have, and that usually means doing fewer things. Readers of Forster's previous books will recognize many of the themes here. Some of the techniques from Do It Tomorrow and some of the exercises from Get Everything Done show up again here. But Forster has streamlined and focused the advice, discarded some things, made his task management recommendations less elaborate and more focused, and spends much more time hammering home the point that the only prioritization that really matters is whether you commit to doing something or don't. The specific task management system he recommends here is one of the variations he's been talking about in his blog and is much simpler than the Do It Tomorrow system: pick five tasks, work on them until you've finished three of them, and then refill to five tasks. I've been using it since reading this book, replacing one of Forster's more elaborate systems from his blog, and it's surprisingly effective. He breaks down in some detail why this works and how to extract additional useful feedback information from it, and now I want to do some of the additional exercises he describes. (That said, I'm still dubious about his advice to not keep any larger to-do list and only rely on your analysis at the time of refilling the list to decide what to do. I use this system with a supplemental, longer list of ideas for future tasks, and that works better for me, although I do have to fight forming a sense of obligation about the things on the longer list. David Allen still has a point that if you don't write down a complete inventory of the things you're worrying about, your brain will try to obsess over them to keep from forgetting them.) The productive projects section told me some things I needed to hear about time commitments. Projects take regular, focused attention, and starting numerous things without giving them that attention is much worse than doing far fewer things but doing them regularly and reliably. I think Forster's coaching on focus and persistence is very valuable; the trick, of course, is building up your willpower for it and learning to say no. This is in-line with recent psychological analysis of multi-tasking and its various negative effects. The working to-do list capped at five things provides enough variety to mentally shift gears if one runs out of steam on some specific project while maintaining enough focus to not leave things behind half-done. The productivity aids section provides a more random collection of tips and tricks, many of which I've seen before in his previous books. I've mostly not tried these, so can't say much about how effective they are, but Forster's ideas almost always sound interesting and plausible. I thought the weakest part of the book was the last section, on applied productivity. Here, Forster takes various life areas (exercise, parenting, finances, writing, etc.) and talks about how the principles of this book can be applied to them. Each area could be a book in itself, and the short essay format of these chapters doesn't do justice to large topics. The result is a rather repetitive section that just stresses analysis, metrics, and repeated, focused attention all valid points, but ones you can pick up from the previous chapters. I don't think these case studies added much value. Do It Tomorrow is still my favorite Forster book to read, but I think Secrets of Productive People is now the best and most polished overview of his time management and productivity approach. If you're interested in this topic and not already sick of Forster from my previous recommendations, this would be a great place to start. I got a lot out of it even with all the time management reading I've previously done, and will probably re-read sections of it and try more of the exercises. Rating: 9 out of 10

Debian Long Term Support (LTS) Those were 8th and 9th months working on Debian LTS started by Raphael Hertzog at Freexian. I had trouble resuming work in November as I had taken a long break during the month and started looking at issues only during the last week of November.

Imagemagick, again I have, again, spent a significant amount of time fighting the ImageMagick (IM) codebase. About 15 more vulnerabilities were found since the last upload, which resulted in DLA-756-1. In the advisory, I unfortunately forgot to mention CVE-2016-8677 and CVE-2016-9559, something that was noticed by my colleague Roberto after the upload... More details about the upload are available in the announcement. When you consider that I worked on IM back in october, which lead to an upload near the end of November covering around 80 more vulnerabilities, it doesn't look good for the project at all. Of the 15 vulnerabilities I worked on, only 6 had CVEs assigned and I had to request CVEs for the other 9 vulnerabilities plus 11 more that were still unassigned. This lead to the assignment of 25 distinct CVE identifiers as a lot of issues were found to be distinct enough to warrant their own CVEs. One could also question how many of those issues affect the fork, Graphicsmagick. A lot of the vulnerabilities were found through fuzzing searches that may not have been tested on Graphicsmagick. It seems clear to me that a public corpus of test data should be available to test regressions and cross-project vulnerabilities. It's already hard enough to track issues withing IM itself, I can't imagine what it would be for the fork to keep track of those issues, especially since upstream doesn't systematically request CVEs for issues that they find, a questionable practice considering the number of issues we all need to keep track of.

Nagios I have also worked on the Nagios package and produced DLA 751-1 which fixed two fairly major issues (CVE-2016-9565 and CVE-2016-9566) that could allow remote root access under certain conditions. Fortunately, the restricted permissions setup by default in the Debian package made both exploits limited to information disclosure and privilege escalation if the debug log is enabled. This says a lot about how proper Debian packaging can help in limiting the attack surface of certain vulnerabilities. It was also "interesting" to have to re-learn dpatch to add patches to the package: I regret not converting it to quilt, as the operation is simple and quilt is so much easier to use. People new to Debian packaging may be curious to learn about the staggering number of patching systems historically used in Debian. On that topic, I started a conversation about how much we want to reuse existing frameworks when we work on those odd packages, and the feedback was interesting. Basically, the answer is "it depends"...

NSS I had already worked on the package in November and continued the work in December. Most of the work was done by Raphael, which fixed a lot of issues with the test suite. I tried to wrap this up by fixing CVE-2016-9074, the build on armel and the test suite. Unfortunately, I had to stop again because I ran out of hours and the fips test suite was still failing, but fortunately Raphael was able to complete the work with DLA-759-1. As things stand now, the package is in better shape than in other suites as tests (Debian bug #806639) and autopkgtest (Debian bug #806207) are still not shipped in the sid or stable releases.

Other work For the second time, I forgot to formally assign myself a package before working on it, which meant that I wasted part of my hours working on the monit package. Those hours, of course, were not counted in my regular hours. I still spent some time reviewing mejo's patch to ensure it was done properly and it turned out we both made similar patches working independently, always a good sign. As I reported in my preliminary November report, I have also triaged issues in libxml2, ntp, openssl and tiff. Finally, I should mention my short review of the phpMyAdmin upload, among the many posts i sent to the LTS mailing list.

Other free software work One reason why I had so much trouble getting paid work done in November is that I was busy with unpaid work...

manpages.debian.org A major time hole for me was trying to tackle the manpages.debian.org service, which had been offline since August. After a thorough evaluation of the available codebases, I figured the problem space wasn't so hard and it was worth trying to do an implementation from scratch. The result is a tool called debmans. It took, obviously, way longer than I expected, as I experimented with Python libraries I had been keeping an eye on for a while. For the commanline interface, I used the click library, which is really a breeze to use, but a bit heavy for smaller scripts. For a web search service prototype, I looked at flask, which was also very interesting, as it is light and simple enough to use that I could get started quickly. It also, surprisingly, fares pretty well in the global TechEmpower benchmarking tests. Those interested in those tools may want to look at the source code, in particular the main command (using an interesting pattern itself, __main__.py) and the search prototype. Debmans is the first project for which I have tried the CII Best Practices Badge program, an interesting questionnaire to review best practices in software engineering. It is an excellent checklist I recommend every project manager and programmer to get familiar with. I still need to complete my work on Debmans: as I write this, I couldn't get access to the new server the DSA team setup for this purpose. It was a bit of a frustrating experience to wait for all the bits to get into place while I had a product ready to test. In the end, the existing manpages.d.o maintainer decided to deploy the existing codebase on the new server while the necessary dependencies are installed and accesses are granted. There's obviously still a bunch of work to be done for this to be running in production so I have postponed all this work to January. My hope is that this tool can be reused by other distributions, but after talking with Ubuntu folks, I am not holding my breath: it seems everyone has something that is "good enough" and that they don't want to break it...

Monkeysign I spent a good chunk of time giving a kick in the Monkeysign project, with the 2.2.2 release, which features contributions from two other developers, which may be a record for a single release. I am especially happy to have adopted a new code of conduct - it has been an interesting process to adapt the code of conduct for such a relatively small project. Monkeysign is becoming a bit of a template on how to do things properly for my Python projects: documentation on readthedocs.org including a code of conduct, support and contribution information, and so on. Even though the code now looks a bit old to me and I am embarrassed to read certain parts, I still think it is a solid project that is useful for a lot of people. I would love to have more time to spend on it.

LWN publishing As you may have noticed if you follow this blog, I have started publishing articles for the LWN magazine, filed here under the lwn tag. It is a way for me to actually get paid for some of my blogging work that used to be done for free. Reports like this one, for example, take up a significant amount of my time and are done without being paid. Converting parts of this work into paid work is part of my recent effort to reduce the amount of time I spend on the computer. An funny note: I always found the layout of the site to be a bit odd, until I looked at my articles posted there in a different web browser, which didn't have my normal ad blocker configuration. It turns out LWN uses ads, and Google ones too, which surprised me. I definitely didn't want to publish my work under banner ads, and will never do so on this blog. But it seems fair that, since I get paid for this work, there is some sort of revenue stream associated with it. If you prefer to see my work without ads, you can wait for it to be published here or become a subscriber which allows you to get rid of the ads on the site. My experience with LWN is great: they're great folks, and very supportive. It's my first experience with a real editor and it really pushed me in improving my writing to make better articles that I normally would here. Thanks to the LWN folks for their support! Expect more of those quality articles in 2017.

Debian packaging I have added a few packages to the Debian archive:

• magic-wormhole: easy file-transfer tool, co-maintained with Jamie Rollins
• slop: screenshot tool
• xininfo: utility used by teiler
• teiler (currently in NEW): GUI for screenshot and screencast tools

I have also updated sopel and atheme-services.

Other work Against my better judgment, I worked again on the borg project. This time I tried to improve the documentation, after a friend asked me for help on "how to make a quick backup". I realized I didn't have any good primer to send regular, non-sysadmin users to and figured that, instead of writing a new one, I could improve the upstream documentation instead. I generated a surprising 18 commits of documentation during that time, mainly to fix display issues and streamline the documentation. My final attempt at refactoring the docs eventually failed, unfortunately, again reminding me of the difficulty I have in collaborating on that project. I am not sure I succeeded in making the project more attractive to non-technical users, but maybe that's okay too: borg is a fairly advanced project and not currently aimed at such a public. This is yet another project I am thinking of creating: a metabackup program like backupninja that would implement the vision created by liw in his A vision for backups in Debian post, which was discarded by the Borg project. Github also tells me that I have opened 19 issues in 14 different repositories in November. I would like to particularly bring your attention to the linkchecker project which seems to be dead upstream and for which I am looking for collaborators in order to create a healthy fork. Finally, I started on reviving the stressant project and changing all my passwords, stay tuned for more!

While learning about and configuring weechat this evening, I noticed a lot of complexity and unsatisfying tradeoffs related to its UI, its mouse support, and its built-in window system. Got to wondering what I'd do differently, if I wrote my own IRC client, to avoid those problems. The first thing I realized is, it is not a good idea to think about writing your own IRC client. Danger will robinson.. So, let's generalize. This blog post is not about designing an IRC client, but about exploring simpler ways that something like an IRC client might handle its UI, and perhaps designing something general-purpose that could be used by someone else to build an IRC client, or be mashed up with an existing IRC client. What any modern IRC client needs to do is display various channels to the user. Maybe more than one channel should be visible at a time in some kind of window, but often the user will have lots of available channel and only want to see a few of them at a time. So there needs to be an interface for picking which channel(s) to display, and if multiple windows are shown, for arranging the windows. Often that interface also indicates when there is activity on a channel. The most recent messages from the channel are displayed. There should be a way to scroll back to see messages that have already scrolled by. There needs to be an interface for sending a message to a channel. Finally, a list of users in the channel is often desired. Modern IRC clients implement their own UI for channel display, windowing, channel selection, activity notification, scrollback, message entry, and user list. Even the IRC clients that run in a terminal include all of that. But how much of that do they need to implement, really? Suppose the user has a tabbed window manager, that can display virtual terminals. The terminals can set their title, and can indicate when there is activity in the terminal. Then an IRC client could just open a bunch of terminals, one per channel. Let the window manager handle channel selection, windowing (naturally), and activity notification. For scrollback, the IRC client can use the terminal's own scrollback buffer, so the terminal's regular scrollback interface can be used. This is slightly tricky; can't use the terminal's alternate display, and have to handle the UI for the message entry line at the bottom. That's all the UI an IRC client needs (except for the user list), and most of that is already implemented in the window manager and virtual terminal. So that's an elegant way to make an IRC client without building much new UI at all. But, unfortunately, most of us don't use tabbed window managers (or tabbed terminals). Such an IRC client, in a non-tabbed window manager, would be a many-windowed mess. Even in a tabbed window manager, it might be annoying to have so many windows for one program. So we need fewer windows. Let's have one channel list window, and one channel display window. There could also be a user list window. And there could be a way to open additional, dedicated display windows for channels, but that's optional. All of these windows can be seperate virtual terminals. A potential problem: When changing the displayed channel, it needs to output a significant number of messages for that channel, so that the scrollback buffer gets populated. With a large number of lines, that can be too slow to feel snappy. In some tests, scrolling 10 thousand lines was noticiably slow, but scrolling 1 thousand lines happens fast enough not to be noticiable. (Terminals should really be faster at scrolling than this, but they're still writing scrollback to unlinked temp files.. sigh!) An IRC client that uses multiple cooperating virtual terminals, needs a way to start up a new virtual terminal displaying the current channel. It could run something like this:

x-terminal-emulator -e the-irc-client --display-current-channel

That would communicate with the main process via a unix socket to find out what to display. Or, more generally:

x-terminal-emulator -e connect-pty /dev/pts/9

connect-pty would simply connect a pty device to the terminal, relaying IO between them. The calling program would allocate the pty and do IO to it. This may be too general to be optimal though. For one thing, I think that most curses libraries have a singleton terminal baked into them, so it might be hard to have a single process control cursors on multiple pty's. And, it might be innefficient to feed that 1 thousand lines of scrollback through the pty and copy it to the terminal. Less general than that, but still general enough to not involve writing an IRC client, would be a library that handled the irc-client-like channel display, with messages scrolling up the terminal (and into the scrollback buffer), a input area at the bottom, and perhaps a few other fixed regions for status bars and etc. Oh, I already implemented that! In concurrent-output, over a year ago: a tiling region manager for the console I wonder what other terminal applications could be simplified/improved by using multiple terminals? One that comes to mind is mutt, which has a folder list, a folder view, and an email view, that all are shoehorned, with some complexity, into a single terminal.

Review: A Man Called Ove, by Fredrik Backman

Translator:	Henning Koch
Publisher:	Washington Square
Copyright:	2012, 2014
Printing:	May 2015
ISBN:	1-4767-3802-5
Format:	Trade paperback
Pages:	337

Ove is 59 years old and drives a Saab. He's grumpy, often taciturn, very particular in how things should be done, and extremely judgmental about how other people do them. He is the sort of person who regularly rants about how poorly things are done these days and how much better they used to be. Ove does not like change. A Man Called Ove opens with him terrorizing the employees of an Apple Store, trying to buy a computer, but this incident is actually foreshadowing, and will only make sense at the very end of the book. The story actually begins in the second chapter, with Ove making his morning rounds of the neighborhood in which he lives, discovering an out-of-place bicycle and a mangy cat, and then starting to put a hook in his ceiling. But just as he's getting started, he's interrupted by new neighbors, who are incapable of backing up a trailer properly without scraping it against his house. Not that motor vehicles are allowed in the area anyway. That inauspicious beginning changes Ove's life, mostly through the sheer persistence of other people's disasters. It's not obvious at first that it will, and at the start A Man Called Ove could be a funny collection of stories about a curmudgeon. But as Backman shows more of Ove's life and tells more of his background and situation, it becomes something so much more, something satisfying and heart-breaking and deeply human. I've been struggling to review this book because I find it hard to capture what makes it so wonderful. Making that even harder, several key plot elements are introduced gradually in the story in ways that add a lot to the rhythm of the plot, and I don't want to spoil them. I think the closest I can get in a spoiler-free review is that A Man Called Ove is about empathy. It's about human connection, even when people seem unlikable, unreachable, or angrily off-putting. And it's a book about seeing the best inside other people, and about finding ways to be persistently oneself while still changing enough to find new connections, and about recognizing those moments when someone is showing you their best without getting caught up on the surface presentation. The man Ove is the center of this story, the subject of tight third person perspective for nearly all of the book. He's 59 when the story opens, but by the end of the book, mostly through flashback chapters, the reader knows his childhood and early adulthood and much of the story of his marriage. At first, he seems to be an obnoxious, surly, angry curmudgeon, the sort of old man who yells at clouds. But the joy of this book is how the reader's perception changes, how one gains sympathy, and then respect, first for Ove's unshakable inner sense of morality that he got from his father and then for his rule-based approach to how the world should work. One never entirely agrees with him, but Backman demystifies and explains Ove's thought process and ties it into a different generation and a different way of interacting with work (although Ove was still uncommon even in his youth, just not as unique). As I write this review, news and opinion in the United States are very focused on the plight of the white working class and how that does or does not explain recent election results. Backman is Swedish (I read this book in translation), so it's not coming from US culture and the cultural fault lines are not quite the same. But I think this book says something deeply valuable and fascinating about the working-class culture of Ove's youth, something that's much less about specific politics and much more about how it feels to make things with one's hands, to build or rebuild one's own house, or to work for thirty years at the same job and not be interested in a promotion to management. Backman does a truly spectacular job conveying the sense of angry frustration at the changes in work and life, the difficulty communicating one's internal feelings meaningfully, and the quiet joy of finding those places in life where one can do things properly. Ove is, of course, not the only character in this book, and every character here is a delight in their own idiosyncratic ways. The main story arc involves the various people in his neighborhood, particularly his new next-door neighbors: a pregnant Iranian woman and her very laid-back husband who is incapable of doing things around the house but keeps trying. I think Parvaneh, the woman, is my favorite character of the story except for Ove, and it's fitting that she's the first to work out the broad outlines of what's happening in Ove's life and the tricky path to effectively helping him. In a way, she's Ove's opposite: fiery, mercurial, talkative, and meddling. But she sees things in Ove that no one else seems to notice. (And the scene between her and Ove when she's learning to drive is a thing of beauty.) This is a book that could have been extremely sad, and yet isn't. It's a book about somber, depressing topics that somehow manages to be delightfully funny. And it's about a curmudgeon who persistently fails to have any sort of stereotyped heart of gold, but is nonetheless one of the most satisfying, fascinating, ethical, and good-willed characters I've ever read about. It manages to treat a collection of very different characters with individualized deep empathy and appreciation, while never pushing them all into the same mold. And the ending is wonderful. I rarely read slice of life stories, but this one is worth making an exception for. It's one of the best books I've ever read. Highly, highly recommended. Rating: 10 out of 10

Review: A Man Called Ove, by Fredrik Backman

Translator:	Henning Koch
Publisher:	Washington Square
Copyright:	2012, 2014
Printing:	May 2015
ISBN:	1-4767-3802-5
Format:	Trade paperback
Pages:	337

Ove is 59 years old and drives a Saab. He's grumpy, often taciturn, very particular in how things should be done, and extremely judgmental about how other people do them. He is the sort of person who regularly rants about how poorly things are done these days and how much better they used to be. Ove does not like change. A Man Called Ove opens with him terrorizing the employees of an Apple Store, trying to buy a computer, but this incident is actually foreshadowing, and will only make sense at the very end of the book. The story actually begins in the second chapter, with Ove making his morning rounds of the neighborhood in which he lives, discovering an out-of-place bicycle and a mangy cat, and then starting to put a hook in his ceiling. But just as he's getting started, he's interrupted by new neighbors, who are incapable of backing up a trailer properly without scraping it against his house. Not that motor vehicles are allowed in the area anyway. That inauspicious beginning changes Ove's life, mostly through the sheer persistence of other people's disasters. It's not obvious at first that it will, and at the start A Man Called Ove could be a funny collection of stories about a curmudgeon. But as Backman shows more of Ove's life and tells more of his background and situation, it becomes something so much more, something satisfying and heart-breaking and deeply human. I've been struggling to review this book because I find it hard to capture what makes it so wonderful. Making that even harder, several key plot elements are introduced gradually in the story in ways that add a lot to the rhythm of the plot, and I don't want to spoil them. I think the closest I can get in a spoiler-free review is that A Man Called Ove is about empathy. It's about human connection, even when people seem unlikable, unreachable, or angrily off-putting. And it's a book about seeing the best inside other people, and about finding ways to be persistently oneself while still changing enough to find new connections, and about recognizing those moments when someone is showing you their best without getting caught up on the surface presentation. The man Ove is the center of this story, the subject of tight third person perspective for nearly all of the book. He's 59 when the story opens, but by the end of the book, mostly through flashback chapters, the reader knows his childhood and early adulthood and much of the story of his marriage. At first, he seems to be an obnoxious, surly, angry curmudgeon, the sort of old man who yells at clouds. But the joy of this book is how the reader's perception changes, how one gains sympathy, and then respect, first for Ove's unshakable inner sense of morality that he got from his father and then for his rule-based approach to how the world should work. One never entirely agrees with him, but Backman demystifies and explains Ove's thought process and ties it into a different generation and a different way of interacting with work (although Ove was still uncommon even in his youth, just not as unique). As I write this review, news and opinion in the United States are very focused on the plight of the white working class and how that does or does not explain recent election results. Backman is Swedish (I read this book in translation), so it's not coming from US culture and the cultural fault lines are not quite the same. But I think this book says something deeply valuable and fascinating about the working-class culture of Ove's youth, something that's much less about specific politics and much more about how it feels to make things with one's hands, to build or rebuild one's own house, or to work for thirty years at the same job and not be interested in a promotion to management. Backman does a truly spectacular job conveying the sense of angry frustration at the changes in work and life, the difficulty communicating one's internal feelings meaningfully, and the quiet joy of finding those places in life where one can do things properly. Ove is, of course, not the only character in this book, and every character here is a delight in their own idiosyncratic ways. The main story arc involves the various people in his neighborhood, particularly his new next-door neighbors: a pregnant Iranian woman and her very laid-back husband who is incapable of doing things around the house but keeps trying. I think Parvaneh, the woman, is my favorite character of the story except for Ove, and it's fitting that she's the first to work out the broad outlines of what's happening in Ove's life and the tricky path to effectively helping him. In a way, she's Ove's opposite: fiery, mercurial, talkative, and meddling. But she sees things in Ove that no one else seems to notice. (And the scene between her and Ove when she's learning to drive is a thing of beauty.) This is a book that could have been extremely sad, and yet isn't. It's a book about somber, depressing topics that somehow manages to be delightfully funny. And it's about a curmudgeon who persistently fails to have any sort of stereotyped heart of gold, but is nonetheless one of the most satisfying, fascinating, ethical, and good-willed characters I've ever read about. It manages to treat a collection of very different characters with individualized deep empathy and appreciation, while never pushing them all into the same mold. And the ending is wonderful. I rarely read slice of life stories, but this one is worth making an exception for. It's one of the best books I've ever read. Highly, highly recommended. Rating: 10 out of 10

Some of the content may be NSFW. viewer discretion advised. I have had a life-long fascination with trains. One of my first memories was that of 5-7 year old, clutching my mother or grandmother s hand seeing the steam engine lumbering down whistling and smoking at the same time. I was both afraid and strangely drawn to the iron beast and the first time I knew and then slowly understood that if we come with luggage and the steam-engine comes, it means we are going to travel. I have travelled some, but there are lots to explore still and I do hope that I cover some more of it during my lifetime. The reason I am writing about trains is an article which caught my eye couple of days. Besides seeing the changing geography, the variety of food one can get on train and in stations is one of the primary reasons that Indians love to travel by trains. It is one place where you could have incredible conversations over cup of tea or favourite food and unlike air travel and the famed IFE (In-flight entertainment) people are actually pretty social even with all the gadgets. For those who are wondering, the author was travelling between Jamshedpur, Gujarat to Kolkatta, a train ride which has now gone on my bucket list for the delectable items the author has described To add to the above, it is still cheaper than air travel, although that is changing a bit as Indian Railways seeks to modernize Railways and make it into world-class bullet trains. Indian Railways has a long, rich culture and some of the most interesting nuggets you learn over time adds to the fascination of the Railways. For instance I m sharing this letter which I read first in book and then saw in the New Delhi Railway Museum. The letter I am sharing below was written by a certain Shri Okhil Chandra Sen to the Sahibganj Railway Office in year 1909, almost 38 years before India became independent. I am arrive by passenger train Ahmedpur station and my belly is too much swelling with jackfruit. I am therefore went to privy. Just I doing the nuisance that guard making whistle blow for train to go off and I am running with lotah in one hand and dhoti in the next when I am fall over and expose all my shocking to man and female women on plateform. I am got leaved at Ahmedpur station. This too much bad, if passenger go to make dung that dam guard not wait train five minutes for him. I am therefore pray your honour to make big fine on that guard for public sake. Otherwise I am making big report! to papers. If it were not for Mr. Okhil Chandra Sen we would still be running with water bottle (improvement) and jeans/shorts/whatever (again improvement) while the possibility of falling over would always be omnipresent in a hurry. Now we do have toilets and some of the better trains even have Bio-toilets which should make things better as well.(/NSFW) For the plane bit, most of my flights have been domestic flying. Some of my most memorable flights is when flying from Mumbai on a clear sky overlooking the Queen s necklace, loving it and landing in Bangalore during mist or rain or both. Delhi is also good as airports go but nothing much adventurous about it. It was only with the experience of my first international flight, I realized the same feeling again, nervousness and sense of adventure as you meet new people. Nowadays every week I do try and broaden my horizon by seeking and learning a bit about International Travel. In this I came across an article on National Geographic site which also evoked similar feelings. While I can t go back to the past and even if I did (in distant past before I was born), I wouldn t want to improve my financial situation at all (as otherwise I would hit the Grandfather Paradox or/and the Butterfly effect (essentially saying there s no free lunch), it still makes you wonder about a time when people had lot more adventure and lot more moving parts. I do wish they had a much bigger snapshot of that plane so I could really see how people sat in the old aircraft. The low-resolution picture doesn t do justice to the poster and the idea of that time. https://en.wikipedia.org/wiki/A_Sound_of_Thunder for an implementation of Butterfly effect. The Grandfather Paradox has been seen plenty of times in fantasy movies like the Back to the Future, Planet of the Apes and many others so will not go there. For the average joe today, s/he has to navigate security,check bags, get her/imself processed through passport control, get boarding pass, get to the gate on-time, get to the aircraft via bridge or bus, get to the seat, somehow make it through the ascent and use your IFE and get snacks and meals till it s time to touch-down and re-do the whole drill again as many times you are connecting. I really admire Gunnar Wolf for the tenacity he showed for the x number of connections he made both ways. While leafing through the interweb today, came across an article . While you can slice and dice the report anyway you want, for me if ever I get a chance again for an International Travel, I would try to see I get a layover at these three airports in order of preference (this is on the basis that none of these airports need a transit visa for the activities shared) a. Changi International Airport It is supposed to have shower amenities, has a movie theatre (+1), free tour of the city (+1) and of course as many Indians do go to Singapore as a destination in itself would have multiple vegetarian options (+2) so would be nice if I need to layover. b. Zurich Airport (ZRH) For passengers with an extended layover, Zurich Airport offers bicycle and inline-skate rentals and excursions to the Swiss Museum of Transport Lucerne. From business-insider.com. While I m not much of a bicycle and inline-skating freak, if the Swiss Museum of Transport Lucerne is anything to the scale of Isiko Museum which I shared in a blog post sometime before, it would be worth by itself. I haven t tried to find the site but can imagine, for e.g. if it has a full-scale model of a submarine or train engine, either steam-engines or ones like SNCF or any of the other bullet-trains and early aircraft, it would just blow my mind. When you are talking about transport, there is so much science, business, logistics etc. that I m sure I ll overload with information, photos and any trinkets they have to buy. c. Central Japan International Airport (NGO) It has a 1,000-foot-long sky deck where passengers can watch ships sail into Nagoya Port. There s also a traditional Japanese bathhouse where you can have a relaxing soak while watching the sunset over the bay. BusinessInsider.com Not a bad place to be if you need a layover. Just sink yourself in the bathhouse and see the bay and ships coming in. Luxury indeed. Honourable mention d. Munich Airport (MUC) A nearby visitors park features mini golf and a display of historic aircraft. Business-Insider.com . Now this would have made my list but I guess one would need a Schengen visa to access the visitors park but then if you have that, then why just stay in the Airport itself, could travel through Europe itself and have a longish stop-over. So all in all, it s indeed a fascinating time to be alive, dreaming and just being. Till later. Update I had forgotten to share one more reason why I was writing this article. Although somewhat of a cynic, am hopeful that Pune metro happens. Also, if I had just waited a day, would have been able to add couple of wonderful articles that would make people wanderlust more
Filed under: Miscellenous Tagged: #Best Airports, #Central Japan International Airport, #Changi International Airport, #Food, #Loo, #Nostalgia, #NSFW, #Planes, #Steam Engine, #Trains, #Zurich Airport, Indian Railways, memories

Since the past few weeks I have been researching and working on creating a white label version of Lumicall with my mentors Daniel, Juliana and Bruno. Lumicall is a free and convenient app for making encrypted phone calls from Android. It uses the SIP protocol to interoperate with other apps and corporate telephone systems. Think of any app that you use to call others using an SIP ID. What does it mean to make a white label version of Lumicall?
White labelling is the idea of taking the whole or piece of Lumicall s code by business users and tweaking it according to their requirements and functionality they want. The white label version would have client s name, logo, icons, themes etc which reflect their brand. Although, the underlying working of both the applications would be same. Think of it like getting a cake from the local bakery, putting it into a home baking dish and passing it off as something you made in front of your friends and earning all the praise. Except cake is more appealing than apps.
What whitelabelling does is it takes away the pain of collecting cake ingredients, mixing them in right proportion, baking at the right temperature and uses the experience of one cake base to create more and more fancier, prettier, grander cakes. This lets the cool bakers (or the business users of lumicall) to focus on other aspects of the party like decoration, games and drinks (or monetization, publicity and new features)
Now you would wonder who these cool bakers or business users of lumicall would be?

• Existing SIP providers can use white label version of Lumicall to expand their business and launch SIP client. This would provide a one stop shop for them!!
• New SIP clients/developers can use Lumicall white label version to get the underlying working of making encrypted phone calls using SIP protocol, it will help them to focus on other additional functionalities they would like to include.
  
• Moreover, Companies that have existing clients such as shopping apps, social network apps can benefit from integrating lumicall to launch new business ideas

While researching this work that despite the commonalities, there would need to be a unique identifier for any application in the app store. I found that there is difference between the package name and the applicationID. When I create a new project in Android Studio, the applicationId exactly matches the Java-style package name I chose during setup. However, the application ID and package name are independent of each other beyond this point. The thing to keep in mind is that app stores identify as a changed application ID as a different app altogether.

So if I were to make N separate copies of the application code for N clients, it would be a maintenance nightmare, If the build system is Gradle, using product flavors is a trick that will make this maintenance easier. Instead of N separate copies, I would simply have N product flavors. Each flavor corresponds to a customized version of my application. Pro, free, whitelabel, Debug (for development purposes), Release (for production purposes) are the basic flavors I have identified.

Each flavor would use the same source code and files of the application but resources, icons, manifests etc that are specific to each flavor can be defined again in main/src/flavor_name directory under res folders etc. according to the requirement.
Here is a snippet from my build.gradle file::

Note: You have to define at least two flavors to be able to build multiple variants.
Because once you have flavors, you can only build flavored application. You can t just build default configuration anymore. Define an empty flavor, with no applicationIdSuffix at all, and it will use all of the default config section.

This week I d be moving forward with the implementation of whitelabelling using productFlavors in Lumicall.
I would love to hear from someone who has done this before!
Comment here or email me and I promise you an excellent home-made-store-brought cake!
Wishing you all HAPPY HOLIDAYS! !
-U

All photos courtesy KK . If any deviations, would put up labels sharing whose copyright it is. Before I get into all of that, I was curious about Canada and taking the opportunity of debconf happening there in a few months, asked few people what they thought of digital payments, fees and expenses in their country and if plastic cash is indeed used therein. The first to answer was Tyler McDonald (no idea if he is anyway related to the fast-food chain McDonalds which is a worldwide operation.) This is what he had to say/share

You can use credit / debit cards almost everywhere. Restaurant waiters also usually have wireless credit / debit terminals that they will bring to your table for you to settle your bill. How much your bank charges depends on your Canadian bank and the banking plan you are on. For instance, on my plan through the Bank Of Montreal, I get (I think) 20 free transactions a month and then after that I m charged $0.50CDN/piece. However, if I go to a Bank Of Montreal ATM and withdraw cash, there is no service fee for that. There is no service fee for using *credit* cards, only *debit* cards tend to have the fee. I live in a really rural area so I can t always get to a Bank Of Montreal machine for cash. So what I usually end up doing, is either pay by credit and then pay of the balance right away so I don t have to pay interest, or when I do use my bank card to pay for something, I ask if I can get cash back as well. Yes, Canada converted to plastic notes a few years ago. We ve also eliminated the penny. For cashless transactions, you pay the exact amount billed. If you re paying somebody in cash, it is rounded up or down to the nearest 5 cents. And for $1 or $2, instead of notes, we ve moved over to coins. I personally like the plastic notes. They re smoother and feel more durable than the paper notes. I ve had one go through a laundry load by accident and it came out the other side fine.

Another gentleman responded with slightly more information which probably would interest travellers from around the world, not just Indians

Quebec has its own interbank system called Interac (https://interac.ca/en/about/our-company.html). Quebec is a very proud and independent region and for many historical reasons they want to stand on their own, which is why they support their local systems. Some vendors will support only Interac for debit card transactions (at least this was the case when I stayed there the beginning of this decade, it might have changed a bit). *Most* vendors (including supermarkets like Provigo, Metro, etc) will accept major credit and debit cards, although MasterCard isn t accepted as widely there as Visa is. So, if you have one of both, load up your Visa card instead of your MasterCard or get a prepaid Visa card from your bank. They support chip cards everywhere so don t worry about that. If you have a 5 digit pin on any of your cards and a vendor asks you for a 4 digit pin, it will work 90%+ of the times if you just enter the first 4 digits, but it s usually a good idea to go change your pin to a 4 digit just to be safe.

From the Indian perspective all of the above fits pretty neat as we also have Pin and Chip cards (domestically though most ATMs still use the magnetic strip and is suspected that the POS terminals aren t any better.) That would be a whole different story so probably left for another day. I do like the bit about pocketing the change tip. As far as number of free transactions go, it was pretty limited in India for few years before the demonetization happening now. Few years before, I do remember doing as many transactions on the ATM as I please but then ATM s have seen a downward spiral in terms of technology upgradation, maintenance etc. There is no penalty to the bank if the ATM is out-of-order. If there was significant penalty then we probably would have seen banks taking more care of ATM s. It is a slightly more complex topic hence would take a stab at it some other day. Do hope though that the terms for ATM usage for bank customers become lenient similar to Canada otherwise it would be difficult for Indians to jump on the digital band-wagon as you cannot function without cheap, user-friendly technology. The image has been taken from this fascinating article which appeared in Indian Express couple of days back. Coming back to the cheese and wine in the evening. I think we started coming back from Eagle Encounters around 16:30/17:00 hrs Cape Town time. Somehow the ride back was much more faster and we played some Bollywood party music while coming back (all cool). Suddenly remembered that I had to buy some cheese as I hadn t bought any from India. There is quite a bit of a post where I m trying to know/understand if spices can be smuggled (which much later I learnt I didn t need to but that s a different story altogether), I also had off-list conversations with people about cheese as well but wasn t able to get any good recommendations. Then saw that KK bought Mysore Pak (apparently she took a chance not declaring it) which while not being exactly cheese fit right into things. In her own words a South Indian ghee sweet fondly nicknamed the blocks of cholesterol and reason #3 for bypass surgery . KK So with Leonard s help we stopped at a place where it looked like a chain of stores. Each store was selling something. Seeing that, I was immediately transported to Connaught Place, Delhi The image comes from http://planetden.com/food/roundabout-world-connaught-place-delhi which attempts to explain Connaught Place. While the article is okish, it lacks soul and not written like a Delhite would write or anybody who has spent a chunk having spent holidays at CP. Another day, another story, sorry. What I found interesting about the stores while they were next to each other, I also eyed an alcohol shop as well as an Adult/Sex shop. I asked Leonard as to how far we were from UCT and he replied hardly 5 minutes by car and was shocked to see both alcohol and a sex shop. While an alcohol shop some distance away from a college is understandable, there are few and far around Colleges all over India, but adult shops are a rarity. Unfortunately, none of us have any photos of the place as till that time everybody s phone was dead or just going to be dead and nobody had thought to bring a portable power pack to juice our mobile devices. A part of me was curious to see what the sex shop would have and look from inside, but as was with younger people didn t think it was appropriate. All of us except Jaminy and someone else (besides Leonard) decided to stay back, while the rest of us went inside to explore the stores. It took me sometime to make my way to the cheese corner and had no idea which was good and which wasn t. So with no idea of brands therein, the only way to figure out was the pricing. So bought two, one a larger 500 gm cheap piece and a smaller slightly more expensive one just to make sure that the Debian cheese team would be happy. We did have a mini-adventure as for sometime Jaminy was missing, apparently she went goofing off or went to freshen up or something and we were unable to connect with her as all our phones were dead or dying. Eventually we came back to UCT, barely freshened up when it was decided by our group to go and give our share of goodies to the cheese and wine party. When I went up to the room to share the cheese, came to know they needed a volunteer for cutting veggies etc. Having spent years seeing Yan Can Cook and having practised quite a bit tried to do some fancy decoration and some julian cutting but as we got dull knives and not much time, just did some plain old cutting I have to share I had a fascinating discussion about cooking in Pressure Cookers. I was under the assumption that everybody knows how to use Pressure Cookers as they are one of the simplest ways to cook food without letting go of all the nutrients. At least, I believe this to be predominant in the Asian subcontinent and even the chinese have similar vessels for cooking. I use what is called the first generation Pressure Cooker. I have been using a 1.5 l Prestige Pressure Cooker over half a decade, almost used daily without issues. http://www.amazon.in/Prestige-Nakshatra-Aluminium-Pressure-Cooker/dp/B00CB7U1OU There are also induction pressure cookers nowadays in the Indian market and this model https://www.amazon.in/Prestige-Deluxe-Induction-Aluminum-Pressure/dp/B01KZVPNGE/ref=sr_1_2 Basmati is long-grain, aromatic rice which most families used in very special occasions such as festivals, marriages, anything good and pure is associated with the rice. I had also shared my lack of knowledge of industrial Microwave Ovens. While I do get most small Microwave Ovens like these , cooking in industrial ovens I simply have no clue. Anyways, after that conversation I went back, freshened up a bit and sometime later found myself in the middle of this Also at times found myself in middle of this I tried quite a few chocolates but the best one I liked (don t remember the name) was a white caramel chocolate which literally melted into my mouth. Got the whole died and went to heaven experience. Who said gluttony is bad Or this As can be seen the French really enjoy their bread. I do remember a story vaguely (don t remember if it was a children s fairy tale or something) about how the French won a war through their french bread. Or this We also had juices for the teetotaller or who can t handle drinks. Unsurprisingly perhaps, by the end of the session, almost all the different wines were finito while there was still some juices left to go around. From the Indian perspective, it wasn t at all exciting, there were no brawls, everybody was too civilized and everybody staggered off when they met their quota. As I was in holiday spirit, stayed up late, staggered to my room, blissed out and woke up without any headache. Pro tip Drink lots and lots and lots of water especially if you are drinking. It flushes out most of the toxins and also helps in not having after-morning headaches. If I m going drinking, I usually drown myself in at least a litre or two of water, even if I had to the bathroom couple of times before going to bed. All in all, a perfect evening. I was able to connect/talk with some of the gods whom I had wanted to for a long time and they actually listened. Don t remember if I mumbled something or made some sense in small-talk or whatever I did. But as shared, a perfect evening
Filed under: Miscellenous Tagged: #ATM usage, #Canada, #Cheese and Wine party, #Cheese shopping, #Connaught Place Delhi, #Debconf16, #Debit card, #French bread, #Julian cutting, #Mysore Pak, #white caramel chocolate

[2016-12-19: some additions] This is another little issue from Python packaging for Debian which I came across lately packaging the compressed NumPy based data container Bcolz. Upstream uses setuptools-scm to determine the software s version during build time from the source code management environment the code is in. This method is convenient for the upstream development because with that the version number doesn t need to be hard-coded, and often people just forget to update that (and other version carrying files like doc/conf.py) when a new version of a project is released. python-setuptools just needs to be added to the setup.py to do its job, and in Bcolz the code goes like this:

setup(
    name="bcolz",
    use_scm_version= 
        'version_scheme': 'guess-next-dev',
        'local_scheme': 'dirty-tag',
        'write_to': 'bcolz/version.py'

The file the version number is written to is bcolz/version.py. This file isn t in the upstream code revision nor in the tarball which was released by the upstream developers, it s always generated during build time. In Debian there is an error if you try to build a package from a source tree which contains files which aren t to be found in the corresponding tarball, like cruft from a previous build, or if any files have changed therefore every new package should be test build also twice in a row in a non-chroot environment. Generally there a two ways to solve this, either you add cruft to debian/clean, or you add the file resp. a matching file pattern to extend-diff-ignore in debian/source/options. Which method is the better one could be discussed, I m generally using the clean option if something isn t in the upstream tarball, and the source/options solution if something is already in the upstream tarball, but gets changed during a build. This is related to your preferred Git procedures, if you remove a file which is in the upstream tarball these removals have to be checked in separately, and that means everytime a new upstream tarball is released that is not very convenient. Another option which is available is to strip certain files from the upstream tarball by putting them on the Files-Excluded in deb/copyright. By the way, the same complex applies to egg-info/: that folder is shipped or is not shipped in the upstream tarball, and files in that folder get changed during build. When the source code is put into a Git environment for Debian packaging, there could be problems with the version number setuptools-scm comes up with. This setuptools extension gets the recent version from the latest Git tag when there is a version number to be found, and that s all right. In Git environments for Debian packaging (like e.g. of the Debian Science group, the Python groups and the others) that is available, like the commonly used upstream tags have that¹. The problem is, sometimes the upstream version which Debian has² doesn t match the original upstream version number which is wanted for version in bcolz/version.py. For example, the suffix +ds is used if the upstream tarball has been stripped from prebuild files or embedded convenience shipments (like it s the case with the Bcolz package where c-blosc/ has been stripped because that s build for another package), of the suffix +dfsg shows that non-DFSG free software has been removed (which can t be distributed through the main archive section). For example, the version string for Bcolz which is found after the build currently (1.1.0+ds1-1) is 1.1.0+ds1:

# coding: utf-8
# file generated by setuptools_scm
# don't change, don't track in version control
version = '1.1.0+ds1'

But that s not wanted because this version never has been released, but appears everywhere:

$ pip list   grep bcolz
bcolz (1.1.0+ds1)
$ python3 -c 'import bcolz; bcolz.print_versions()'
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
bcolz version:     1.1.0+ds1

There are several different ways how to fix this. The one with the crowbar (like said in German) is to patch use_scm_version out from setup.py, but if you don t provide any version in exchange the version number which is used by Setuptools then is 0.0.0. The upstream version could be hard-coded into the patch, but then again it has not to be forgotten to update it manually by the maintainer, which is not very convenient. Plus, setup.py could change and the patch then might need to be unfuzzed, thus more work. Bad. A patch could be spared by manipulating and exporting the SETUPTOOLS_SCM_PRETEND_VERSION environment variable for setuptools-scm in debian/rules, which is sometimes used when I see the returns for that string on Debian Code Search. But how to prevent to hard code the version number, here? The dpkg-dev package (pulled by build-essential) ships a Makefile snippet /usr/share/dpkg/pkg-info.mk which could be included into debian/rules. It defines several variables which are useful for packaging, like DEB_SOURCE contains the source package name, they are extracted from debian/changelog. But, DEB_VERSION_UPSTREAM which is available through that puts out the upstream version without epoch and Debian revision, but it s not getting any finer grained out of the box. For a custom fix, a regular expression which removes the +... extensions (if present) from the bare upstream version string would be s/\+[^+]*//:

$ echo "1.1.0+ds1"   sed -e 's/\+[^+]*//'
1.1.0
$ echo "1.1.0"   sed -e 's/\+[^+]*//'
1.1.0
$ echo "1.1.0+dfsg12"   sed -e 's/\+[^+]*//'
1.1.0

With that, a custom variable VERSION_UPSTREAM could be set on the top of DEB_VERSION_UPSTREAM (from pkg-info.mk) in debian/rules:

include /usr/share/dpkg/pkg-info.mk
VERSION_UPSTREAM = $(shell echo '$(DEB_VERSION_UPSTREAM)'   sed -e 's/\+[^+]*//')
export SETUPTOOLS_SCM_PRETEND_VERSION = $(VERSION_UPSTREAM)

Bam, that works (see the commit here):

# coding: utf-8
# file generated by setuptools_scm
# don't change, don't track in version control
version = '1.1.0'

An addition, I ve seen that dh-python also takes care of SETUPTOOLS_SCM_PRETEND_VERSION since 2.20160609. The environment variable is set by the Debhelper build system if python 3, -setuptools-scm is among the build-dependencies in debian/control. The Perl code for that is in dh/pybuild.pm. I think the version number string above comes from dh-python s pretended version, and not from any of the Git tags (which are currently debian/1.1.0+ds1-1 and upstream/1.1.0+ds1).

I dunno if people heard or didn t hear about the demonetisation of INR 500 and INR 1000 which happened in India on 8th November 2016 with new currency designed in India of INR 2000 and INR 500. What they did was from that moment onwards, paper currency of INR 500 and INR 1000 notes were declared invalid except few places (Government Hospitals, Petrol Pumps, Booking of Air and Train tickets) . The reasons given were as a. End of corruption There is/was suspicion that there are people who have loads of unaccounted wealth which they keep in the form of Cash in hand, b. Charge against fake/duplicate currency There is/was suspicion that quite a bit of the money esp, high value notes such as INR 500 and INR 1000, so having made them illegal, people had to hand over cash to banks and fake money would go outside the system. c. Terror funding This is related with the above point. There is a popular theory/myth/fact that terrorists use fake money to buy people, arms and ammunition while further devaluing the value of INR against dollar and basket of other high-value currencies that Indian currency follows/bases itself on. Each of these theories/myths/facts has been contested. Every day we are seeing and reading reports of people being caught with new currency in absurd numbers while RBI , our central bank and Lender of Last Resort has had to play multiple roles such as policing along with the country s Income Tax Department as well as pumping in new notes of the NEW INR 2000/- and INR 500/- into ATM s and Bank branches around the country. Now while the above may seem to be reasonable, there have been multiple factors which has made the whole exercise less effective while implementing a. Banking reach While India does and can boast of somewhat good indicators of banking reach . But Quarter of these accounts were opened only in the last couple of years under the Pradhan Mantri Jan Dhana Yojana . There are quite a few limitations of such accounts. It is a good scheme as if you develop a good rapport with a bank and show good credit/debit understanding then there is possibility to move to normal full-fledged bank account. Almost all of these accounts had zero-balances till the demonetization move. Many of these accounts are suspected to have been conduits to convert black money to white as the Govt. had said it will not scrutinize small savings bank accounts. Also many bank accounts historically have laid dormant over the years. One of my first jobs was of a data entry operator in a bank and I used to see hundreds of bank accounts lying dormant for years together. This was in bank digitization in early 90s. Small Savings accounts would not be scrutinized if they bring upto INR 250000 while Jan Dhan accounts have an upper limit of only INR 50000 . Even then, it has lead to a huge surge in balances specifically in Zero balances account. What begs the question is if it is their hard-earned money why hadn t they deposited money before 8th November 2016. While I can t speak about them, I can certainly speak about myself. I hardly keep at the most INR INR 5/10K for medical emergencies in-house for number of years. Unless you are a businessman who has need of cash or have some function, nobody that I know would keep such amounts in their homes, simply for the possibility of theft in homes. So how did such people who are not able to open a full-fledged saving account get access to such large amounts? In most public sector banks, to have a full-fledged savings account the only requirements are a. Have INR 500 to 1000 as balance at all times.
b. Have permanent identity and residential proof
c. Two photographs
d. 2-3 people who are account holders who can act as guarantor. Of the above, b. and d. are probably sticking points for most migrants, while d. may be a sticking point for labourers, craftsman etc. hence the need for that specific scheme. Which leads to the natural suspicion that they may have been white-washing somebody s untaxed, unaccounted money which is being put into bank and made into legitimate white money. People do not have to file an Income Tax Return (ITR) unless they earn more than 250,000 in a single financial year. One good off-shot of the scheme though is the transparency gained about Bank Mitras b. Number of banks, quality of Bank services, number of people per bank at least in Nationalized Banks leaves much to be desired. We can t even try to compare with other BRIC countries, leave alone Germany. One another positive off-shoot has been the introduction of Mobile ATM Vans around the country. I had experienced such vans in Mumbai since ages, but not anywhere else. I do hope that both Bank Mitras as well as such Van Mobile ATMs happen more. There are huge swathes of people who are currently unbanked. Getting them into the banking infrastructure, getting them to *think* about taking rational financial decisions, i.e. saving and spending, different types of saving etc. should not make citizens and the banking systems more productive and efficient, but hopefully improves our GDP and make it more resilient to any outside financial shocks. c. Many bank websites have everything in English. That norm needs to change. I do have few queries though, one of the countries who is supposed to be a prominent supporter and user of cashless society is supposed to be Canada. Could any Canadians (also because debconf is going to happen in Canada in 2017) share how and if they had seen the Canadian banking system evolve in their country ? Also how much of Canada s economy is cashless i.e used to Electronic Money Transfer and other means (but not cash) and how much is cash, more in day-to-day usage and transactions. I am trying to get people s perspective rather than some website which may serve only raw numbers, although even that would be appreciable. Also what, if any charges/commission are paid to a Canadian bank for paying via card/electronic money transfer. I ask as India has reduced charges overall to 1% from 2% for making transactions upto INR 2000 in a day. There has also been recent talk of plastic notes instead of paper currency. Plastic notes are supposed to be more copy-proof and also will work for much longer time. They will not soil as paper notes do. How have countries been looking at Plastic currencies. I do suspect there would be issues while destroying plastic money vis-a-vis paper currencies. A sort of interesting discussion that I had with Bernelle before venturing into South Africa was asking her about monetary transactions in SA. She had replied that the highest denomination notes was 200 ZAR which is roughly equal to ( ZAR 200 x 5 = INR 1000) . What is/was interesting that Bernelle told me to be careful and as far as possible not to show 200 ZAR note, whereas in India, even the cheapest worker I have met, they have seen and used INR 1000 note. The context of the discussion was being safe in South Africa and doing transactions with people around as to what works. It would be curiouser to know how things work in Canada for instance ? Also has Canada or any other country have experimented with plastic notes. If yes, how has the experience been ? I would have to say this is in no way a definitive guide of the different impressions and repercussion that the decision and the way it s playing out even now. Another thing, while researching for the article there were lots of interesting knowledge, for e.g. the Big Mac Index and it s limitations which I didn t know how to integrate into the decision and Policy taken. I also came to know/saw that lots of Policy initiatives being taken by the current (NDA)Government is similar to initiatives taken elsewhere in the world.. Whether the Policy would be fruitful in getting the desired outcome or would it lead to more chaos and down-turn will know in next quarter only. It would be nice and interesting if people have observed something similar in their country s economic policies as well.
Filed under: Miscellenous Tagged: #Bank Mitra, #Bank reach, #blackmoney, #debconf17, #Demonetisation, #fake currencies, #full-fledged savings account, #Jan Dhan scheme, #Moile ATM Van, #Plastic money, #Public Sector Banks (PSB), #Reserve Bank of India, Big Mac Index

The Debian project is looking at possibly making automatic minor upgrades to installed packages the default for newly installed systems. While Debian has a reliable and stable package update system that has been an inspiration for multiple operating systems (the venerable APT), upgrades are, usually, a manual process on Debian for most users. The proposal was brought up during the Debian Cloud sprint in November by longtime Debian Developer Steve McIntyre. The rationale was to make sure that users installing Debian in the cloud have a "secure" experience by default, by installing and configuring the unattended-upgrades package within the images. The unattended-upgrades package contains a Python program that automatically performs any pending upgrade and is designed to run unattended. It is roughly the equivalent of doing apt-get update; apt-get upgrade in a cron job, but has special code to handle error conditions, warn about reboots, and selectively upgrade packages. The package was originally written for Ubuntu by Michael Vogt, a longtime Debian developer and Canonical employee. Since there was a concern that Debian cloud images would be different from normal Debian installs, McIntyre suggested installing unattended-upgrades by default on all Debian installs, so that people have a consistent experience inside and outside of the cloud. The discussion that followed was interesting as it brought up key issues one would have when deploying automated upgrade tools, outlining both the benefits and downsides to such systems.

Problems with automated upgrades An issue raised in the following discussion is that automated upgrades may create unscheduled downtime for critical services. For example, certain sites may not be willing to tolerate a master MySQL server rebooting in conditions not controlled by the administrators. The consensus seems to be that experienced administrators will be able to solve this issue on their own, or are already doing so. For example, Noah Meyerhans, a Debian developer, argued that "any reasonably well managed production host is going to be driven by some kind of configuration management system" where competent administrators can override the defaults. Debian, for example, provides the policy-rc.d mechanism to disable service restarts on certain packages out of the box. unattended-upgrades also features a way to disable upgrades on specific packages that administrators would consider too sensitive to restart automatically and will want to schedule during maintenance windows. Reboots were another issue discussed: how and when to deploy kernel upgrades? Automating kernel upgrades may mean data loss if the reboot happens during a critical operation. On Debian systems, the kernel upgrade mechanisms already provide a /var/run/reboot-required flag file that tools can monitor to notify users of the required reboot. For example, some desktop environments will popup a warning prompting users to reboot when the file exists. Debian doesn't currently feature an equivalent warning for command-line operation: Vogt suggested that the warning could be shown along with the usual /etc/motd announcement. The ideal solution here, of course, is reboot-less kernel upgrades, which is also known as "live patching" the kernel. Unfortunately, this area is still in development in the kernel (as was previously discussed here). Canonical deployed the feature for the Ubuntu 16.04 LTS release, but Debian doesn't yet have such capability, since it requires extra infrastructure among other issues. Furthermore, system reboots are only one part of the problem. Currently, upgrading packages only replaces the code and restarts the primary service shipped with a given package. On library upgrades, however, dependent services may not necessarily notice and will keep running with older, possibly vulnerable, libraries. While libc6, in Debian, has special code to restart dependent services, other libraries like libssl do not notify dependent services that they need to restart to benefit from potentially critical security fixes. One solution to this is the needrestart package which inspects all running processes and restarts services as necessary. It also covers interpreted code, specifically Ruby, Python, and Perl. In my experience, however, it can take up to a minute to inspect all processes, which degrades the interactivity of the usually satisfying apt-get install process. Nevertheless, it seems like needrestart is a key component of a properly deployed automated upgrade system.

Benefits of automated upgrades One thing that was less discussed is the actual benefit of automating upgrades. It is merely described as "secure by default" by McIntyre in the proposal, but no one actually expanded on this much. For me, however, it is now obvious that any out-of-date system will be systematically attacked by automated probes and may be taken over to the detriment of the whole internet community, as we are seeing with Internet of Things devices. As Debian Developer Lars Wirzenius said:

The ecosystem-wide security benefits of having Debian systems keep up to date with security updates by default overweigh any inconvenience of having to tweak system configuration on hosts where the automatic updates are problematic.

One could compare automated upgrades with backups: if they are not automated, they do not exist and you will run into trouble without them. (Wirzenius, coincidentally, also works on the Obnam backup software.) Another benefit that may be less obvious is the acceleration of the feedback loop between developers and users: developers like to know quickly when an update creates a regression. Automation does create the risk of a bad update affecting more users, but this issue is already present, to a lesser extent, with manual updates. And the same solution applies: have a staging area for security upgrades, the same way updates to Debian stable are first proposed before shipping a point release. This doesn't have to be limited to stable security updates either: more adventurous users could follow rolling distributions like Debian testing or unstable with unattended upgrades as well, with all the risks and benefits that implies.

Possible non-issues That there was not a backlash against the proposal surprised me: I expected the privacy-sensitive Debian community to react negatively to another "phone home" system as it did with the Django proposal. This, however, is different than a phone home system: it merely leaks package lists and one has to leak that information to get the updated packages. Furthermore, privacy-sensitive administrators can use APT over Tor to fetch packages. In addition, the diversity of the mirror infrastructure makes it difficult for a single entity to profile users. Automated upgrades do imply a culture change, however: administrators approve changes only a posteriori as opposed to deliberately deciding to upgrade parts they chose. I remember a time when I had to maintain proprietary operating systems and was reluctant to enable automated upgrades: such changes could mean degraded functionality or additional spyware. However, this is the free-software world and upgrades generally come with bug fixes and new features, not additional restrictions.

Automating major upgrades? While automating minor upgrades is one part of the solution to the problem of security maintenance, the other is how to deal with major upgrades. Once a release becomes unsupported, security issues may come up and affect older software. While Debian LTS extends releases lifetimes significantly, it merely delays the inevitable major upgrades. In the grand scheme of things, the lifetimes of Linux systems (Debian: 3-5 years, Ubuntu: 1-5 years) versus other operating systems (Solaris: 10-15 years, Windows: 10+ years) is fairly short, which makes major upgrades especially critical. While major upgrades are not currently automated in Debian, they are usually pretty simple: edit sources.list then:

    # apt-get update && apt-get dist-upgrade

But the actual upgrade process is really much more complex. If you run into problems with the above commands, you will quickly learn that you should have followed the release notes, a whopping 20,000-word, ten-section document that outlines all the gory details of the release. This is a real issue for large deployments and for users unfamiliar with the command line. The solutions most administrators seem to use right now is to roll their own automated upgrade process. For example, the Debian.org system administrators have their own process for the "jessie" (8.0) upgrade. I have also written a specification of how major upgrades could be automated that attempts to take into account the wide variety of corner cases that occur during major upgrades, but it is currently at the design stage. Therefore, this problem space is generally unaddressed in Debian: Ubuntu does have a do-release-upgrade command but it is Ubuntu-specific and would need significant changes in order to work in Debian.

Future work Ubuntu currently defaults to "no automation" but, on install, invites users to enable unattended-upgrades or Landscape, a proprietary system-management service from Canonical. According to Vogt, the company supports both projects equally as they differ in scope: unattended-upgrades just upgrades packages while Landscape aims at maintaining thousands of machines and handles user management, release upgrades, statistics, and aggregation. It appears that Debian will enable unattended-upgrades on the images built for the cloud by default. For regular installs, the consensus that has emerged points at the Debian installer prompting users to ask if they want to disable the feature as well. One reason why this was not enabled before is that unattended-upgrades had serious bugs in the past that made it less attractive. For example, it would simply fail to follow security updates, a major bug that was fortunately promptly fixed by the maintainer. In any case, it is important to distribute security and major upgrades on Debian machines in a timely manner. In my long experience in professionally administering Unix server farms, I have found the upgrade work to be a critical but time-consuming part of my work. During that time, I successfully deployed an automated upgrade system all the way back to Debian woody, using the simpler cron-apt. This approach is, unfortunately, a little brittle and non-standard; it doesn't address the need of automating major upgrades, for which I had to revert to tools like cluster-ssh or more specialized configuration management tools like Puppet. I therefore encourage any effort towards improving that process for the whole community. More information about the configuration of unattended-upgrades can be found in the Ubuntu documentation or the Debian wiki.

Note: this article first appeared in the Linux Weekly News.

In recent years, privacy issues have become a growing concern among free-software projects and users. As more and more software tasks become web-based, surveillance and tracking of users is also on the rise. While some software may use advertising as a source of revenue, which has the side effect of monitoring users, the Django community recently got into an interesting debate surrounding a proposal to add user tracking actually developer tracking to the popular Python web framework.

Tracking for funding A novel aspect of this debate is that the initiative comes from concerns of the Django Software Foundation (DSF) about funding. The proposal suggests that "relying on the free labor of volunteers is ineffective, unfair, and risky" and states that "the future of Django depends on our ability to fund its development". In fact, the DSF recently hired an engineer to help oversee Django's development, which has been quite successful in helping the project make timely releases with fewer bugs. Various fundraising efforts have resulted in major new Django features, but it is difficult to attract sponsors without some hard data on the usage of Django. The proposed feature tries to count the number of "unique developers" and gather some metrics of their environments by using Google Analytics (GA) in Django. The actual proposal (DEP 8) is done as a pull request, which is part of Django Enhancement Proposal (DEP) process that is similar in spirit to the Python Enhancement Proposal (PEP) process. DEP 8 was brought forward by a longtime Django developer, Jacob Kaplan-Moss. The rationale is that "if we had clear data on the extent of Django's usage, it would be much easier to approach organizations for funding". The proposal is essentially about adding code in Django to send a certain set of metrics when "developer" commands are run. The system would be "opt-out", enabled by default unless turned off, although the developer would be warned the first time the phone-home system is used. The proposal notes that an opt-in system "severely undercounts" and is therefore not considered "substantially better than a community survey" that the DSF is already doing.

Information gathered The pieces of information reported are specifically designed to run only in a developer's environment and not in production. The metrics identified are, at the time of writing:

• an event category (the developer commands: startproject, startapp, runserver)
• the HTTP User-Agent string identifying the Django, Python, and OS versions
• a user-specific unique identifier (a UUID generated on first run)

The proposal mentions the use of the GA aip flag which, according to GA documentation, makes "the IP address of the sender 'anonymized'". It is not quite clear how that is done at Google and, given that it is a proprietary platform, there is no way to verify that claim. The proposal says it means that "we can't see, and Google Analytics doesn't store, your actual IP". But that is not actually what Google does: GA stores IP addresses, the documentation just says they are anonymized, without explaining how. GA is presented as a trade-off, since "Google's track record indicates that they don't value privacy nearly as high" as the DSF does. The alternative, deploying its own analytics software, was presented as making sustainability problems worse. According to the proposal, Google "can't track Django users. [...] The only thing Google could do would be to lie about anonymizing IP addresses, and attempt to match users based on their IPs". The truth is that we don't actually know what Google means when it "anonymizes" data: Jannis Leidel, a Django team member, commented that "Google has previously been subjected to secret US court orders and was required to collaborate in mass surveillance conducted by US intelligence services" that limit even Google's capacity of ensuring its users' anonymity. Leidel also argued that the legal framework of the US may not apply elsewhere in the world: "for example the strict German (and by extension EU) privacy laws would exclude the automatic opt-in as a lawful option". Furthermore, the proposal claims that "if we discovered Google was lying about this, we'd obviously stop using them immediately", but it is unclear exactly how this could be implemented if the software was already deployed. There are also concerns that an implementation could block normal operation, especially in countries (like China) where Google itself may be blocked. Finally, some expressed concerns that the information could constitute a security problem, since it would unduly expose the version number of Django that is running.

In other projects Django is certainly not the first project to consider implementing analytics to get more information about its users. The proposal is largely inspired by a similar system implemented by the OS X Homebrew package manager, which has its own opt-out analytics. Other projects embed GA code directly in their web pages. This is apparently the option chosen by the Oscar Django-based ecommerce solution, but that was seen by the DSF as less useful since it would count Django administrators and wasn't seen as useful as counting developers. Wagtail, a Django-based content-management system, was incorrectly identified as using GA directly, as well. It actually uses referrer information to identify installed domains through the version updates checks, with opt-out. Wagtail didn't use GA because the project wanted only minimal data and it was worried about users' reactions. NPM, the JavaScript package manager, also considered similar tracking extensions. Laurie Voss, the co-founder of NPM, said it decided to completely avoid phoning home, because "users would absolutely hate it". But NPM users are constantly downloading packages to rebuild applications from scratch, so it has more complete usage metrics, which are aggregated and available via a public API. NPM users seem to find this is a "reasonable utility/privacy trade". Some NPM packages do phone home and have seen "very mixed" feedback from users, Voss said. Eric Holscher, co-founder of Read the Docs, said the project is considering using Sentry for centralized reporting, which is a different idea, but interesting considering Sentry is fully open source. So even though it is a commercial service (as opposed to the closed-source Google Analytics), it may be possible to verify any anonymity claims.

Debian's response Since Django is shipped with Debian, one concern was the reaction of the distribution to the change. Indeed, "major distros' positions would be very important for public reception" to the feature, another developer stated. One of the current maintainers of Django in Debian, Rapha l Hertzog, explicitly stated from the start that such a system would "likely be disabled by default in Debian". There were two short discussions on Debian mailing lists where the overall consensus seemed to be that any opt-out tracking code was undesirable in Debian, especially if it was aimed at Google servers. I have done some research to see what, exactly, was acceptable as a phone-home system in the Debian community. My research has revealed ten distinct bug reports against packages that would unexpectedly connect to the network, most of which were not directly about collecting statistics but more often about checking for new versions. In most cases I found, the feature was disabled. In the case of version checks, it seems right for Debian to disable the feature, because the package cannot upgrade itself: that task is delegated to the package manager. One of those issues was the infamous "OK Google" voice activation binary blog controversy that was previously reported here and has since then been fixed (although other issues remain in Chromium). I have also found out that there is no clearly defined policy in Debian regarding tracking software. What I have found, however, is that there seems to be a strong consensus in Debian that any tracking is unacceptable. This is, for example, an extract of a policy that was drafted (but never formally adopted) by Ian Jackson, a longtime Debian developer:

Software in Debian should not communicate over the network except: in order to, and as necessary to, perform their function[...]; or for other purposes with explicit permission from the user.

In other words, opt-in only, period. Jackson explained that "when we originally wrote the core of the policy documents, the DFSG [Debian Free Software Guidelines], the SC [Social Contract], and so on, no-one would have considered this behaviour acceptable", which explains why no explicit formal policy has been adopted yet in the Debian project. One of the concerns with opt-out systems (or even prompts that default to opt-in) was well explained back then by Debian developer Bas Wijnen:

It very much resembles having to click through a license for every package you install. One of the nice things about Debian is that the user doesn't need to worry about such things: Debian makes sure things are fine.

One could argue that Debian has its own tracking systems. For example, by default, Debian will "phone home" through the APT update system (though it only reports the packages requested). However, this is currently not automated by default, although there are plans to do so soon. Furthermore, Debian members do not consider APT as tracking, because it needs to connect to the network to accomplish its primary function. Since there are multiple distributed mirrors (which the user gets to choose when installing), the risk of surveillance and tracking is also greatly reduced. A better parallel could be drawn with Debian's popcon system, which actually tracks Debian installations, including package lists. But as Barry Warsaw pointed out in that discussion, "popcon is 'opt-in' and [...] the overwhelming majority in Debian is in favour of it in contrast to 'opt-out'". It should be noted that popcon, while opt-in, defaults to "yes" if users click through the install process. [Update: As pointed out in the comments, popcon actually defaults to "no" in Debian.] There are around 200,000 submissions at this time, which are tracked with machine-specific unique identifiers that are submitted daily. Ubuntu, which also uses the popcon software, gets around 2.8 million daily submissions, while Canonical estimates there are 40 million desktop users of Ubuntu. This would mean there is about an order of magnitude more installations than what is reported by popcon. Policy aside, Warsaw explained that "Debian has a reputation for taking privacy issues very serious and likes to keep it".

Next steps There are obviously disagreements within the Django project about how to handle this problem. It looks like the phone-home system may end up being implemented as a proxy system "which would allow us to strip IP addresses instead of relying on Google to anonymize them, or to anonymize them ourselves", another Django developer, Aymeric Augustin, said. Augustin also stated that the feature wouldn't "land before Django drops support for Python 2", which is currently estimated to be around 2020. It is unclear, then, how the proposal would resolve the funding issues, considering how long it would take to deploy the change and then collect the information so that it can be used to spur the funding efforts. It also seems the system may explicitly prompt the user, with an opt-out default, instead of just splashing a warning or privacy agreement without a prompt. As Shai Berger, another Django contributor, stated, "you do not get [those] kind of numbers in community surveys". Berger also made the argument that "we trust the community to give back without being forced to do so"; furthermore:

I don't believe the increase we might get in the number of reports by making it harder to opt-out, can be worth the ill-will generated for people who might feel the reporting was "sneaked" upon them, or even those who feel they were nagged into participation rather than choosing to participate.

Other options may also include gathering metrics in pip or PyPI, which was proposed by Donald Stufft. Leidel also proposed that the system could ask to opt-in only after a few times the commands are called. It is encouraging to see that a community can discuss such issues without heating up too much and shows great maturity for the Django project. Every free-software project may be confronted with funding and sustainability issues. Django seems to be trying to address this in a transparent way. The project is willing to engage with the whole spectrum of the community, from the top leaders to downstream distributors, including individual developers. This practice should serve as a model, if not of how to do funding or tracking, at least of how to discuss those issues productively. Everyone seems to agree the point is not to surveil users, but improve the software. As Lars Wirzenius, a Debian developer, commented: "it's a very sad situation if free software projects have to compromise on privacy to get funded". Hopefully, Django will be able to improve its funding without compromising its principles.

Note: this article first appeared in the Linux Weekly News.

Today Gammu 1.38.0 has been released. Changes in last two testing releases have been stabilized and this is the outcome. You can expect changes in API or SMSD tables as well as some additional features. Also this is first stable release after several years which comes with Windows binaries. These are built using AppVeyor and will help bring Windows users back to latest versions. Full list of changes and new features can be found on Gammu 1.38.0 release page. Would you like to see more features in Gammu? You an support further Gammu development at Bountysource salt or by direct donation.

Filed under: Debian English Gammu 0 comments