Last month, we released nanotime, a package to work with nanosecond timestamps. See the initial release announcement for some background material and a few first examples. nanotime relies on the RcppCCTZ package for high(er) resolution time parsing and formatting: R itself stops a little short of a microsecond. And it uses the bit64 package for the actual arithmetic: time at this granularity is commonly represented at (integer) increments (at nanosecond resolution) relative to an offset, for which the standard epoch of Januar 1, 1970 is used. int64 types are a perfect match here, and bit64 gives us an integer64. Naysayers will point out some technical limitations with R's S3 classes, but it works pretty much as needed here. The one thing we did not have was Windows support. RcppCCTZ and the CCTZ library it uses need real C++11 support, and the g++-4.9 compiler used on Windows falls a little short lacking inter alia a suitable std::get_time() implementation. Enter Dan Dillon who ported this from LLVM's libc++ which lead to Sunday's RcppCCTZ 0.2.0 release. And now we have all our ducks in a row: everything works on Windows too. The next paragraph summarizes the changes for both this release as well as the initial one last month:

Changes in version 0.1.0 (2017-01-10)

• Added Windows support thanks to expanded RcppCCTZ (closes #6)
• Added "mocked up" demo with nanosecond delay networking analysis
• Added 'fmt' and 'tz' options to output functions, expanded format.nanotime (closing #2 and #3)
• Added data.frame support
• Expanded tests

Changes in version 0.0.1 (2016-12-15)

• Initial CRAN upload.
• Package is functional and provides examples.

We also have a diff to the previous version thanks to CRANberries. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Debian LTS November marked the 20th month I contributed to Debian LTS under the Freexian umbrella. I had 8 hours allocated which I used by:

• some rather quiet frontdesk days
• updating icedove to 45.5.1 resulting in DLA-752-1 fixing 7 CVEs
• looking whether Wheezy is affected by xsa-202, xsa-203, xsa-204 and handling the communication with credativ for these (update not yet released)
• Assessing cURL/libcURL CVE-2016-9586
• Assessing whether Wheezy's QEMU is affeced by security issues in 9pfs "proxy" and "handle" code
• Releasing DLA-776-1 for samba fixing CVE-2016-2125

Other Debian stuff

• Allow travis.debian.net to start services when running autopkgtests
• Usual bunch of libvirt and related uploads
• Uploaded git-buildpackage 0.8.8 and 0.8.9 to unstable (list of changes)

Some other Free Software activites

• Some fixes to libvirt's configure.ac for merged /usr
• Made some progress on getting the merkur board clones to life. The first stage bootloader and contiki are working now.
• Some ansible_spec fixes for roles in non default directories and robustness
• Due to excessive XMPP spam started a jabber-spam-blacklist
• Unbroke ansible-module-foreman
• Added support to configure Foreman settings via ansible-module-foreman and python-foreman
• Speed up VM removal with Foreman and VMware ESX
• Started foreman_serverspec a Foreman plugin to store serverspec reports in the Foreman. This is in a very early stage.

Finding a Libc for tiny embedded ARM systems You'd think this problem would have been solved a long time ago. All I wanted was a C library to use in small embedded systems -- those with a few kB of flash and even fewer kB of RAM. Small system requirements A small embedded system has a different balance of needs:

• Stack space is limited. Each thread needs a separate stack, and it's pretty hard to move them around. I'd like to be able to reliably run with less than 512 bytes of stack.
• Dynamic memory allocation should be optional. I don't like using malloc on a small device because failure is likely and usually hard to recover from. Just make the linker tell me if the program is going to fit or not.
• Stdio doesn't have to be awesomely fast. Most of our devices communicate over full-speed USB, which maxes out at about 1MB/sec. A stdio setup designed to write to the page cache at memory speeds is over-designed, and likely involves lots of buffering and fancy code.
• Everything else should be fast. A small CPU may run at only 20-100MHz, so it's reasonable to ask for optimized code. They also have very fast RAM, so cycle counts through the library matter.

Available small C libraries I've looked at:

• Clibc. This targets embedded Linux systems, and also appears dead at this time.
• musl libc. A more lively project; still, definitely targets systems with a real Linux kernel.
• dietlibc. Hasn't seen any activity for the last three years, and it isn't really targeting tiny machines.
• newlib. This seems like the 'normal' embedded C library, but it expects a fairly complete "kernel" API and the stdio bits use malloc.
• avr-libc. This has lots of Atmel assembly language, but is otherwise ideal for tiny systems.
• pdclib. This one focuses on small source size and portability.

Current AltOS C library We've been using pdclib for a couple of years. It was easy to get running, but it really doesn't match what we need. In particular, it uses a lot of stack space in the stdio implementation as there's an additional layer of abstraction that isn't necessary. In addition, pdclib doesn't include a math library, so I've had to 'borrow' code from other places where necessary. I've wanted to switch for a while, but there didn't seem to be a great alternative. What's wrong with newlib? The "obvious" embedded C library is newlib. Designed for embedded systems with a nice way to avoid needing a 'real' kernel underneath, newlib has a lot going for it. Most of the functions have a good balance between speed and size, and many of them even offer two implementations depending on what trade-off you need. Plus, the build system 'just works' on multi-lib targets like the family of cortex-m parts. The big problem with newlib is the stdio code. It absolutely requires dynamic memory allocation and the amount of code necessary for 'printf' is larger than the flash space on many of our devices. I was able to get a cortex-m3 application compiled in 41kB of code, and that used a smattering of string/memory functions and printf. How about avr libc? The Atmel world has it pretty good -- avr-libc is small and highly optimized for atmel's 8-bit avr processors. I've used this library with success in a number of projects, although nothing we've ever sold through Altus Metrum. In particular, the stdio implementation is quite nice -- a 'FILE' is effectively a struct containing pointers to putc/getc functions. The library does no buffering at all. And it's tiny -- the printf code lacks a lot of the fancy new stuff, which saves a pile of space. However, much of the places where performance is critical are written in assembly language, making it pretty darn hard to port to another processor. Mixing code together for fun and profit! Today, I decided to try an experiment to see what would happen if I used the avr-libc stdio bits within the newlib environment. There were only three functions written in assembly language, two of them were just stubs while the third was a simple ultoa function with a weird interface. With those coded up in C, I managed to get them wedged into newlib. Figuring out the newlib build system was the only real challenge; it's pretty awful having generated files in the repository and a mix of autoconf 2.64 and 2.68 version dependencies. The result is pretty usable though; my STM 32L discovery board demo application is only 14kB of flash while the original newlib stdio bits needed 42kB and that was still missing all of the 'syscalls', like read, write and sbrk. Here's gitweb pointing at the top of the tiny-stdio tree: gitweb And, of course you can check out the whole thing

git clone git://keithp.com/git/newlib

'master' remains a plain upstream tree, although I do have a fix on that branch. The new code is all on the tiny-stdio branch. I'll post a note on the newlib mailing list once I've managed to subscribe and see if there is interest in making this option available in the upstream newlib releases. If so, I'll see what might make sense for the Debian libnewlib-arm-none-eabi packages.

In my previous blog I had tried to explain what White Labelling is and my approach of implementing it in Lumicall. This week I went ahead with the implementation by using productFlavors feature in Android. I baked my cake!
I ve created two flavors :

1. Lumicall (which runs like the default version)
2. Whitelabel

To switch to the desired version, there is a build variant window on the bottom left of Android Studio, when you open it, you can change your current active flavor from the ones you defined. So the idea is, there would be different flavors for each client. And all the client specific resources would go under src/(flavorname) folder. The client specific resources could be:

1. Application name
2. Client s logo (Drawable)
3. Details about the client s organization which would go under about tab
4. Colors / Themes (Colors.xml)
5. Strings (Strings.xml)
6. Additional Files which include new features

To understand how to modify these resources in the flavored version, Let s take an example in which we would like to replace the application name from from Lumicall to, lets say ClientApp .

1. Go to file : /res/values/strings.xml which has the application name <string name= app_name >Lumicall</string>
2. To replace it with client s app name, You d have to create a new strings.xml file (Note : Same name as that of the existing strings.xml file ) in the directory LumicallWhitelabel/src/whitelabel/res/values/strings.xml So, in our project, there are two strings.xml file. One is /res/values/strings.xml and one flavored specfic file in LumicallWhitelabel/src/whitelabel/res/values/strings.xml.
   
   Only add the values which would be different in the flavor version.
   If there are particular things which you d like to be same, then there is no need of adding them again with the same value in the flavored file.
   
   -Just define the values you would want to replace. Not the values you would like to be same- Gradle will take care of the overwriting while merging the resources when you run the flavored version.

Now, the most important thing is changing the Application ID. In my previous blog I explained the difference between ApplicationID and package name. I also added a snippet from my build.gradle file which would suffix .whitelabel at the end of the orignal applicationID. So for configuring ApplicationID for each flavor, add applicationIdSuffix suffix_you_want .

Link to the the cake : https://github.com/Urvika-gola/LumicallWhitelabel Thanks for reading,
U

Do you have a large iCalendar file with lots of old entries, and would like to archive them to save space and resources? At least those of us using KOrganizer know that turning on and off an event set become slower and slower the more entries are in the set. While working on migrating our calendars to a Radicale CalDAV server on our Freedombox server, my loved one wondered if I could find a way to split up the calendar file she had in KOrganizer, and I set out to write a tool. I spent a few days writing and polishing the system, and it is now ready for general consumption. The code for ical-archiver is publicly available from a git repository on github. The system is written in Python and depend on the vobject Python module. To use it, locate the iCalendar file you want to operate on and give it as an argument to the ical-archiver script. This will generate a set of new files, one file per component type per year for all components expiring more than two years in the past. The vevent, vtodo and vjournal entries are handled by the script. The remaining entries are stored in a 'remaining' file. This is what a test run can look like:

% ical-archiver t/2004-2016.ics 
Found 3612 vevents
Found 6 vtodos
Found 2 vjournals
Writing t/2004-2016.ics-subset-vevent-2004.ics
Writing t/2004-2016.ics-subset-vevent-2005.ics
Writing t/2004-2016.ics-subset-vevent-2006.ics
Writing t/2004-2016.ics-subset-vevent-2007.ics
Writing t/2004-2016.ics-subset-vevent-2008.ics
Writing t/2004-2016.ics-subset-vevent-2009.ics
Writing t/2004-2016.ics-subset-vevent-2010.ics
Writing t/2004-2016.ics-subset-vevent-2011.ics
Writing t/2004-2016.ics-subset-vevent-2012.ics
Writing t/2004-2016.ics-subset-vevent-2013.ics
Writing t/2004-2016.ics-subset-vevent-2014.ics
Writing t/2004-2016.ics-subset-vjournal-2007.ics
Writing t/2004-2016.ics-subset-vjournal-2011.ics
Writing t/2004-2016.ics-subset-vtodo-2012.ics
Writing t/2004-2016.ics-remaining.ics
%

As you can see, the original file is untouched and new files are written with names derived from the original file. If you are happy with their content, the *-remaining.ics file can replace the original the the others can be archived or imported as historical calendar collections. The script should probably be improved a bit. The error handling when discovering broken entries is not good, and I am not sure yet if it make sense to split different entry types into separate files or not. The program is thus likely to change. If you find it interesting, please get in touch. :) As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

What happened in the Reproducible Builds effort between Sunday December 25 and Saturday December 31 2016: Media coverage

• Brett Smith wrote a report on our recent summit meeting on behalf of the Software Freedom Conservancy.

Reproducible bugs filed Chris West:

• #849667 filed against htslib.

Chris Lamb:

• #849314 filed against node-gulp.
• #849333 filed against faker.

Rob Browning:

• #849692 filed against oneliner-el.

Reviews of unreproducible packages 7 package reviews have been added, 12 have been updated and 14 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• build_id_differences_only
• cython_captures_build_path

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris West (19)
• Chris Lamb (7)
• Rob Browning (1)

diffoscope development

• Chris Lamb:
  • Optimisations:
    • Avoid unnecessary string manipulation writing --text output (~20x speedup).
    • Avoid n iterations over archive files (~8x speedup).
    • Memoize calls to distutils.spawn.find_executable to avoid excessive stat(1) syscalls.
  • Progress bar:
    • Show current file / ELF section under analysis etc. in progress bar.
    • Move the --status-fd output to use JSON and to include the current filename.
  • Code tidying:
    • Split out the try.diffoscope.org client so that it can be released separately on PyPI.
    • Completely rework the diffoscope and diffoscope.comparators modules, grouping similar utilities into their own modules, etc.
  • Miscellaneous:
    • Ensure that running from Git will always use that checkout's Python modules.
• Mattia Rizzolo:
  • Follow the rename of the debian package 'python-jsbeautifier' to 'jsbeautifier'
• Reiner Herrmann:
  • apk: Extend recognition regex to also match zip archives
• siamezzze:
  • Fixed no newline being classified as order-like difference.

strip-nondeterminism development

• strip-nondeterminism 0.029-2 was uploaded to unstable by Mattia Rizzolo to include a fix by Chris Lamb to fix the autopkgtest tests.

try.diffoscope.org development

• Chris Lamb:
  • Show progress bar and position in queue, etc.
  • Promote command-line client with PyPI instructions.
  • Increase comparison time limit to 90 seconds.

tests.reproducible-builds.org

• Run half of the arm64 build nodes in the future. (h01ger)
• Resume testing scheduling (on i386 and armhf) now #846564 and #844701 bugs in dpkg are fixed in that suite. (h01ger)

Misc. This week's edition was written by Chris Lamb, Holger Levsen and was reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

May as well start 2017 with a burst of recorded optimism: the last books I bought in 2016 that I'm queuing up to read. The hopoe is that this year I'll actually read more of them! Becky Chambers A Closed and Common Orbit (sff)
T. Kingfisher The Raven and the Reindeer (sff)
Joseph R. Lallo The Book of Deacon Anthology (sff)
M. Louisa Locke Maids of Misfortune (historical)
Rebecca Solnit Hope in the Dark (nonfiction)
K.B. Spangler Maker Space (sff)
K.B. Spangler State Machine (sff)
Steven W. White New World (sff) Most of these are various StoryBundle add-ons that I'd somehow missed downloading the first time (and hence are fairly low priority on the reading list). The rest is a mixed bag of Kindle purchases. I started A Closed and Common Orbit today and could barely put it down. An auspicious start to the new year.

A Safe and Happy New Year to all. While Debconf India is still a pipe-dream as of now, did see that India has been gradually doing it easier for tourists and casual business visitors to come visit India. This I take as very positive development for India itself. The 1st condition is itself good for anybody visiting India

Eligibility International Travellers whose sole objective of visiting India is recreation , sight-seeing , casual visit to meet friends or relatives, short duration medical treatment or casual business visit.

https://indianvisaonline.gov.in/visa/tvoa.html That this facility is being given to 130 odd countries is better still

Albania, Andorra, Anguilla, Antigua & Barbuda, Argentina, Armenia, Aruba, Australia, Austria, Bahamas, Barbados, Belgium, Belize, Bolivia, Bosnia & Herzegovina, Botswana, Brazil, Brunei, Bulgaria, Cambodia, Canada, Cape Verde, Cayman Island, Chile, China, China- SAR Hong-Kong, China- SAR Macau, Colombia, Comoros, Cook Islands, Costa Rica, Cote d lvoire, Croatia, Cuba, Czech Republic, Denmark, Djibouti, Dominica, Dominican Republic, East Timor, Ecuador, El Salvador, Eritrea, Estonia, Fiji, Finland, France, Gabon, Gambia, Georgia, Germany, Ghana, Greece, Grenada, Guatemala, Guinea, Guyana, Haiti, Honduras, Hungary, Iceland, Indonesia, Ireland, Israel, Jamaica, Japan, Jordan, Kenya, Kiribati, Laos, Latvia, Lesotho, Liberia, Liechtenstein, Lithuania, Luxembourg, Madagascar, Malawi, Malaysia, Malta, Marshall Islands, Mauritius, Mexico, Micronesia, Moldova, Monaco, Mongolia, Montenegro, Montserrat, Mozambique, Myanmar, Namibia, Nauru, Netherlands, New Zealand, Nicaragua, Niue Island, Norway, Oman, Palau, Palestine, Panama, Papua New Guinea, Paraguay, Peru, Philippines, Poland, Portugal, Republic of Korea, Republic of Macedonia, Romania, Russia, Saint Christopher and Nevis, Saint Lucia, Saint Vincent & the Grenadines, Samoa, San Marino, Senegal, Serbia, Seychelles, Singapore, Slovakia, Slovenia, Solomon Islands, South Africa, Spain, Sri Lanka, Suriname, Swaziland, Sweden, Switzerland, Taiwan, Tajikistan, Tanzania, Thailand, Tonga, Trinidad & Tobago, Turks & Caicos Island, Tuvalu, UAE, Ukraine, United Kingdom, Uruguay, USA, Vanuatu, Vatican City-Holy See, Venezuela, Vietnam, Zambia and Zimbabwe.

This should make it somewhat easier for any Indian organizer as well as any participants from any of the member countries shared. There is possibility that this list would even get longer, provided we are able to scale our airports and all and any necessary infrastructure that would be needed for International Visitors to have a good experience. What has been particularly interesting is to know which ports of call are being used by International Visitors as well as overall growth rate

The Percentage share of Foreign Tourist Arrivals (FTAs) in India during November, 2016 among the top 15 source countries was highest from USA (15.53%) followed by UK (11.21%), Bangladesh (10.72%), Canada (4.66%), Russian Fed (4.53%), Australia (4.04%), Malaysia (3.65%), Germany (3.53%), China (3.14%), France (2.88%), Sri Lanka (2.49%), Japan (2.49%), Singapore (2.16%), Nepal (1.46%) and Thailand (1.37%).

And port of call

The Percentage share of Foreign Tourist Arrivals (FTAs) in India during November 2016 among the top 15 ports was highest at Delhi Airport (32.71%) followed by Mumbai Airport (18.51%), Chennai Airport (6.83%), Bengaluru Airport (5.89%), Haridaspur Land check post (5.87%), Goa Airport (5.63%), Kolkata Airport (3.90%), Cochin Airport (3.29%), Hyderabad Airport (3.14%), Ahmadabad Airport (2.76%), Trivandrum Airport (1.54%), Trichy Airport (1.53%), Gede Rail (1.16%), Amritsar Airport (1.15%), and Ghojadanga land check post (0.82%) .

The Ghojadanga land check post seems to be between West Bengal, India and Bangladesh. Gede Railway Station is also in West Bengal as well. So all and any overlanders could take any of those ways.Even Hardispur Land Check post comes in the Bengal-Bangladesh border only. In the airports, Delhi Airport seems to be attracting lot more business than the Mumbai Airport. Part of the reason I *think* is the direct link of Delhi Airport to NDLS via the Delhi Airport Express Line . The same when it will happen in Mumbai should be a game-changer for city too. Now if you are wondering why I have been suddenly talking about visas and airports in India, it came because Hong Kong is going to Withdraw Visa Free Entry Facility For Indians. Although, as rightly pointed out in the article doesn t make sense from economic POV and seems to be somewhat politically motivated. Not that I or anybody else can do anything about that. Seeing that, I thought it was a good opportunity to see how good/Bad our Government is and it seems to be on the right path. Although the hawks (Intelligence and Counter-Terrorist Agencies) will probably become a bit more paranoid , their work becomes tougher.
Filed under: Miscellenous Tagged: #Airport Metro Line 3, #CSIA, #Incredible India, #India, #International Tourism

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. If you re interested in Android, Java, Games and LTS topics, this might be interesting for you. Debian Android

• I sponsored a new package for Seamlik called android-platform-tools-analytics-library.

Debian Games

• We have entered the final straight for Stretch, so I kept a close eye on new game releases and bug reports in packages which I think should be part of the next stable release. Bzflag is certainly one of them, a tank battling game that can be played in the first-person perspective and which has arrived in version 2.4.8. I also packaged new releases of trigger-rally, a racing game, Renpy, pygame-sdl2 and Minetest.
• B lint R czey introduced libopenhmd to Debian a while ago and asked me in #845657 to enable OpenHMD support for neverball. Neverball is now the first game in the archive, at least as far as I know, that is ready for virtual reality. I have never tried it though because I don t own the necessary gear from Oculus myself but it sounds like a cool feature.
• A user of caveexpress reported a bug (#847147) in one level that prevented him from finishing it. I forwarded this one to upstream and he was able to quickly fix the issue and I could release 2.4+git20160609-3 later.
• I triaged several RC bugs which were reported against our D language games and it turned out that the bug was in gdc (#845377).
• I also made some small improvements to monopd s packaging and applied a patch from Laurent Bigonville to Freeciv that corrected a problem with AppData files (#848720).
• I worked around another RC FTBFS bug in spring (#846921) which is apparently a regression in binutils (#847356) but its maintainer does not consider this to be release critical.
• I tried to fix #848063 in ri-li but it seems to surface again under special circumstances. Since compilation works on all buildds for all release architectures and on my systems I downgraded the severity to important.
• I uploaded Bullet 2.85.1 to experimental. It is currently waiting in the NEW queue due to the SONAME bump and because I decided to simplify the packaging. I don t think it is longer worth it to provide several standalone binary packages. All Bullet 2 and 3 core libraries can be found in libbullet2.85 now while all the extra stuff is part of libbullet-extras2.85.
• Last but not least I released debian-games 1.7 and updated the list of games. Castle Combat was removed this month from Debian.

Debian Java

• I promised to complete my work on Netbeans and spent several hours on completing this goal. Although I am aware of the package s obstacles by now it never ceases to amaze me how many build failures occur due to the apparent mismatch between Debian (build-)dependencies and the ones upstream is using. This time I got stuck when several core modules could not be built from source because they required classes from OpenJDK 9. Since JDK 9 will not be released with Stretch I am afraid to say that Netbeans 8.2 won t be part of it. I worked around these issues by replacing the modules with versions from 8.1 but this is obviously just a very crude hack. Nevertheless the new package is available in experimental now.
• New upstream releases this month: activemq, netbeans, libnb-platform18-java, svnclientadapter, jackrabbit, jboss-xnio, undertow.
• Package updates: jboss-classfilewriter, jboss-logging, libpicocontainer-java.
• RC bug fixes: jruby-maven-plugins (#844841), vorbis-java (#844753), jackson-datatype-guava (#849449), triaged libhamcrest-java (#846116).
• I sponsored freeplane 1.5.18-1 for Felix Natter.

Debian LTS This was my tenth month as a paid contributor and I have been paid to work 13,5 hours on Debian LTS, a project started by Rapha l Hertzog. In that time I did the following:

• From 12. December until 18. December I was in charge of our LTS frontdesk. I triaged bugs in jasper, openjdk-6, bluez, game-music-emu, simplesamlphp, imagemagick, nagios3, most, rabbitmq-server, html5lib and dcmtk.
• DLA-742-1. Issued a security update for chrony fixing 1 CVE. This update was prepared by Vincent Blut.
• DLA-745-1. Issued a security update for most fixing 1 CVE.
• DLA-746-1. Issued a security update for tomcat6 fixing 1 CVE and two regressions from previous updates which were reported to Debian s bug tracker.
• DLA-747-1. Issued a security update for libupnp fixing 1 CVE.
• DLA-748-1. Issued a security update for libupnp4 fixing 1 CVE.
• DLA-746-2. Issued a regression update for tomcat6.
• DLA-753-1. Issued a security update for tomcat7 fixing 1 CVE and three regressions that were similar in nature to the ones fixed in Tomcat 6.
• DLA-761-1. Issued a security update for python-bottle fixing 1 CVE.
• DLA-763-1. Issued a security update for squid3 fixing 1 CVE.
• DLA-766-1. Issued a security update for libcrypto++ fixing 1 CVE.
• I also worked on two CVEs for Asterisk, an Open Source PBX and telephony toolkit. The work is done and can currently be found at this location. I asked on the debian-lts mailing list for feedback and testing and already got some positive feedback. I will wait a few more days before I release the security update.

Non-maintainer uploads

• I did two NMUs this month. I sponsored an upload of libtorrent for Peter Pentchev fixing #828414 and I fixed a trivial bug in gnash myself (#845847).

systemd: Masking units [archive]

A good explanation of the three level of "stopping" a service in systemd, with a focus on masking.

How Skype fixes security vulnerabilities

This post describes my fruitless effort to convince Microsoft employees that Their service is vulnerable, and the humiliation one has to go through should One s account be blocked by a hacker. This is a story of ignorance, pain and Despair.

Mapping the Shadows of New York City: Every Building, Every Block

You re looking at a map of all of the shadows produced by thousands of buildings in New York City over the course of one day. This inverted view tells the story of the city s skyline at the ground level.

Back in 2010, inspired by Juliet, I set about doing 101 things in 1001 days. I had various levels of success, but one of the things I did complete was the aim of watching half of the IMDB Top 250. I didn t stop at that point, but continued to work through it at a much slower pace until I realised that through the Queen s library I had access to quite a few DVDs of things I was missing, and that it was perfectly possible to complete the list by the end of 2016. So I did. I should point out that I didn t set out to watch the list because I m some massive film buff. It was more a mixture of watching things that I wouldn t otherwise choose to, and also watching things I knew were providing cultural underpinnings to films I had already watched and enjoyed. That said, people have asked for some sort of write up when I was done. So here are some random observations, which are almost certainly not what they were looking for.

My favourite film is not in the Top 250 First question anyone asks is What s your favourite film? . That depends a lot on what I m in the mood for really, but fairly consistently my answer is The Hunt for Red October. This has never been in the Top 250 that I ve noticed. Which either says a lot about my taste in films, or the Top 250, or both. Das Boot was in the list and I would highly recommend it (but then I like all submarine movies it seems).

The Shawshank Redemption is overrated I can t recall a time when The Shawshank Redemption was not top of the list. It s a good film, and I ve watched it many times, but I don t think it s good enough to justify its seemingly unbroken run. I don t have a suggestion for a replacement, however.

The list is constantly changing I say I ve completed the Top 250, but that s working from a snapshot I took back in 2010. Today the site is telling me I ve watched 215 of the current list. Last night it was 214 and I haven t watched anything in between. Some of those are films released since 2010 (in particular new releases often enter high and then fall out of the list over a month or two), but the current list has films as old as 1928 (The Passion of Joan of Arc) that weren t there back in 2010. So keeping up to date is not simply a matter of watching new releases.

The best way to watch the list is terrestrial TV There were various methods I used to watch the list. Some I d seen in the cinema when they came out (or was able to catch that way anyway - the QFT showed Duck Soup, for example). Netflix and Amazon Video had some films, but overall a very disappointing percentage. The QUB Library, as previously mentioned, had a good number of DVDs on the list (especially the older things). I ended up buying a few (Dial M for Murder on 3D Bluray was well worth it; it s beautifully shot and unobtrusively 3D), borrowed a few from friends and ended up finishing off the list by a Lovefilm one month free trial. The single best source, however, was UK terrestrial TV. Over the past 6 years Freeview (the free-to-air service here) had the highest percentage of the list available. Of course this requires some degree of organisation to make sure you don t miss things.

Films I enjoyed Not necessarily my favourite, but things I wouldn t have necessarily watched and was pleasantly surprised by. No particular order, and I m leaving out a lot of films I really enjoyed but would have got around to watching anyway.

• Clint Eastwood films - Gran Torino and Million Dollar Baby were both excellent but neither would have appealed to me at first glance. I hated Unforgiven though.
• Jimmy Stewart. I m not a fan of It s a Wonderful Life (which I d already watched because it s Lister s favourite film), but Harvey is obviously the basis of lots of imaginary friend movies and Rear Window explained a Simpsons episode (there were a lot of Simpsons episodes explained by watching the list).
• Spaghetti Westerns. I wouldn t have thought they were my thing, but I really enjoyed the Sergio Leone films (A Fistful of Dollars etc.). You can see where Tarantino gets a lot of his inspiration.
• Foreign language films. I wouldn t normally seek these out. And in general it seems I cannot get on with Italian films (except Life is Beautiful), but Amores Perros, Amelie and Ikiru were all better than expected.
• Kind Hearts and Coronets. For some reason I didn t watch this until almost the end; I think the title always put me off. Turned out to be very enjoyable.

Films I didn t enjoy I m sure these mark me out as not being a film buff, but there are various things I would have turned off if I d caught them by accident rather than setting out to watch them.

• Charlie Chaplin. The Kid is the only one I really enjoyed. Nothing else did it for me.
• Italian films. La Strada and Nights of Cabiria in particular. Oh, and 8 . Hated it.
• Who s Afraid of Virginia Woolf?. I just didn t get it.
• Metropolis. Too damn long.

I ve kept the full list available, if you re curious.

Here is my monthly update covering what I have been doing in the free software world (previous month):

• Celebrated my 10-year anniversary of contributing to Debian. An excerpt of this post was quoted on LWN.
• Made a number of improvements to AptFS, my FUSE-based filesystem that provides a view on unpacked Debian source packages as regular folders, including move from the popen2 Python module to subprocess and correcting the parsing of package lists.
• Corrected an UnboundLocalError exception in the Finnish social security number generator in faker, a tool to generate test data in Python applications. (#441)
• Made a small change to travis.debian.net (my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every code change) to fix an issue with malformed YAML.
• Added the ability to specify the clone target to gbp-import-dsc etc. in git-buildpackage, a tool to build Debian packages using Git. (commit)
• Filed three issues against the Redis key-value database:
  • Tests fail on the alpha architecture due to "memory efficiency". (#3666)
  • Please update hiredis (#3687)
  • Correct "whenever" typo. (#3652)

---

Burkeman: Why time management is ruining our lives Over the past semester I ve been trying to convince one graduate student and one professor in my department to use Inbox Zero to get a better handle on their e-mail inboxes. The goal is not to be more productive. The two of them get far more academic work done than I do. However, both of them are far more stressed than I am. And in the case of the graduate student, I have to add items to my own to-do list to chase up e-mails that I ve sent him, which only spreads this stress and tension around. The graduate student sent me this essay by Oliver Burkeman about how these techniques can backfire, creating more stress, tension and anxiety. It seems to me that this happens when we think of these techniques as having anything to do with productivity. Often people will say use this technique and you ll be less stressed, more productive, and even more productive because you re less stressed. Why not just say use this technique and you ll be less anxious and stressed ? This is a refusal to treat lower anxiety as merely a means to some further end. People can autonomously set their own ends, and they ll probably do a better job of this when they re less anxious. Someone offering a technique to help with their sense of being overwhelmed need not tell them what to do with their new calm. It might be argued that this response to Burkeman fails to address the huge sense of obligation that an e-mail inbox can generate. Perhaps the only sane response to this infinite to-do list is to let it pile up. If we follow a technique like Inbox Zero, don t we invest our inbox with more importance than it has? Like a lot of areas of life, the issue is that the e-mails that will advance truly valuable projects and relationships, projects of both ourselves and of others, are mixed in with reams of stuff that doesn t matter. We face this situation whenever we go into a supermarket, or wonder what to do during an upcoming vacation. In all these situations, we have a responsibility to learn how to filter the important stuff out, just as we have a responsibility to avoid reading celebrity gossip columns when we are scanning through a newspaper. Inbox Zero is a technique to do that filtering in the case of e-mail. Just letting our inbox pile up is an abdication of responsibility, rather than an intelligent response to a piece of technology that most of the world abuses.

Debian Long Term Support (LTS) Those were 8th and 9th months working on Debian LTS started by Raphael Hertzog at Freexian. I had trouble resuming work in November as I had taken a long break during the month and started looking at issues only during the last week of November.

Imagemagick, again I have, again, spent a significant amount of time fighting the ImageMagick (IM) codebase. About 15 more vulnerabilities were found since the last upload, which resulted in DLA-756-1. In the advisory, I unfortunately forgot to mention CVE-2016-8677 and CVE-2016-9559, something that was noticed by my colleague Roberto after the upload... More details about the upload are available in the announcement. When you consider that I worked on IM back in october, which lead to an upload near the end of November covering around 80 more vulnerabilities, it doesn't look good for the project at all. Of the 15 vulnerabilities I worked on, only 6 had CVEs assigned and I had to request CVEs for the other 9 vulnerabilities plus 11 more that were still unassigned. This lead to the assignment of 25 distinct CVE identifiers as a lot of issues were found to be distinct enough to warrant their own CVEs. One could also question how many of those issues affect the fork, Graphicsmagick. A lot of the vulnerabilities were found through fuzzing searches that may not have been tested on Graphicsmagick. It seems clear to me that a public corpus of test data should be available to test regressions and cross-project vulnerabilities. It's already hard enough to track issues withing IM itself, I can't imagine what it would be for the fork to keep track of those issues, especially since upstream doesn't systematically request CVEs for issues that they find, a questionable practice considering the number of issues we all need to keep track of.

Nagios I have also worked on the Nagios package and produced DLA 751-1 which fixed two fairly major issues (CVE-2016-9565 and CVE-2016-9566) that could allow remote root access under certain conditions. Fortunately, the restricted permissions setup by default in the Debian package made both exploits limited to information disclosure and privilege escalation if the debug log is enabled. This says a lot about how proper Debian packaging can help in limiting the attack surface of certain vulnerabilities. It was also "interesting" to have to re-learn dpatch to add patches to the package: I regret not converting it to quilt, as the operation is simple and quilt is so much easier to use. People new to Debian packaging may be curious to learn about the staggering number of patching systems historically used in Debian. On that topic, I started a conversation about how much we want to reuse existing frameworks when we work on those odd packages, and the feedback was interesting. Basically, the answer is "it depends"...

NSS I had already worked on the package in November and continued the work in December. Most of the work was done by Raphael, which fixed a lot of issues with the test suite. I tried to wrap this up by fixing CVE-2016-9074, the build on armel and the test suite. Unfortunately, I had to stop again because I ran out of hours and the fips test suite was still failing, but fortunately Raphael was able to complete the work with DLA-759-1. As things stand now, the package is in better shape than in other suites as tests (Debian bug #806639) and autopkgtest (Debian bug #806207) are still not shipped in the sid or stable releases.

Other work For the second time, I forgot to formally assign myself a package before working on it, which meant that I wasted part of my hours working on the monit package. Those hours, of course, were not counted in my regular hours. I still spent some time reviewing mejo's patch to ensure it was done properly and it turned out we both made similar patches working independently, always a good sign. As I reported in my preliminary November report, I have also triaged issues in libxml2, ntp, openssl and tiff. Finally, I should mention my short review of the phpMyAdmin upload, among the many posts i sent to the LTS mailing list.

Other free software work One reason why I had so much trouble getting paid work done in November is that I was busy with unpaid work...

manpages.debian.org A major time hole for me was trying to tackle the manpages.debian.org service, which had been offline since August. After a thorough evaluation of the available codebases, I figured the problem space wasn't so hard and it was worth trying to do an implementation from scratch. The result is a tool called debmans. It took, obviously, way longer than I expected, as I experimented with Python libraries I had been keeping an eye on for a while. For the commanline interface, I used the click library, which is really a breeze to use, but a bit heavy for smaller scripts. For a web search service prototype, I looked at flask, which was also very interesting, as it is light and simple enough to use that I could get started quickly. It also, surprisingly, fares pretty well in the global TechEmpower benchmarking tests. Those interested in those tools may want to look at the source code, in particular the main command (using an interesting pattern itself, __main__.py) and the search prototype. Debmans is the first project for which I have tried the CII Best Practices Badge program, an interesting questionnaire to review best practices in software engineering. It is an excellent checklist I recommend every project manager and programmer to get familiar with. I still need to complete my work on Debmans: as I write this, I couldn't get access to the new server the DSA team setup for this purpose. It was a bit of a frustrating experience to wait for all the bits to get into place while I had a product ready to test. In the end, the existing manpages.d.o maintainer decided to deploy the existing codebase on the new server while the necessary dependencies are installed and accesses are granted. There's obviously still a bunch of work to be done for this to be running in production so I have postponed all this work to January. My hope is that this tool can be reused by other distributions, but after talking with Ubuntu folks, I am not holding my breath: it seems everyone has something that is "good enough" and that they don't want to break it...

Monkeysign I spent a good chunk of time giving a kick in the Monkeysign project, with the 2.2.2 release, which features contributions from two other developers, which may be a record for a single release. I am especially happy to have adopted a new code of conduct - it has been an interesting process to adapt the code of conduct for such a relatively small project. Monkeysign is becoming a bit of a template on how to do things properly for my Python projects: documentation on readthedocs.org including a code of conduct, support and contribution information, and so on. Even though the code now looks a bit old to me and I am embarrassed to read certain parts, I still think it is a solid project that is useful for a lot of people. I would love to have more time to spend on it.

LWN publishing As you may have noticed if you follow this blog, I have started publishing articles for the LWN magazine, filed here under the lwn tag. It is a way for me to actually get paid for some of my blogging work that used to be done for free. Reports like this one, for example, take up a significant amount of my time and are done without being paid. Converting parts of this work into paid work is part of my recent effort to reduce the amount of time I spend on the computer. An funny note: I always found the layout of the site to be a bit odd, until I looked at my articles posted there in a different web browser, which didn't have my normal ad blocker configuration. It turns out LWN uses ads, and Google ones too, which surprised me. I definitely didn't want to publish my work under banner ads, and will never do so on this blog. But it seems fair that, since I get paid for this work, there is some sort of revenue stream associated with it. If you prefer to see my work without ads, you can wait for it to be published here or become a subscriber which allows you to get rid of the ads on the site. My experience with LWN is great: they're great folks, and very supportive. It's my first experience with a real editor and it really pushed me in improving my writing to make better articles that I normally would here. Thanks to the LWN folks for their support! Expect more of those quality articles in 2017.

Debian packaging I have added a few packages to the Debian archive:

• magic-wormhole: easy file-transfer tool, co-maintained with Jamie Rollins
• slop: screenshot tool
• xininfo: utility used by teiler
• teiler (currently in NEW): GUI for screenshot and screencast tools

I have also updated sopel and atheme-services.

Other work Against my better judgment, I worked again on the borg project. This time I tried to improve the documentation, after a friend asked me for help on "how to make a quick backup". I realized I didn't have any good primer to send regular, non-sysadmin users to and figured that, instead of writing a new one, I could improve the upstream documentation instead. I generated a surprising 18 commits of documentation during that time, mainly to fix display issues and streamline the documentation. My final attempt at refactoring the docs eventually failed, unfortunately, again reminding me of the difficulty I have in collaborating on that project. I am not sure I succeeded in making the project more attractive to non-technical users, but maybe that's okay too: borg is a fairly advanced project and not currently aimed at such a public. This is yet another project I am thinking of creating: a metabackup program like backupninja that would implement the vision created by liw in his A vision for backups in Debian post, which was discarded by the Borg project. Github also tells me that I have opened 19 issues in 14 different repositories in November. I would like to particularly bring your attention to the linkchecker project which seems to be dead upstream and for which I am looking for collaborators in order to create a healthy fork. Finally, I started on reviving the stressant project and changing all my passwords, stay tuned for more!

I got tired of manually fetching installation images via either FTP or by manually transferring files to z/VM to test s390x installs. Hence it was about time to automate it. Originally I wanted to instrument an installation via vmcp from another instance on the same host but I figured that I cannot really rely on a secondary instance when I need it and went the s3270/x3270-script way instead.

The resulting script isn't something I'm particularly proud of, especially as it misses error handling that really should be there. But this is not expect instead you operate on whole screens of data and z/VM is not particularly helpful in telling you that you just completed your logon either. Anyway, it seems to work for me. It downloads the most recent stable or daily image if they are not present yet, uploads them via DFT to CMS and makes sure that the installation does not terminate when the script disconnects. Sadly DFT is pretty slow, so I'm stuck with 70 kB/s and about five minutes of waiting until kernel and initrd are finally uploaded. Given that installations themselves are usually blazingly fast on System z, I'm not too annoyed by that, though.

I previously wrote about a parmfile example that sets enough options to bring debian-installer to the point of a working network console via SSH without further prompting. It's a little unfortunate that s390-netdevice needs to be preseeded with the hardware addresses of the network card in all cases, even if only one is available. I should go and fix that. For now this means that the parmfile will be dependent on the actual VM system definition. With that in mind there is an example script in the same gist that writes out a parmfile and then calls the reinstall script mentioned above. Given that debian-installer now supports HTTPS (so far only in the daily images) you can even do a reasonably secure bootstrapping of the network console credentials and preseeding settings.

If you put this pretty generic preseed configuration file onto a securely accessible webserver and reference it from the parmfile, you can also skip the more tedious questions at the beginning of debian-installer. A secure transport is encouraged as preseed files can do anything to your installation process. Unfortunately it seems that there is no way to preseed SSH keys for the resulting installation yet, neither for the created user nor for root. So I haven't achieved my desired target of a fully automated installation just yet. Debian's Jenkins setup just went with insecure defaults, but given that my sponsored VMs are necessarily connected to the public Internet that seemed like a bad idea to me. I suppose one way out would be to IP/password ACL the preseed file. Another one to somehow get SSH key support into user-setup.

A friend of mine choose for $reasons to install the latest OpenSuSE 42.2 release as his new laptop operating system. It's been a while that I had contact with the SuSE Linux distribution. Must be around 12 years or so. The unsual part here is that I've to support a somewhat eccentric, but mostly ordinary user of computers. And to my surprise it's still hard to just plug in your existing stuff and expect it work. I've done so many dirty things to this installation in the last three days, my system egineering heart is bleeding. printing with a Canon Pixma iP100 printer This is a small portable Canon printer, about four years old. It provides a decent quality and its main strength is that it's small and really portable. Sadly the gutenprint driver just pushes through a blank page. No ink wasted on it at all. So the only reasonable other choice was a four year old binary rpm package provided by Canon. It has a file dependency on "libtiff.so.3" which is no longer available in recent GNU/Linux distributions. So I cheated and

- unpacked the tarball
- installed the rpm from the "packages" folder
zypper install cnijfilter-common-3.70-1.x86_64.rpm cnijfilter-ip100series-3.70-1.x86_64.rpm
... and choose to ignore the missing file dependency on libtiff.so.5.
ln -s /usr/lib64/libtiff.so /usr/lib64/libtiff.so.5
- re-ran the ./install.sh which registered the printer with cups and does whatever else
  magic is included in 1906 lines of shell.

To my surprise this driver still works and provides the expected quality. Though it's just a question of time until this setup will break. Be it an incompatible ABI change in libtiff or another lib in use by those Canon provided tools. QGIS and gdal with ECW support While the printer stuff is a rather common use case, having a map viewer for map files in the ECW format is the eccentric part. I found some hints on stackoverflow and subsequently https://trac.osgeo.org/gdal/wiki/ECW that a non-free library is required and a specific build of gdal. Then QGIS should be able to work with ECW files. Lucky us there is at least a OpenSuSE repository for gdal and QGIS. So I did the following:

zypper addrepo http://download.opensuse.org/repositories/Application:/Geo/openSUSE_Leap_42.2/Application:Geo.repo
zypper install qgis

Then I had to download the non-free ECW SDK from http://download.hexagongeospatial.com/downloads/erdas-ecw-jp2-sdk-v5.3-%28linux%29 - you'll and up with a '.bin' installer file. The installation process left me with "ERDAS-ECW_JPEG_2000_SDK-5.3.0" folder in my $HOME. I moved that one to /opt. Next step is adding the library to the ldconfig search path.

echo "/opt/ERDAS-ECW_JPEG_2000_SDK-5.3.0/Desktop_Read-Only/lib/x64/release/" > /etc/ld.so.conf.d/ecw.conf; ldconfig

Now it was "just" about rebuild gdal with ECW support. So I downloaded the required source packages with "zypper source-install gdal", edited the spec somewhere in "/usr/src/" to make the following modifications

--with-ecw=/opt/ERDAS-ECW_JPEG_2000_SDK-5.3.0/Desktop_Read-Only

added to the "./configure" invocation. And somewhere at the top we had to relax the requirement that all installed files have to be referenced inside the package.

%define _unpackaged_files_terminate_build 0

As a last step I had to "rpmbuild -ba" the package and force the installation via zypper once more, because this time we have a file depedency on the libecw stuff and it's obviously not listed in the rpm database. Last but not least I tried to put the gdal build on hold with

zypper addlock gdal libgdal20

to ensure it's not removed on the next update. Other non-free tools Beside of those two issues I had to install a range of other non-free tools, but currently they work without further issues or modifications. One is Teamviewer (i686 multiarch rpm) and the other one is XnViewMP. XnView is also able to show ECW files, but only the smaller ones. It crashes on bigger ones but that's also the case on Windows. Then there is also (required by some Italian map related websites) the ugly Adobe Flash Plugin for Firefox, but that one is sadly still a widespread issue. We also tried to try out the nvidia graphic drivers but at the moment we could only get the build in Intel card to work. Usually the preferred solution from my point of view but sometimes we see rendering glitches and I'm not sure if it's the driver or something else. my personal take away I hate to admit it but it's nothing extraordinary that was requested here. But still it took me the better part of two evenings to figure everything out. And even now it's not properly integrated and doomed to fail any day due to various updates and changes in the surounding ecosystem. I've full sympathy for every average user that would give up after two hours of research and try&error on this journey. For the printer drivers I'm happy to blame Canon. The printer situation as a whole improved from my point of view during the last decade, but it's still a pain in the ass with the very short shelf life you usually see with consumer models. For the ECW case one could discuss if it would be legally possible and helpful to do ugly dlopen() stuff to dynamcially load the shared libs. But then again someone has to make his hands dirty during the build and discussions about the legal use of header files will be the next chapter (hello Oracle). It's just ugly. Actually I know too little about the world of image formats to judge if someone has a good reason to keep this format commercial or not. From my personal point of view it's not useful and maybe even morally wrong. Technically one could argue if it would make sense to keep a local copy of the gdal build in "/opt" and start QGIS with a modified library path to prefer the private gdal build. Not sure if that is any better. On the other hand there are evolving mechanism like flatpack that would ease the handling of such situations. Buth then again we would be catering non-free software. It feels a lot like giving up. While my private working environment is except for firmware blobs free, I now created for someone a real "FrankenSuSE" to satisfy his everyday needs. On the one hand we now have another mostly satisfied user of a mostly free operating system. On the other hand that was only possible by adding a vast amount of non-free software. For sure we did not win the war, I'm not even sure if we've won a single battle here. It's just frustrating to see what is required to get someone up and running. With my personal attitude towards open source software it even feels wrong to invest so much time into fiddling with non-free components. What is still missing We currently lack an image viewer that allows us to print only a selection of an image, which is useful to print parts of a map. That usually works with XnView on Windows but does not work with the Linux version at the moment. I also tried gwenview and geeqie and had the same issue. Not sure if it's maybe a bug in XnView or one of the Qt parts (gwenview is also Qt based). I did not research that yet. Update: I spent quite some time looking into open bug reports for geeqie and gwenview. Seems the feature to print only a section of an image is something new. I've created #374299 (gwenview) and #457 (geeqie). For XnView I expect it's a difference between XnViewMP (the portable version) and the Windows only XnView Classic. Needs to be clarified and it might be worth to try XnView Classic with wine. Maybe printing with wine via cups works, I found at least some results for it on the internet.

The year 2016, which is about to end, has been full of work and contributions to the FLOSS comunity. Most of my focus goes to two important projects: Debian and Netfilter. This is no coincidence, since my main interests in the IT world are systems and networks. Some numbers (no exhaustive count):

• Netfilter patches/commits: ~60 contributions
• Netfilter docs/wiki: ~20 contributions
• Debian patches/commits: ~200 contributions
• Debian package uploads: ~30 uploads (also some sponsored uploads)
• Debian package maintenance: ~10 packages
• Number of non-technical people migrated to linux: 2!!

I would like to note that most of my work is done in my spare time, and nobody is paying for it (with the exeption of the Suricata debian package). My expectation for 2017 is to continue in this line, learn more and increment the quality of my contributions. I m especially proud of the the non-technical people who migrated to desktop linux due to my help. I m targeting 2 or 3 more friends and relatives for 2017. So, goodbye 2016! Exciting 2017 ahead.

I used to rely on ifupdown to bring up my iptables firewall automatically using a config like this in /etc/network/interfaces:

allow-hotplug eth0
iface eth0 inet dhcp
    pre-up /sbin/iptables-restore /etc/network/iptables.up.rules
    pre-up /sbin/ip6tables-restore /etc/network/ip6tables.up.rules
allow-hotplug wlan0
iface wlan0 inet dhcp
    pre-up /sbin/iptables-restore /etc/network/iptables.up.rules
    pre-up /sbin/ip6tables-restore /etc/network/ip6tables.up.rules

but that doesn't seem to work very well in the brave new NetworkManager world. What does work reliably is a "pre-up" NetworkManager script, something that gets run before a network interface is brought up. However, despite what the documentation says, a dispatcher script in /etc/NetworkManager/dispatched.d/ won't work on my Debian and Ubuntu machines. Instead, I had to create a new iptables script in /etc/NetworkManager/dispatcher.d/pre-up.d/:

#!/bin/sh
LOGFILE=/var/log/iptables.log
if [ "$1" = lo ]; then
    echo "$0: ignoring $1 for \ $2'" >> $LOGFILE
    exit 0
fi
case "$2" in
    pre-up)
        echo "$0: restoring iptables rules for $1" >> $LOGFILE
        /sbin/iptables-restore /etc/network/iptables.up.rules >> $LOGFILE 2>&1
        /sbin/ip6tables-restore /etc/network/ip6tables.up.rules >> $LOGFILE 2>&1
        ;;
    *)
        echo "$0: nothing to do with $1 for \ $2'" >> $LOGFILE
        ;;
esac
exit 0

and then make that script executable:

chmod a+x /etc/NetworkManager/dispatcher.d/pre-up.d/iptables

With this in place, I can put my iptables rules in the usual place (/etc/network/iptables.up.rules and /etc/network/ip6tables.up.rules) and use the handy iptables-apply and ip6tables-apply commands to test any changes to my firewall rules.

I recently needed to get access to a DataEase database; the person I helped was the legitimate owner of the data, but had forgotten the password, as the database was largely from 1996. There are various companies around the world that seem to do this, or something similar (like give you an API), for a usually unspecified fee; they all have very 90s homepages and in general seem like they have gone out of business a long time ago. And I wasn't prepared to wait. For those of you who don't know DataEase, it's a sort-of relational database for DOS that had its heyday in the late 80s and early 90s (being sort of the cheap cousin of dBase); this is before SQL gained traction as the standard query language, before real multiuser database access, and before variable-width field storage. It is also before reasonable encryption. Let's see what we can do. DataEase has a system where tables are mapped through the data dictionary, which is a table on its own. (Sidenote: MySQL pre-8.0 still does not have this.) This is the file RDRRTAAA.DBM; I don't really know what RDRR stands for, but T is the database letter in case you wanted more than one database in the same directory, and AAA, AAB, AAC etc. is a counter so that a table grows to be too big for one file. (There's also .DBA files for structure of non-system tables, and then some extra stuff for indexes.) DBM files are pretty much the classical, fixed-length 80s-style database files; each row has some flags (I believe these are for e.g. row is deleted ) and then just the rows in fixed format right after each other. For instance, here's one I created as part of testing (just the first few lines of the hexdump are shown):

00000000: 0e 00 01 74 65 73 74 62 61 73 65 00 00 00 00 00  ...testbase.....
00000010: 00 00 00 00 00 00 00 73 46 cc 29 37 00 09 00 00  .......sF.)7....
00000020: 00 00 00 00 00 43 3a 52 44 52 52 54 41 41 41 2e  .....C:RDRRTAAA.
00000030: 44 42 4d 00 00 01 00 0e 00 52 45 50 4f 52 54 20  DBM......REPORT 
00000040: 44 49 52 45 43 54 4f 52 59 00 00 00 00 00 1c bd  DIRECTORY.......
00000050: d4 1a 27 00 00 00 00 00 00 00 00 00 43 3a 52 45  ..'.........C:RE
00000060: 50 4f 54 41 41 41 2e 44 42 4d 00 00 01 00 0e 00  POTAAA.DBM......
00000070: 52 65 6c 61 74 69 6f 6e 73 68 69 70 73 00 00 00  Relationships...

Even without going in-depth, we can see the structure here; there's testbase which maps to C:RDRRTAA.DBM (the RDRR itself), there's a table called REPORT DIRECTORY that maps to C:REPOTAAA.DBM, and then more stuff after that, and so on. However, other tables are not so easily read, because you can ask DataEase to encrypt a table. Let's look at such an encrypted table, like the Users table (containing usernames, passwords not password hashes and some extra information like access level), which is always encrypted:

00000000: 0c 01 9f ed 94 f7 ed 34 ba 88 9f 78 21 92 7b 34  .......4...x!. 4
00000010: ba 88 0f d9 94 05 1e 34 ba 88 a0 78 21 92 7b 34  .......4...x!. 4
00000020: e2 88 9f 78 21 92 7b 34 ba 88 9f 78 21 92 7b 34  ...x!. 4...x!. 4
00000030: ba 88 9f 78 21 92 7b 34 ba 88 9f 78 21 92 7b     ...x!. 4...x!. 

Clearly, this isn't very good encryption; it uses a very short, repetitive key of eight bytes (64 bits). (The data is mostly zero padding, which makes it much easier to spot this.) In fact, in actual data tables, only five of these bytes are set to a non-zero value, which means we have a 40-bit key; export controls? My first assumption here was of course XOR, but through some experimentation, it turned out what you need is actually 8-bit subtraction (with wraparound). The key used is derived from both a database key and a per-table key, both stored in the RDRR; again, if you disassemble, I'm sure you can find the key derivation function, but that's annoying, too. Note, by the way, that this precludes making an attack by just copying tables between databases, since the database key is different. So let's do a plaintext attack. If you assume the plaintext of the bottom row is all padding, that's your key and here's what you end up with:

00000000: 52 79 00 75 73 65 72 00 00 00 00 00 00 00 00 00  Ry.user.........
00000010: 00 00 70 61 73 73 a3 00 00 00 01 00 00 00 00 00  ..pass..........
00000020: 28 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  (...............
00000030: 00 00 00 00 00 00 00 00                          ........ 

Not bad, eh? Actually the first byte of the key here is wrong as far as I know, but it didn't interfere with the fields, so we have what we need to log in. (At that point, we've won, because DataEase will helpfully decrypt everything transparent for us.) However, there's a twist; if the password is longer than four characters, the entire decryption of the Users table changes. Of course, we could run our plaintext attack against every data table and pick out the information by decoding the structure, but again; annoying. So let's see what it looks like if we choose passs instead:

00000000: 0e 01 9f 7a ae 9e 21 f5 08 63 07 6d a3 a1 17 5d  ...z..!..c.m...]
00000010: 70 cb df 36 7e 7c 91 c5 d8 33 d8 3d 73 71 e7 2d  p..6~ ...3.=sq.-
00000020: 7b 9b 3f a5 db d9 4f 95 a8 03 a7 0d 43 41 b7 fd   .?...O.....CA..
00000030: 10 6b 0f 75 ab a9 1f 65 78 d3 77 dd 13 11 87     .k.u...ex.w....

Distinctly more confusing. At this point, of course, we know at which byte positions the username and password start, so if we wanted to, we could just try setting the start byte of the password to every possible byte in turn until we hit 0x00 (DataEase truncates fields at the first zero byte), which would allow us to get in with an empty password. However, I didn't know the username either, and trying two bytes would mean 65536 tries, and I wasn't up for automating macros through DOSBox. So an active attack wasn't too tempting. However, we can look at the last hex byte (where we know the plaintext is 0); it goes 0x5d, 0x2d, 0xfd... and some other bytes go 0x08, 0xd8, 0xa8, 0x78, and so on. So clearly there's an obfuscation here where we have a per-line offset that decreases with 0x30 per line. (Actually, the increase/decrease per line seems to be derived from the key somehow, too.) If we remove that, we end up with:

00000000: 0e 01 9f 7a ae 9e 21 f5 08 63 07 6d a3 a1 17 5d  ...z..!..c.m...]
00000010: a0 fb 0f 66 ae ac c1 f5 08 63 08 6d a3 a1 17 5d  ...f.....c.m...]
00000020: db fb 9f 05 3b 39 af f5 08 63 07 6d a3 a1 17 5d  ....;9...c.m...]
00000030: a0 fb 9f 05 3b 39 af f5 08 63 07 6d a3 a1 17     ....;9...c.m...

Well, OK, this wasn't much more complicated; our fixed key is now 16 bytes long instead of 8 bytes long, but apart from that, we can do exactly the same plaintext attack. (Also, it seems to change per-record now, but we don't see it here, since we've only added one user.) Again, assume the last line is supposed to be all 0x00 and thus use that as a key (plus the last byte from the previous line), and we get:

00000000: 6e 06 00 75 73 65 72 00 00 00 00 00 00 00 00 00  n..user.........
00000010: 00 00 70 61 73 73 12 00 00 00 01 00 00 00 00 00  ..pass..........
00000020: 3b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ;...............
00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00     ...............

Well, OK, it wasn't perfect; we got pass\x12 instead of passs , so we messed up somehow. I don't know exactly why the fifth character gets messed up like this; actually, it cost me half an hour of trying because the password looked very real but the database wouldn't let me in, but eventually, we just guessed at what the missing letter was supposed to be. So there you have it; practical small-scale cryptanalysis of DOS-era homegrown encryption. Nothing advanced, but the user was happy about getting the data back after a few hours of work. :-)