• Holger Levsen wrote and published details about our upcoming Berlin summit. Expect a more detailed announced soon and consider planning your travel!

• Chris Lamb:
  • #875700 filed against gtk+3.0.
  • #875704 filed against gdk-pixbuf.
  • #875792 filed against doit.
• Vagrant Cascadian:
  • #875711 filed against qemu.

• Holger Levsen:
  • fonts-dustin/20030517-11 uploaded, fixing #815723 with patch by Scarlett Clark.

• Bernhard M. Wiedemann:
  • cadabra2 build timestamp
  • jimtcl build timestamp, fixed another way
  • itpp build timestamp, merged
  • dunst file list ordering, merged
  • HSAIL-Tools hash table ordering, merged
  • kubernetes hash set ordering, no patch

• Add build_path_captured_in_assembly_objects

• Juliana Oliveira Rodrigues:
  • Fix comparisons between different container types not comparing inside files. It was caused by falling back to binary comparison for different file types even for unextracted containers.
  • Add many tests for the fixed behaviour.
  • Other code quality improvements.
• Chris Lamb:
  • Various code quality and style improvements, some of it using Flake8.
• Mattia Rizzolo:
  • Add a check to prevent installation with python < 3.4

• Ximin Luo:
  • Split up the very large __init__.py and remove obsolete earlier code.
  • Extend the syntax for the --variations flag to support parameters to certain variations like user_group, and document examples in README.
  • Add a --vary flag for the new syntax and deprecate --dont-vary.
  • Heavily refactor internals to support > 2 builds.
  • Support >2 builds using a new --extra-build flag.
  • Properly sanitize artifact_pattern to avoid arbitrary shell execution.

• Chris Lamb:
  • Packaging maintenance updates.
  • Developer documentation updates.

• Holger Levsen:
  • Add a page for the Reproducible Builds World Summit 3 in Berlin 2017.
• Chris Lamb:
  • Moved isdebianreproducibleyet.com to HTTPS.
  • Updated the SSL certificate for buildinfo.debian.net.

• Vagrant Cascadian and Holger Levsen:
  • Added two armhf boards to the build farm. #874682
• Holger also:
  • use timeout to limit the diffing of the two build logs to 30min, which greatly reduced jenkins load again.

Many voices arise now and then against risks linked to the Github use by Free Software projects. Yet the infatuation for the collaborative forge of the Octocat Californian start-ups doesn t seem to fade away.

These recent years, Github and its services take an important role in software engineering as they are seen as easy to use, efficient for a daily workload with interesting functions in enterprise collaborative workflow or amid a Free Software project. What are the arguments against using its services and are they valid? We will list them first, then we ll examine their validity.

1. Critical points

1.1 Centralization

The Github application belongs to a single entity, Github Inc, a US company which manage it alone. So, a unique company under US legislation manages the access to most of Free Software application code sources, which may be a problem with groups using it when a code source is no longer available, for political or technical reason.

The Octocat, the Github mascot

This centralization leads to another trouble: as it obtained critical mass, it becomes more and more difficult not having a Github account. People who don t use Github, by choice or not, are becoming a silent minority. It is now fashionable to use Github, and not doing so is seen as out of date . The same phenomenon is a classic, and even the norm, for proprietary social networks (Facebook, Twitter, Instagram).

1.2 A Proprietary Software

When you interact with Github, you are using a proprietary software, with no access to its source code and which may not work the way you think it is. It is a problem at different levels. First, ideologically, but foremost in practice. In the Github case, we send them code we can control outside of their interface. We also send them personal information (profile, Github interactions). And mostly, Github forces any project which goes through the US platform to use a crucial proprietary tools: its bug tracking system.

Windows, the epitome of proprietary software, even if others took the same path

1.3 The Uniformization

Working with Github interface seems easy and intuitive to most. Lots of companies now use it as a source repository, and many developers leaving a company find the same Github working environment in the next one. This pervasive presence of Github in free software development environment is a part of the uniformization of said developers working space.

Uniforms always bring Army in my mind, here the Clone army

2 Critical points cross-examination

2.1 Regarding the centralization

2.1.1 Service availability rate

As said above, nowadays, Github is the main repository of Free Software source code. As such it is a favorite target for cyberattacks. DDOS hit it in March and August 2015. On December 15, 2015, an outage led to the inaccessibility of 5% of the repositories. The same occurred on November 15. And these are only the incident reported by Github itself. One can imagine that the mean outage rate of the platform is underestimated.

2.1.2 Chain reaction could block Free Software development

Today many dependency maintenance tools, as npm for javascript, Bundler for Ruby or even pip for Python can access an application source code directly from Github. Free Software projects getting more and more linked and codependents, if one component is down, all the developing process stop.

One of the best examples is the npmgate. Any company could legally demand that Github take down some source code from its repository, which could create a chain reaction and blocking the development of many Free Software projects, as suffered the Node.js community from the decisions of Npm, Inc, the company managing npm.

2.2 A historical precedent: SourceForge

Github didn t appear out of the blue. In his time, its predecessor, SourceForge, was also extremely popular.

Heavily centralized, based on strong interaction with the community, SourceForge is now seen as an aging SAAS (Software As A Service) and sees most of its customers fleeing to Github. Which creates lots of hurdles for those who stayed. The Gimp project suffered from spams and terrible advertising, which led to the departure of the VLC project, then from installers corrupted with adwares instead of the official Gimp installer for Windows. And finally, the Project Gimp s SourceForge account was hacked by SourceForge team itself!

These are very recent examples of what can do a commercial entity when it is under its stakeholders pressure. It is vital to really understand what it means to trust them with data and exchange centralization, where it could have tremendous repercussion on the day-to-day life and the habits of the Free Software and open source community.

2.3. Regarding proprietary software

2.3.1 One community, several opinions on proprietary software

Mostly based on ideology, this point deals with the definition every member of the community gives to Free Software and open source. Mostly about one thing: is it viral or not? Or GPL vs MIT/BSD.

Those on the side of the viral Free Software will have trouble to use a proprietary software as this last one shouldn t even exist. It must be assimilated, to quote Star Trek, as it is a connected black box, endangering privacy, corrupting for profit our uses and restrain our freedom to use as we re pleased what we own, etc.

Those on the side of complete freedom have no qualms using proprietary software as their very existence is a consequence of freedom without restriction. They even agree that code they developed may be a part of proprietary software, which is quite a common occurrence. This part of the Free Software community has no qualm using Github, which is well within their ideology parameters. Just take a look at the Janson amphitheater during Fosdem and check how many Apple laptops running on macOS are around.

FreeBSD, the main BSD project under the BSD license

2.3.2 Data loss and data restrictions linked to proprietary software use

Even without ideological consideration, and just focusing on Github infrastructure, the bug tracking system is a major problem by itself.

Bug report builds the memory of Free Software projects. It is the entrance point for new contributors, the place to find bug reporting, requests for new functions, etc. The project history can t be limited only to the code. It s very common to find bug reports when you copy and paste an error message in a search engine. Not their historical importance is precious for the project itself, but also for its present and future users.

Github gives the ability to extract bug reports through its API. What would happen if Github is down or if the platform doesn t support this feature anymore? In my opinion, not that many projects ever thought of this outcome. How could they move all the data generated by Github into a new bug tracking system? One old example now is Astrid, a TODO list bought by Yahoo a few years ago. Very popular, it grew fast until it was closed overnight, with only a few weeks for its users to extract their data. It was only a to-do list. The same situation with Github would be tremendously difficult to manage for several projects if they even have the ability to deal with it. Code would still be available and could still live somewhere else, but the project memory would be lost. A project like Debian has today more than 800,000 bug reports, which are a data treasure trove about problems solved, function requests and where the development stand on each. The developers of the Cpython project have anticipated the problem and decided not to use Github bug tracking systems.

Issues, the Github proprietary bug tracking system

Another thing we could lose if Github suddenly disappear: all the work currently done regarding the push requests (aka PRs). This Github function gives the ability to clone one project s Github repository, to modify it to fit your needs, then to offer your own modification to the original repository. The original repository s owner will then review said modification, and if he or she agrees with them will fuse them into the original repository. As such, it s one of the main advantages of Github, since it can be done easily through its graphic interface.

However reviewing all the PRs may be quite long, and most of the successful projects have several ongoing PRs. And this PRs and/or the proprietary bug tracking system are commonly used as a platform for comment and discussion between developers.

Code itself is not lost if Github is down (except one specific situation as seen below), but the peer review works materialized in the PRs and the bug tracking system is lost. Let s remember than the PR mechanism let you clone and modify projects and then generate PRs directly from its proprietary web interface without downloading a single code line on your computer. In this particular case, if Github is down, all the code and the work in progress is lost. Some also use Github as a bookmark place. They follow their favorite projects activity through the Watch function. This technological watch style of data collection would also be lost if Github is down.

Debian, one of the main Free Software projects with at least a thousand official contributors

2.4 Uniformization

The Free Software community is walking a thigh rope between normalization needed for an easier interoperability between its products and an attraction for novelty led by a strong need for differentiation from what is already there.

Github popularized the use of Git, a great tool now used through various sectors far away from its original programming field. Step by step, Git is now so prominent it s almost impossible to even think to another source control manager, even if awesome alternate solutions, unfortunately not as popular, exist as Mercurial.

A new Free Software project is now a Git repository on Github with README.md added as a quick description. All the other solutions are ostracized? How? None or very few potential contributors would notice said projects. It seems very difficult now to encourage potential contributors into learning a new source control manager AND a new forge for every project they want to contribute. Which was a basic requirement a few years ago. It s quite sad because Github, offering an original experience to its users, cut them out of a whole possibility realm. Maybe Github is one of the best web versioning control systems. But being the main one doesn t let room for a new competitor to grow. And it let Github initiate development newcomers into a narrow function set, totally unrelated to the strength of the Git tool itself.

3. Centralization, uniformization, proprietary software What s next? Laziness?

Fight against centralization is a main part of the Free Software ideology as centralization strengthens the power of those who manage it and who through it control those who are managed by it. Uniformization allergies born against main software companies and their wishes to impose a closed commercial software world was for a long time the main fuel for innovation thirst and intelligent alternative development. As we said above, part of the Free Software community was built as a reaction to proprietary software and their threat. The other part, without hoping for their disappearance, still chose a development model opposite to proprietary software, at least in the beginning, as now there s more and more bridges between the two.

The Github effect is a morbid one because of its consequences: at least centralization, uniformization, proprietary software usage as their bug tracking system. But some years ago the Dear Github buzz showed one more side effect, one I ve never thought about: laziness. For those who don t know what it is about, this letter is a complaint from several spokespersons from several Free Software projects which demand to Github team to finally implement, after years of polite asking, new functions. Since when Free Software project facing a roadblock request for clemency and don t build themselves the path they need? When Torvalds was involved in the Bitkeeper problem and the Linux kernel development team couldn t use anymore their revision control software, he developed Git. The mere fact of not being able to use one tool or functions lacking is the main motivation to seek alternative solutions and, as such, of the Free Software movement. Every Free Software community member able to code should have this reflex. You don t like what Github offers? Switch to Gitlab. You don t like it Gitlab? Improve it or make your own solution.

The Gitlab logo

Let s be crystal clear. I ve never said that every Free Software developers blocked should code his or her own alternative. We all have our own priorities, and some of us even like their beauty sleep, including me. But, to see that this open letter to Github has 1340 names attached to it, among them some spokespersons for major Free Software project showed me that need, willpower and strength to code a replacement are here. Maybe said replacement will be born from this letter, it would be the best outcome of this buzz.

In the end, Github usage is just another example of Internet usage massification. As Internet users are bound to go to massively centralized social network as Facebook or Twitter, developers are following the same path with Github. Even if a large fraction of developers realize the threat linked this centralized and proprietary organization, the whole community is following this centralization and uniformization trend. Github service is useful, free or with a reasonable price (depending on the functions you need) easy to use and up most of the time. Why would we try something else? Maybe because others are using us while we are savoring the convenience? The Free Software community seems to be quite sleepy to me.

The lion enjoying the hearth warm

About Me Carl Chenet, Free Software Indie Hacker, founder of the French-speaking Hacker News-like Journal du hacker. Follow me on social networks

• My Mastodon account : @carlchenet
• My Diaspora* account : @carlchenet
• My Twitter account : @carl_chenet

Translated from French by St phanie Chaptal. Original article written in 2015.

• Antoine Beaupr did 16h.
• Ben Hutchings did 10 hours (out of 15h allocated + 1 extra hour, thus keeping 6 extra hour for September).
• Chris Lamb did 18 hours.
• Emilio Pozuelo Monfort did 20.5 hours (out of 20.25 hours allocated + 13 hours remaining, thus keeping 12.75 hours for September).
• Guido G nther did 10 hours.
• Hugo Lefeuvre did 14h (out of 2h allocated + 12 extra hours).
• Lucas Kanashiro did 20.25 hours.
• Markus Koschany did 20.25 hours.
• Ola Lundqvist did 9h (out of 14h allocated + 16 extra hours, he gave back 14 hours, thus keeping 7 extra hours for September).
• Rapha l Hertzog did 12 hours.
• Roberto C. Sanchez did 27.25 hours (out of 20.25 hours allocated + 16 hours remaining, thus keeping 9 extra hours for September).
• Thorsten Alteholz did 20.25 hours.

• Platinum sponsors:
• TOSHIBA (for 22 months)
• GitHub (for 13 months)
• Gold sponsors:
• The Positive Internet (for 38 months)
• Blablacar (for 37 months)
• Linode (for 27 months)
• Babiel GmbH (for 16 months)
• Plat Home (for 16 months)
• Silver sponsors:
• Domeneshop AS (for 37 months)
• Universit Lille 3 (for 37 months)
• Trollweb Solutions (for 35 months)
• Nantes M tropole (for 32 months)
• Dalenys (for 28 months)
• Univention GmbH (for 23 months)
• Universit Jean Monnet de St Etienne (for 23 months)
• Sonus Networks (for 17 months)
• UR Communications BV (for 12 months)
• maxcluster GmbH (for 11 months)
• Exonet B.V. (for 7 months)
• Leibniz Rechenzentrum
• Bronze sponsors:
• David Ayers IntarS Austria (for 38 months)
• Evolix (for 38 months)
• Offensive Security (for 38 months)
• Seznam.cz, a.s. (for 38 months)
• Freeside Internet Service (for 37 months)
• MyTux (for 37 months)
• Intevation GmbH (for 35 months)
• Linuxhotel GmbH (for 35 months)
• Daevel SARL (for 33 months)
• Bitfolk LTD (for 32 months)
• Megaspace Internet Services GmbH (for 32 months)
• NUMLOG (for 32 months)
• Greenbone Networks GmbH (for 31 months)
• WinGo AG (for 31 months)
• Ecole Centrale de Nantes LHEEA (for 27 months)
• Sig-I/O (for 24 months)
• Entr ouvert (for 22 months)
• Adfinis SyGroup AG (for 19 months)
• GNI MEDIA (for 14 months)
• Quarantainenet BV (for 14 months)
• RHX Srl (for 11 months)
• Bearstech (for 5 months)
• LiHAS (for 5 months)
• People Doc
• Catalyst IT Ltd

No comment Liked this article? Click here. My blog is Flattr-enabled.

• There were a lot of GCC-7 bugs to fix this month which claimed most of my games related time.
• Bug fixes: torcs (RC #853685), berusky2 (RC #853325), lordsawar (RC #853529), simutrans(#869029), gngb (RC #853425), libclaw (RC #853488), funguloids (RC #853408), plee-the-bear (RC #853618), alien-arena (RC #871218), fretsonfire (RC #872934), minetest, (RC #873324) widelands, (RC #871114), pingus (RC #853614)
• I released version 2.1 of the Debian Games Blend.
• I completely overhauled the gngb package, a color gameboy emulator.
• I packaged new upstream versions of freeciv, peg-e and blockattack.
• I backported the memory leak fix for unknown-horizons and fife to Stretch (#871037).
• I investigated some graphical glitches in Neverball which appear to be related to OpenGL and the graphic stack in general but I couldn t find an immediate solution. (#871223)

• I sponsored libimglib2-java for Ghislain Vaillant.
• New upstream releases: apktool, jboss-modules, jboss-logging-tools, jboss-logmanager.
• For jboss-xnio I packaged two new build-dependencies which are wildfly-common and wildfly-client-config and they are currently waiting in the NEW queue.
• The last build-dependency for PDFsam was accepted this month and I was able to upload the new version to experimental. Unfortunately the program is currently not really usable due to a bug in libhibernate-validator-java (#874579)

• From 31. July until 06. August I was in charge of our LTS frontdesk. I triaged bugs in tinyproxy, mantis, sox, timidity, ioquake3, varnish, libao, clamav, binutils, smplayer, libid3tag, mpg123 and shadow.
• DLA-1064-1. Issued a security update for freeradius fixing 6 CVE.
• DLA-1068-1. Issued a security update for git fixing 1 CVE.
• DLA-1077-1. Issued a security update for faad2 fixing 11 CVE.
• DLA-1083-1. Issued a security update for openexr fixing 3 CVE.
• DLA-1095-1. Issued a security update for freerdp fixing 5 CVE.

• I uploaded a security fix for openexr (#864078) to fix CVE-2017-9110, CVE-2017-9112 and CVE-2017-9116.

• packages depends on lsb-core, no longer present in Buster testing.
• packages depends on libpango1.0-0, however testing contains libpango-1.0-0

• 1) create a temporal working directory

% user@debian:~ $ mkdir pkg
% user@debian:~ $ cd pkg/

• 2) get the original .deb package, the Google Hangout talk plugin.

% user@debian:~/pkg $ wget https://dl.google.com/linux/direct/google-talkplugin_current_amd64.deb
[...]

• 3) extract the original .deb package

% user@debian:~/pkg $ dpkg-deb -R google-talkplugin_current_amd64.deb google-talkplugin_current_amd64/

• 4) edit the control file, replace libpango1.0-0 with libpango-1.0-0

% user@debian:~/pkg $ nano google-talkplugin_current_amd64/DEBIAN/control

• 5) rebuild the package and install it!

% user@debian:~/pkg $ dpkg -b google-talkplugin_current_amd64
% user@debian:~/pkg $ sudo dpkg -i google-talkpluging_current_amd64.deb

Requirements:

• Synology NAS running DSM >= 6.0
• Sub/domain name with ability to update DNS records
• SSH access to your Synology NAS

DSM: Control Panel > Security > Certificates > Add

Add a new certificate > Get a certificate from Let's Encrypt

Name: SynoCommunity
Location: http://packages.synocommunity.com

DSM: Control Panel > Application Portal > Reverse Proxy > Create

/usr/syno/etc/certificate/_archive/

drwx------ 2 root root 4096 Sep 10 12:57 JeRh3Y

johndoe/freenode:password

• XMPP chatroom
• email: tails-dev@boum.org (public) or tails@boum.org (private)

1. Install the build dependencies:

   sudo apt install \
       git \
       rake \
       libvirt-daemon-system \
       dnsmasq-base \
       ebtables \
       qemu-system-x86 \
       qemu-utils \
       vagrant \
       vagrant-libvirt \
       vmdebootstrap && \
   sudo systemctl restart libvirtd
   

2. Ensure your user is in the relevant groups:

   for group in kvm libvirt libvirt-qemu ; do
      sudo adduser "$(whoami)" "$group"
   done
   

3. Logout and log back in to apply the new group memberships.

git clone https://git-tails.immerda.ch/tails && \
cd tails && \
git checkout 3.2-alpha2 && \
git submodule update --init && \
rake build

sudo apt install apt-show-versions && \
(
  for f in /etc/issue /proc/cpuinfo
  do
    echo "--- File: $ f  ---"
    cat "$ f "
    echo
  done
  for c in free locale env 'uname -a' '/usr/sbin/libvirtd --version' \
            'qemu-system-x86_64 --version' 'vagrant --version'
  do
    echo "--- Command: $ c  ---"
    eval "$ c "
    echo
  done
  echo '--- APT package versions ---'
  apt-show-versions qemu:amd64 linux-image-amd64:amd64 vagrant \
                    libvirt0:amd64
)   bzip2 > system-info.txt.bz2

bzless system-info.txt.bz2

• set Category to Build system;
• paste the output of rake build;
• attach system-info.txt.bz2 (this will publish that file).

sha512sum tails-amd64-3.2~alpha2.iso

9b4e9e7ee7b2ab6a3fb959d4e4a2db346ae322f9db5409be4d5460156fa1101c23d834a1886c0ce6bef2ed6fe378a7e76f03394c7f651cc4c9a44ba608dda0bc

1. Install diffoscope version 83 or higher and all the packages it recommends. For example, if you're using Debian Stretch:

   sudo apt remove diffoscope && \
   echo 'deb http://ftp.debian.org/debian stretch-backports main' \
       sudo tee /etc/apt/sources.list.d/stretch-backports.list && \
   sudo apt update && \
   sudo apt -o APT::Install-Recommends="true" \
            install diffoscope/stretch-backports
   

2. Download the official Tails 3.2~alpha2 ISO image.
3. Compare the official Tails 3.2~alpha2 ISO image with yours:

   diffoscope \
          --text diffoscope.txt \
          --html diffoscope.html \
          --max-report-size 262144000 \
          --max-diff-block-lines 10000 \
          --max-diff-input-lines 10000000 \
          path/to/official/tails-amd64-3.2~alpha2.iso \
          path/to/your/own/tails-amd64-3.2~alpha2.iso
   bzip2 diffoscope. txt,html 
   

4. Send an email to tails-dev@boum.org (public) or tails@boum.org (private) with the subject "Reproduction of Tails 3.2~alpha2 failed", attaching:
   • system-info.txt.bz2;
   • the smallest file among diffoscope.txt.bz2 and diffoscope.html.bz2, except if they are larger than 100 KiB, in which case better upload the file somewhere (e.g. share.riseup.net and share the link in your email.

• The main container is using the (MS-CFB), which is notably used by MS Office 97-2003 .doc files. It contains several streams:
  • Metadata stream: in OLE Property Set format (MS-OLEPS), contains 2 blobs in a specific Type-Length-Value (TLV) format:
    • _ctlfile: global archive properties and access list
      It is obfuscated by means of static-key AES encryption.
      The properties include archive initial filename and a global IV.
      A global encryption key is itself encrypted in each user entry.
    • _catalog: file list
      Contains each file metadata indexed with a 15-bytes identifier.
      Directories are supported.
      Full filename is encrypted using AES.
      File extension is (redundantly) stored in clear, and so are file metadata such as modification time.
  • Each file in the archive compressed with zlib and encrypted with the standard AES algorithm, in a separate stream.
    Several encryption schemes and key sizes are supported.
    The file stream is split in chunks of 512 bytes, individually encrypted.
  • Optional streams, contain additional metadata as well as pictures to display in the application background ("watermarks"). They are not discussed here.

+----------------------------------------------------------------------------------------------------+
  .zed archive (MS-CBF)                                                                               
                                                                                                      
   stream #1                         stream #2                       stream #3...                     
  +------------------------------+  +---------------------------+  +---------------------------+      
    metadata (MS-OLEPS)               encryption (AES)               encryption (AES)                 
                                      512-bytes chunks               512-bytes chunks                 
    +--------------------------+                                                                      
      obfuscation (static key)        +-----------------------+      +-----------------------+        
      +----------------------+       -  compression (zlib)     -    -  compression (zlib)     -       
       _ctlfile (TLV)                                                                            ...  
      +----------------------+          +---------------+              +---------------+               
    +--------------------------+          file contents                  file contents                
                                                                                                      
    +--------------------------+     -  +---------------+      -    -  +---------------+      -       
      _catalog (TLV)                                                                                  
    +--------------------------+      +-----------------------+      +-----------------------+        
  +------------------------------+  +---------------------------+  +---------------------------+      
+----------------------------------------------------------------------------------------------------+

• obfuscated metadata encrypted with static AES key
• filenames and files in archives with 'encryption_mode' set to "AES-CBC-STREAM"
• any AES ciphertext of size < 16 bytes, regardless of encryption mode

• the second-to-last block of the ciphertext is encrypted in AES-ECB mode (i.e. block cipher encryption only, without XORing with the IV)
• then XOR-ed with the last partial block (hence truncated to the length of the partial block)

• filenames and files in archives with 'encryption_mode' set to "AES-CBC-CTS".
  • exception: if the size of the ciphertext is < 16 bytes, then "STREAM" is used instead.

• static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by 0100 (hexadecimal) (18 bytes total)
• 16-byte IV
• ciphertext
• 1 uint32be representing the length of all the above
• static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by "ZoneCentral (R)" (ASCII) and a NUL byte (32 bytes total)

• global archive properties as a 'fileprops' structure,
• extra archive properties as a 'archive_extraprops' structure
• users access list as a series of 'passworduser' and 'rsauser entries.

• 4 bytes for Type (specified as a 4-bytes hexadecimal below)
• 4 bytes for value Length (uint32be)
• Value

• 80110600: fileprops, used for the file list as well as for the global archive properties
• 001b0600: archive_extraprops
• 80140600: accesslist

• 80110600: fileprops (TLV structure): global archive properties
  • 00230400: archive_pathname (UTF-16LE string): initial archive filename (past versions also leaked the full pathname of the initial archive)
  • 80270200: encryption_mode (utf32be): 103 for "AES-CBC-STREAM", 104 for "AES-CBC-CTS"
  • 80260200: encryption_strength (utf32be): AES key size, in bytes (e.g. 32 means AES with a 256-bit key)
  • 80280500: files_iv (sequence of bytes): global IV for all filenames and file contents
• 001b0600: archive_extraprops (TLV structure): additionnal archive properties (optional)
  • 00c40500: archive_creationtime (FILETIME): date and time when archive was initially created (optional)
  • 00c00400: archive_createdwith (UTF-16LE string): uuid-like structure describing the application that initialized the archive (optional)
    00000188-1000-3CA8-8868-36F59DEFD14D is Zed! Free 1.0.188.
• 80140600: accesslist (TLV structure): describe the users, their key encryption and their permissions
  • 80610600: passworduser (TLV structure): user identified by password (0 or more)
  • 80620600: rsauser (TLV structure): user identified by RSA key (via file or PKCS#11 token) (0 or more)
  • Fields common to passworduser and rsauser:
    • 80710400: login (UTF-16LE string): user name
    • 80720300: login_md5 (sequence of bytes): used by the application to search for a user name
    • 807e0100: priv1 (uchar): user privileges; present and set to 1 when user is admin (optional)
    • 00830200: priv2 (uint32be): user privileges; present and set to 2 when user is admin, present and set to 5 when user is a marked as mandatory, e.g. for recovery keys (optional)
    • 80740500: files_key_ciphertext (sequence of bytes): the archive encryption key, itself encrypted
    • 00840500: user_creationtime (FILETIME): date and time when the user was added to the archive
  • passworduser-specific fields:
    • 80760500: pbe_salt (sequence of bytes): salt for PBE
    • 80770200: pbe_iter (uint32be): number of iterations for PBE
    • 80780200: pkcs12_hashfunc (uint32be): hash function used for PBE and PBA key derivation
    • 80790500: pba_checksum (sequence of bytes): password derived with PBA to check for password validity
    • 807a0500: pba_salt (sequence of bytes): salt for PBA
    • 807b0200: pba_iter (uint32be): number of iterations for PBA
  • rsauser-specific fields:
    • 807d0500: certificate (sequence of bytes): user X509 certificate in DER format

• 80110600: fileprops (TLV structure): describe the archive files (0 or more)
  • 80300500: file_id (sequence of bytes): a 16-byte unique identifier
  • 80310400: filename_halfanon (UTF-16LE string): half-anonymized filename, e.g. File1.txt (leaking filename extension)
  • 00380500: filename_ciphertext (sequence of bytes): encrypted filename; may have a trailing NUL byte once decrypted
  • 80330500: file_size (uint64le): decompressed file size in bytes
  • 80340500: file_creationtime (FILETIME): file creation date and time
  • 80350500: file_lastwritetime (FILETIME): file last modification date and time
  • 80360500: file_lastaccesstime (FILETIME): file last access date and time
  • 00370500: parent_directory_id (sequence of bytes): file_id of the parent directory, 0 is top-level
  • 80320100: is_dir (uint32be): 1 if entry is directory (optional)

• 21: SHA-1
• 22: SHA-256

• ID: 3
• 'pba_salt': the salt, typically an 8-byte random sequence
• 'pba_iter': the iteration count, typically 200000

• 'pbe_salt': the salt, typically an 8-bytes random sequence
• 'pbe_iter': the iteration count, typically 100000

• ID: 1
• size: 32

• ID: 2
• size: 16

Initial  offset: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Shuffled offset: 3 2 1 0 5 4 7 6 8 9 10 11 12 13 14 15

• expressing the current chunk number as little endian on 16 bytes
• XORing it with the global IV
• encrypting with the global AES key in ECB mode (without IV).

• uint32le: unsigned int on 4 bytes, little endian.
• uint32be: unsigned int on 4 bytes, big endian.
• uint64le: unsigned int on 8 bytes, little endian.
• uchar: unsigned char on 1 byte.
• (hexadecimal): represents 1 byte with 2 hexadecimal numbers.
  e.g. 4142 (hexadecimal) means decimal values 65 and then 66.
• (ascii): represents 1 byte with the matching 1-byte ASCII value.
  e.g. 'AB\0' means decimal values 65 then 66 and then 0.
• UTF-16LE: text encoded using 2-bytes Unicode, little-endian, without Byte Order Mark (BOM), as commonly used under MS Windows.
• IV: initialization vector for AES encryption and decryption.
• CBC: Cipher Block Chaining, see Recommendation for Block Cipher Modes of Operation and Wikipedia.
• CTS: CipherText Stealing, one type of end-of-stream handling, see Recommendation for Block Cipher Modes of Operation: Three Variants of Ciphertext Stealing for CBC Mode and Wikipedia.
• PBA, PBE: password-based authentication, password-based encryption; see PKCS #11 Cryptographic Token Interface Current Mechanisms Specification.
• MS-CFB: Compound File Binary Format, documented at MSDN, libolecf, Wikipedia.
  Libraries that can manipulate this format include olefile, libolecf and libforensics.
• MS-OLEPS: OLE Property Set, documented at MSDN and libfole.
  The 'oleps.py' demo from libforensics displays detailed information on MS-OLEPS streams; olefile and libolecf have basic support.
• TLV : Type-Length-Value encoding ; 4 bytes for type, 4 bytes for length (uint32be), and a variable-sized value; see Type-length-value at Wikipedia for the general principle.
• FILETIME: timestamp as 1/10000000s since January 1, 1601 (UTC), see FILETIME structure at MSDN.

• The main container is using the (MS-CFB), which is notably used by MS Office 97-2003 .doc files. It contains several streams:
  • Metadata stream: in OLE Property Set format (MS-OLEPS), contains 2 blobs in a specific Type-Length-Value (TLV) format:
    • _ctlfile: global archive properties and access list
      It is obfuscated by means of static-key AES encryption.
      The properties include archive initial filename and a global IV.
      A global encryption key is itself encrypted in each user entry.
    • _catalog: file list
      Contains each file metadata indexed with a 15-bytes identifier.
      Directories are supported.
      Full filename is encrypted using AES.
      File extension is (redundantly) stored in clear, and so are file metadata such as modification time.
  • Each file in the archive compressed with zlib and encrypted with the standard AES algorithm, in a separate stream.
    Several encryption schemes and key sizes are supported.
    The file stream is split in chunks of 512 bytes, individually encrypted.
  • Optional streams, contain additional metadata as well as pictures to display in the application background ("watermarks"). They are not discussed here.

+----------------------------------------------------------------------------------------------------+
  .zed archive (MS-CBF)                                                                               
                                                                                                      
   stream #1                         stream #2                       stream #3...                     
  +------------------------------+  +---------------------------+  +---------------------------+      
    metadata (MS-OLEPS)               encryption (AES)               encryption (AES)                 
                                      512-bytes chunks               512-bytes chunks                 
    +--------------------------+                                                                      
      obfuscation (static key)        +-----------------------+      +-----------------------+        
      +----------------------+       -  compression (zlib)     -    -  compression (zlib)     -       
       _ctlfile (TLV)                                                                            ...  
      +----------------------+          +---------------+              +---------------+               
    +--------------------------+          file contents                  file contents                
                                                                                                      
    +--------------------------+     -  +---------------+      -    -  +---------------+      -       
      _catalog (TLV)                                                                                  
    +--------------------------+      +-----------------------+      +-----------------------+        
  +------------------------------+  +---------------------------+  +---------------------------+      
+----------------------------------------------------------------------------------------------------+

• obfuscated metadata encrypted with static AES key
• filenames and files in archives with 'encryption_mode' set to "AES-CBC-STREAM"
• any AES ciphertext of size < 16 bytes, regardless of encryption mode

• the second-to-last block of the ciphertext is encrypted in AES-ECB mode (i.e. block cipher encryption only, without XORing with the IV)
• then XOR-ed with the last partial block (hence truncated to the length of the partial block)

• filenames and files in archives with 'encryption_mode' set to "AES-CBC-CTS".
  • exception: if the size of the ciphertext is < 16 bytes, then "STREAM" is used instead.

• static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by 0100 (hexadecimal) (18 bytes total)
• 16-byte IV
• ciphertext
• 1 uint32be representing the length of all the above
• static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by "ZoneCentral (R)" (ASCII) and a NUL byte (32 bytes total)

• global archive properties as a 'fileprops' structure,
• extra archive properties as a 'archive_extraprops' structure
• users access list as a series of 'passworduser' and 'rsauser entries.

• 4 bytes for Type (specified as a 4-bytes hexadecimal below)
• 4 bytes for value Length (uint32be)
• Value

• 80110600: fileprops, used for the file list as well as for the global archive properties
• 001b0600: archive_extraprops
• 80140600: accesslist

• 80110600: fileprops (TLV structure): global archive properties
  • 00230400: archive_pathname (UTF-16LE string): initial archive filename (past versions also leaked the full pathname of the initial archive)
  • 80270200: encryption_mode (utf32be): 103 for "AES-CBC-STREAM", 104 for "AES-CBC-CTS"
  • 80260200: encryption_strength (utf32be): AES key size, in bytes (e.g. 32 means AES with a 256-bit key)
  • 80280500: files_iv (sequence of bytes): global IV for all filenames and file contents
• 001b0600: archive_extraprops (TLV structure): additionnal archive properties (optional)
  • 00c40500: archive_creationtime (FILETIME): date and time when archive was initially created (optional)
  • 00c00400: archive_createdwith (UTF-16LE string): uuid-like structure describing the application that initialized the archive (optional)
    00000188-1000-3CA8-8868-36F59DEFD14D is Zed! Free 1.0.188.
• 80140600: accesslist (TLV structure): describe the users, their key encryption and their permissions
  • 80610600: passworduser (TLV structure): user identified by password (0 or more)
  • 80620600: rsauser (TLV structure): user identified by RSA key (via file or PKCS#11 token) (0 or more)
  • Fields common to passworduser and rsauser:
    • 80710400: login (UTF-16LE string): user name
    • 80720300: login_md5 (sequence of bytes): used by the application to search for a user name
    • 807e0100: priv1 (uchar): user privileges; present and set to 1 when user is admin (optional)
    • 00830200: priv2 (uint32be): user privileges; present and set to 2 when user is admin, present and set to 5 when user is a marked as mandatory, e.g. for recovery keys (optional)
    • 80740500: files_key_ciphertext (sequence of bytes): the archive encryption key, itself encrypted
    • 00840500: user_creationtime (FILETIME): date and time when the user was added to the archive
  • passworduser-specific fields:
    • 80760500: pbe_salt (sequence of bytes): salt for PBE
    • 80770200: pbe_iter (uint32be): number of iterations for PBE
    • 80780200: pkcs12_hashfunc (uint32be): hash function used for PBE and PBA key derivation
    • 80790500: pba_checksum (sequence of bytes): password derived with PBA to check for password validity
    • 807a0500: pba_salt (sequence of bytes): salt for PBA
    • 807b0200: pba_iter (uint32be): number of iterations for PBA
  • rsauser-specific fields:
    • 807d0500: certificate (sequence of bytes): user X509 certificate in DER format

• 80110600: fileprops (TLV structure): describe the archive files (0 or more)
  • 80300500: file_id (sequence of bytes): a 16-byte unique identifier
  • 80310400: filename_halfanon (UTF-16LE string): half-anonymized filename, e.g. File1.txt (leaking filename extension)
  • 00380500: filename_ciphertext (sequence of bytes): encrypted filename; may have a trailing NUL byte once decrypted
  • 80330500: file_size (uint64le): decompressed file size in bytes
  • 80340500: file_creationtime (FILETIME): file creation date and time
  • 80350500: file_lastwritetime (FILETIME): file last modification date and time
  • 80360500: file_lastaccesstime (FILETIME): file last access date and time
  • 00370500: parent_directory_id (sequence of bytes): file_id of the parent directory, 0 is top-level
  • 80320100: is_dir (uint32be): 1 if entry is directory (optional)

• 21: SHA-1
• 22: SHA-256

• ID: 3
• 'pba_salt': the salt, typically an 8-byte random sequence
• 'pba_iter': the iteration count, typically 200000

• 'pbe_salt': the salt, typically an 8-bytes random sequence
• 'pbe_iter': the iteration count, typically 100000

• ID: 1
• size: 32

• ID: 2
• size: 16

Initial  offset: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Shuffled offset: 3 2 1 0 5 4 7 6 8 9 10 11 12 13 14 15

• expressing the current chunk number as little endian on 16 bytes
• XORing it with the global IV
• encrypting with the global AES key in ECB mode (without IV).

• uint32le: unsigned int on 4 bytes, little endian.
• uint32be: unsigned int on 4 bytes, big endian.
• uint64le: unsigned int on 8 bytes, little endian.
• uchar: unsigned char on 1 byte.
• (hexadecimal): represents 1 byte with 2 hexadecimal numbers.
  e.g. 4142 (hexadecimal) means decimal values 65 and then 66.
• (ascii): represents 1 byte with the matching 1-byte ASCII value.
  e.g. 'AB\0' means decimal values 65 then 66 and then 0.
• UTF-16LE: text encoded using 2-bytes Unicode, little-endian, without Byte Order Mark (BOM), as commonly used under MS Windows.
• IV: initialization vector for AES encryption and decryption.
• CBC: Cipher Block Chaining, see Recommendation for Block Cipher Modes of Operation and Wikipedia.
• CTS: CipherText Stealing, one type of end-of-stream handling, see Recommendation for Block Cipher Modes of Operation: Three Variants of Ciphertext Stealing for CBC Mode and Wikipedia.
• PBA, PBE: password-based authentication, password-based encryption; see PKCS #11 Cryptographic Token Interface Current Mechanisms Specification.
• MS-CFB: Compound File Binary Format, documented at MSDN, libolecf, Wikipedia.
  Libraries that can manipulate this format include olefile, libolecf and libforensics.
• MS-OLEPS: OLE Property Set, documented at MSDN and libfole.
  The 'oleps.py' demo from libforensics displays detailed information on MS-OLEPS streams; olefile and libolecf have basic support.
• TLV : Type-Length-Value encoding ; 4 bytes for type, 4 bytes for length (uint32be), and a variable-sized value; see Type-length-value at Wikipedia for the general principle.
• FILETIME: timestamp as 1/10000000s since January 1, 1601 (UTC), see FILETIME structure at MSDN.

Context aka. Sections This is probably the most relevant of the three proposals. Consider a bunch of related functions, say analyseExpr and analyseAlt, like these:

analyseExpr :: Expr -> Expr
analyseExpr (Var v) = change v
analyseExpr (App e1 e2) =
  App (analyseExpr e1) (analyseExpr e2)
analyseExpr (Lam v e) = Lam v (analyseExpr flag e)
analyseExpr (Case scrut alts) =
  Case (analyseExpr scrut) (analyseAlt <$> alts)
analyseAlt :: Alt -> Alt
analyseAlt (dc, pats, e) = (dc, pats, analyseExpr e)

You have written them, but now you notice that you need to make them configurable, e.g. to do different things in the Var case. You thus add a parameter to all these functions, and hence an argument to every call:

type Flag = Bool
analyseExpr :: Flag -> Expr -> Expr
analyseExpr flag (Var v) = if flag then change1 v else change2 v
analyseExpr flag (App e1 e2) =
  App (analyseExpr flag e1) (analyseExpr flag e2)
analyseExpr flag (Lam v e) = Lam v (analyseExpr (not flag) e)
analyseExpr flag (Case scrut alts) =
  Case (analyseExpr flag scrut) (analyseAlt flag <$> alts)
analyseAlt :: Flag -> Alt -> Alt
analyseAlt flag (dc, pats, e) = (dc, pats, analyseExpr flag e)

I find this code problematic. The intention was: flag is a parameter that an external caller can use to change the behaviour of this code, but when reading and reasoning about this code, flag should be considered constant. But this intention is neither easily visible nor enforced. And in fact, in the above code, flag does change , as analyseExpr passes something else in the Lam case. The idiom is indistinguishable from the environment idiom, where a locally changing environment (such as variables in scope ) is passed around. So we are facing exactly the same problem as when reasoning about a loop in an imperative program with mutable variables. And we (pure functional programmers) should know better: We cherish immutability! We want to bind our variables once and have them scope over everything we need to scope over! The solution I d like to see in Haskell is common in other languages (Gallina, Idris, Agda, Isar), and this is what it would look like here:

type Flag = Bool
section (flag :: Flag) where
  analyseExpr :: Expr -> Expr
  analyseExpr (Var v) = if flag then change1 v else change2v 
  analyseExpr (App e1 e2) =
    App (analyseExpr e1) (analyseExpr e2)
  analyseExpr (Lam v e) = Lam v (analyseExpr e)
  analyseExpr (Case scrut alts) =
    Case (analyseExpr scrut) (analyseAlt <$> alts)
  analyseAlt :: Alt -> Alt
  analyseAlt (dc, pats, e) = (dc, pats, analyseExpr e)

Now the intention is clear: Within a clearly marked block, flag is fixed and when reasoning about this code I do not have to worry that it might change. Either all variables will be passed to change1, or all to change2. An important distinction! Therefore, inside the section, the type of analyseExpr does not mention Flag, whereas outside its type is Flag -> Expr -> Expr. This is a bit unusual, but not completely: You see precisely the same effect in a class declaration, where the type signature of the methods do not mention the class constraint, but outside the declaration they do. Note that idioms like implicit parameters or the Reader monad do not give the guarantee that the parameter is (locally) constant. More details can be found in the GHC proposal that I prepared, and I invite you to raise concern or voice support there. Curiously, this problem must have bothered me for longer than I remember: I discovered that seven years ago, I wrote a Template Haskell based implementation of this idea in the seal-module package!

Less parentheses 1: Bulleted argument lists The next two proposals are all about removing parentheses. I believe that Haskell s tendency to express complex code with no or few parentheses is one of its big strengths, as it makes it easier to visualy parse programs. A common idiom is to use the $ operator to separate a function from a complex argument without parentheses, but it does not help when there are multiple complex arguments. For that case I propose to steal an idea from the surprisingly successful markup language markdown, and use bulleted lists to indicate multiple arguments:

foo :: Baz
foo = bracket
          some complicated code
          that is evaluated first
          other complicated code for later
          even more complicated code

I find this very easy to visually parse and navigate. It is actually possible to do this now, if one defines ( ) = id with infixl 0 . A dedicated syntax extension (-XArgumentBullets) is preferable:

• It only really adds readability if the bullets are nicely vertically aligned, which the compiler should enforce.
• I would like to use $ inside these complex arguments, and multiple operators of precedence 0 do not mix. (infixl -1 would help).
• It should be possible to nest these, and distinguish different nesting levers based on their indentation.

Less parentheses 1: Whitespace precedence The final proposal is the most daring. I am convinced that it improves readability and should be considered when creating a new language. As for Haskell, I am at the moment not proposing this as a language extension (but could be convinced to do so if there is enough positive feedback). Consider this definition of append:

(++) :: [a] -> [a] -> [a]
[]     ++ ys = ys
(x:xs) ++ ys = x : (xs++ys)

Imagine you were explaining the last line to someone orally. How would you speak it? One common way to do so is to not read the parentheses out aloud, but rather speak parenthesised expression more quickly and add pauses otherwise. We can do the same in syntax!

(++) :: [a] -> [a] -> [a]
[]   ++ ys = ys
x:xs ++ ys = x : xs++ys

The rule is simple: A sequence of tokens without any space is implicitly parenthesised. The reaction I got in Oxford was horror and disgust. And that is understandable we are very used to ignore spacing when parsing expressions (unless it is indentation, of course. Then we are no longer horrified, as our non-Haskell colleagues are when they see our code). But I am convinced that once you let the rule sink in, you will have no problem parsing such code with ease, and soon even with greater ease than the parenthesised version. It is a very natural thing to look at the general structure, identify compact chunks of characters , mentally group them, and then go and separately parse the internals of the chunks and how the chunks relate to each other. More natural than first scanning everything for ( and ), matching them up, building a mental tree, and then digging deeper. Incidentally, there was a non-programmer present during my presentation, and while she did not openly contradict the dismissive groan of the audience, I later learned that she found this variant quite obvious to understand and more easily to read than the parenthesised code. Some FAQs about this:

• What about an operator with space on one side but not on the other? I d simply forbid that, and hence enforce readable code.
• Do operator sections still require parenthesis? Yes, I d say so.
• Does this overrule operator precedence? Yes! a * b+c == a * (b+c).
• What is a token? Good question, and I am not not decided. In particular: Is a parenthesised expression a single token? If so, then (Succ a)+b * c parses as ((Succ a)+b) * c, otherwise it should probably simply be illegal.
• Can we extend this so that one space binds tighter than two spaces, and so on? Yes we can, but really, we should not.
• This is incompatible with Agda s syntax! Indeed it is, and I really like Agda s mixfix syntax. Can t have everything.
• Has this been done before? I have not seen it in any language, but Lewis Wall has blogged this idea before.

Well, let me know what you think!

• [1] https://etbe.coker.com.au/2016/11/06/thinkpad-rolls-royce/

~ ripmime -i .icedove/XXXXXXX.default/Mail/pop.xxxx/tmp -d target-directory

• Albanian (sq) level1 99%
• Albanian (sq) level2 25% (the rest is pending at #874497)
• Albanian (sq) level3 100%

• Bernhard M. Wiedemann:
  • File ordering:
    • klee-uclibc: sort
    • libdnet: sort
    • libinvm-cim: sort
    • libinvm-cli: sort
  • Embedded build-date timestamps:
    • robinhood: SOURCE_DATE_EPOCH support
    • ceph/rocksdb: SOURCE_DATE_EPOCH support
    • hylafax: use changelog modtime
    • gnucash: use changelog modtime
  • Warzone2100, merged: omit timestamps, sort file lists
• Chris Lamb:
  • glib2.0: sort file lists
  • (old) xorg, merged: SOURCE_DATE_EPOCH support

• Adrian Bunk:
  • #873608 filed against uhd.
  • #874186 filed against svgpp.
• Chris Lamb:
  • #873625 filed against glib2.0, filed upstream.
  • #874102 filed against texlive-bin.

• Chris Lamb:
  • bittornado/0.3.18-10.3 from #796212
  • cgilib/0.6-1.1 from #776935
  • dict-gazetteer2k/1.0.0-5.4 from #776376
  • dict-moby-thesaurus/1.0-6.4 from #776375
  • dtaus/0.9-1.1 from #777321
  • wily/0.13.41-7.3 from #777360

• Adrian Bunk (46)
• Mart n Ferrari (1)
• Steve Langasek (1)

• Mattia Rizzolo
  • tests/binary: skip a test if the 'distro' module is not available.
  • Some code quality and style improvements.
• Guangyuan Yang
  • tests/iso9660: support both cdrtools' genisoimage's versions of isoinfo.
• Chris Lamb
  • comparators/xml: Use name attribute over path to avoid leaking comparison full path in output.
  • Tidy diffoscope.progress a little.
• Ximin Luo
  • Add a --tool-prefix-binutils CLI flag. Closes: #869868
  • On non-GNU systems, prefer some tools that start with "g". Closes: #871029
  • presenters/html: Don't traverse children whose parents were already limited. Closes: #871413
• Santiago Torres-Arias
  • diffoscope.progress: Support the new fork of python-progressbar. Closes: #873157

• Ximin Luo:
  • Add -v/--verbose which is a bit more popular.
  • Make it possible to omit "auto" when building packages.
  • Refactor how the config file works, in preparation for new features.
  • chown -h for security.

2017, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

MeteoSurf is a free multi-source weather forecasting App designed to provide wind and wave conditions of the Mediterranean Sea. It is an application for smartphones and tablets, built as a Progressive Web App able to supply detailed and updated maps and data showing heights of sea waves (and other information) in the Central Mediterranean. It is mainly targeted for surfers and wind-surfers but anyone who needs to know the sea conditions will take advantage from this app.

Data can be displayed as animated graphical maps, or as detailed table data. The maps refer to the whole Mediterranean Sea, while the table data is able to provide specific information for any of the major surf spots in the Med.

As of current version, MeteoSurf shows data collecting them from 3 different forecasting systems Read More [by Fabio Marzocca]

Debian Long Term Support (LTS) This is my monthly Debian LTS report. This month I worked on a few major packages that took a long time instead of multiple smaller issues. Affected packages were Mercurial, libdbd-mysql-perl and Ruby.

Mercurial updates Mercurial was vulnerable to two CVEs: CVE-2017-1000116 (command injection on clients through malicious ssh URLs) and CVE-2017-1000115 (path traversal via symlink). The former is an issue that actually affects many other similar software like Git (CVE-2017-1000117), Subversion (CVE-2017-9800) and even CVS (CVE-2017-12836). The latter symlink issue is a distinct issue that came up during an internal audit. The fix, shipped as DLA-1072-1, involved a rather difficult backport, especially because the Mercurial test suite takes a long time to complete. This reminded me of the virtues of DEB_BUILD_OPTIONS=parallel=4, which sped up the builds considerably. I also discovered that the Wheezy build chain doesn't support sbuild's --source-only-changes flag which I had hardcoded in my sbuild.conf file. This seems to be simply because sbuild passes --build=source to dpkg-buildpackage, an option that is supported only in jessie or later.

libdbd-mysql-perl I have worked on fixing two issues with the libdbd-mysql-perl package, CVE-2017-10788 and CVE-2017-10789, which resulted in the DLA-1079-1 upload. Behind this mysteriously named package sits a critical piece of infrastructure, namely the mysql commandline client which is probably used and abused by hundreds if not thousands of home-made scripts, but also all of Perl's MySQL support, which is probably used by even a larger base of software. Through the Debian bug reports (Debian bug #866818 and Debian bug #866821), I have learned that the patches existed in the upstream tracker but were either ignored or even reverted in the latest 4.043 upstream release. It turns out that there are talks of forking that library because of maintainership issue. It blows my mind that such an important part of MySQL is basically unmaintained. I ended up backporting the upstream patches, which was also somewhat difficult because of the long-standing issues with SSL support in MySQL. The backport there was particularly hard to test, as you need to run that test suite by hand, twice: once with a server configured with a (valid!) SSL certificate and one without (!). I'm wondering how much time it is really worth spending on trying to fix SSL in MySQL, however. It has been badly broken forever, and while the patch is an improvement, I would actually still never trust SSL transports in MySQL over an untrusted network. The few people that I know use such transports wrap their connections around a simpler stunnel instead. The other issue was easier to fix so I submitted a pull request upstream to make sure that work isn't lost, although it is not clear what the future of that patch (or project!) will be at this point.

Rubygems I also worked on the rubygems issues, which, thanks to the "vendoring" practice of the Ruby community, also affects the ruby1.9 package. 4 distinct CVEs were triaged here (CVE-2017-0899, CVE-2017-0900, CVE-2017-0901 and CVE-2017-0902) and I determined the latter issue didn't affect wheezy as rubygems doesn't do its own DNS resolution there (later versions lookup SRV records). This is another package where the test suite takes a long time to run. Worse, the packages in Wheezy actually fails to build from source: the test suites just fail in various steps, particularly because of dh key too small errors for Rubygems, but also other errors for Ruby. I also had trouble backporting one test which I had to simply skip for Rubygems. I uploaded and announced test packages and hopefully I'll be able to complete this work soon, although I would certainly appreciate any help on this...

Triage I took a look at the sox, libvorbis and exiv2 issues. None had fixes available. sox and exiv2 were basically a list of fuzzing issues, which are often minor or at least of unknown severity. Those would have required a significant amount of work and I figured I would prioritize other work first. I also triaged CVE-2017-7506, which doesn't seem to affect the spice package in wheezy, after doing a fairly thorough audit of the code. The vulnerability is specifically bound to the reds_on_main_agent_monitors_config function, which is simply not present in our older version. A hostile message would fall through the code and not provoke memory allocation or out of bounds access, so I simply marked the wheezy version as not-affected, something which usually happens during the original triage but can also happen during the actual patching work, as in this case.

Other free software work This describes the volunteer work I do on various free software projects. This month, again, my internal reports show that I spent about the same time on volunteer and paid time, but this is probably a wrong estimate because I spent a lot of time at Debconf which I didn't clock in...

Debconf So I participated in the 17th Debian Conference in Montreal. It was great to see (and make!) so many friends from all over the world in person again, and I was happy to work on specific issues together with other Debian developers. I am especially thankful to David Bremner for fixing the syncing of the flagged tag when added to new messages (patch series). This allows me to easily sync the one tag (inbox) that is not statically assigned during notmuch new, by using flagged as a synchronization tool. This allows me to use notmuch more easily across multiple machines without having to sync all tags with dump/restore or using muchsync which wasn't working for me (although a new release came out which may fix my issues). The magic incantation looks something like this:

notmuch tag -inbox tag:inbox and not tag:flagged
notmuch tag +inbox not tag:inbox and tag:flagged

However, most of my time in the first week (Debcamp) was spent trying to complete the networking setup: configure switches, setup wiring and so on. I also configured an apt-cacher-ng proxy to serve packages to attendees during the conference. I configured it with Avahi to configure clients automatically, which led me to discover (and fix) issue Debian bug #870321) although there are more issues with the autodiscovery mechanism... I spent extra time to document the (somewhat simple) configuration of such a server in the Debian wiki because it was not the first time I had research that procedure... I somehow thought this was a great time to upgrade my laptop to stretch. Normally, I keep that device running stable because I don't use it often and I don't want to have major traumatizing upgrades every time I leave with it on a trip. But this time was special: there were literally hundreds of Debian developers to help me out if there was trouble. And there was, of course, trouble as it turns out! I had problems with the fonts on my display, because, well, I had suspended (twice) my laptop during the install. The fix was simply to flush the fontconfig cache, and I tried to document this in the fonts wiki page and my upgrades page. I also gave a short training called Debian packaging 101 which was pretty successful. Like the short presentation I made at the last Montreal BSP, the workshop was based on my quick debian development guide. I'm thinking of expanding this to a larger audience with a "102" course that would discuss more complex packaging problems. But my secret plan (well, secret until now I guess) is to make packaging procedures more uniform in Debian by training new Debian packagers using that same training for the next 2 decades. But I will probably start by just trying to do this again at the next Debconf, if I can attend.

Debian uploads I also sponsored two packages during Debconf: one was a "scratch an itch" upload (elpa-ivy) which I requested (Debian bug #863216) as part of a larger effort to ship the Emacs elisp packages as Debian packages. The other was an upload of diceware to build the documentation in a separate package and fix other issues I have found in the package during a review. I also uploaded a bunch of other fixes to the Debian archive:

• slop upstream update and matching maim NMU
• asked for gnome-web-photo removal (Debian bug #873015)
• charybdis 3.5.5-2 to officially switch to mbedtls (Debian bug #705369)
• gmpc: ship pending patches from git and Ubuntu
• horst: new upstream release and workaround for sparse bug Debian bug #873508
• uploaded percol to NEW (Debian bug #754972)

Signing keys rotation I also started the process of moving my main OpenPGP certification key by adding a signing subkey. The subkey is stored in a cryptographic token so I can sign things on more than one machine without storing that critical key on all those devices physically. Unfortunately, this meant that I need to do some shenanigans when I want to sign content in my Debian work, because the new subkey takes time to propagate to the Debian archive. For example, I have to specify the primary key with a "bang" when signing packages (debsign -k '792152527B75921E!' ...) or use inline signatures in email sent for security announcement (since that trick doesn't work in Mutt or Notmuch). I tried to figure out how to better coordinate this next time by reading up documentation on keyring.debian.org, but there is no fixed date for key changes on the rsync interface. There are "monthly changes" so one's best bet is to look for the last change in their git repository.

GitLab.com and LFS migration I finally turned off my src.anarc.at git repository service by moving the remaining repos to GitLab. Unfortunately, GitLab removed support for git-annex recently, so I had to migrate my repositories to Git-LFS, which was an interesting experience. LFS is pretty easy to use, definitely simpler than git-annex. It also seems to be a good match for the use-case at hand, which is to store large files (videos, namely) as part of slides for presentations. It turns out that their migration guide could have been made much simpler. I tried to submit those changes to the documentation but couldn't fork the GitLab EE project to make a patch, so I just documented the issue in the original MR for now. While I was there I filed a feature request to add a new reference shortcut (GL-NNN) after noticing a similar token used on GitHub. This would be a useful addition because I often have numbering conflicts between Debian BTS bug numbers and GitLab issues in packages I maintain there. In particular, I have problems using GitLab issue numbers in Monkeysign, because commit logs end up in Debian changelogs and will be detected by the Debian infrastructure even though those are GitLab bug numbers. Using such a shortcut would avoid detection and such a conflict.

Numpy-stats I wrote a small tool to extract numeric statistics from a given file. I often do ad-hoc benchmarks where I store a bunch of numbers in a file and then try to make averages and so on. As an exercise in learning NumPy, I figured I would write such a simple tool, called numpy-stats, which probably sounds naive to seasoned Python scientists. My incentive was that I was trying to figure out what was the distribution of password length in a given password generator scheme. So I wrote this simple script:

for i in seq 10000 ; do
    shuf -n4 /usr/share/dict/words   tr -d '\n'
done > length

And then feed that data in the tool:

$ numpy-stats lengths 
 
  "max": 60, 
  "mean": 33.883293722913464, 
  "median": 34.0, 
  "min": 14, 
  "size": 143060, 
  "std": 5.101490225062775
 

I am surprised that there isn't such a tool already: hopefully I am wrong and will just be pointed towards the better alternative in the comments here!

Safe Eyes I added screensaver support to the new SafeEyes project, which I am considering as a replacement to the workrave project I have been using for years. I really like how the interruptions basically block the whole screen: way more effective than only blocking the keyboard, because all potential distractions go away. One feature that is missing is keystrokes and mouse movement counting and of course an official Debian package, although the latter would be easy to fix because upstream already has an unofficial build. I am thinking of writing my own little tool to count keystrokes, since the overlap between SafeEyes and such a counter isn't absolutely necessary. This is something that workrave does, but there are "idle time" extensions in Xorg that do not need to count keystrokes. There are already certain tools to count input events, but none seem to do what I want (most of them are basically keyloggers). It would be an interesting test to see if it's possible to write something that would work both for Xorg and Wayland at the same time. Unfortunately, preliminary research show that:

1. in Xorg, the only way to implement this is to sniff all events, ie. to implement a keylogger
2. in Wayland, this is completely unsupported. it seems some compositors could implement such a counter, but then it means that this is compositor specific, or, in other words, unportable

So there is little hope here, which brings to my mind "painmeter" as an appropriate name for this future programming nightmare.

Ansible I sent my first contribution to the ansible project with a small documentation fix. I had an eye opener recently when I discovered a GitLab ansible prototype that would manipulate GitLab settings. When I first discovered Ansible, I was frustrated by the YAML/Jinja DSL: it felt silly to write all this code in YAML when you are a Python developer. It was great to see reasonably well-written Python code that would do things and delegate the metadata storage (and only that!) to YAML, as opposed to using YAML as a DSL. So I figured I would look at the Ansible documentation on how this works, but unfortunately, the Ansible documentation is severly lacking in this area. There are broken links (I only fixed one page) and missing pieces. For example, the developing plugins page doesn't explain how to program a plugin at all. I was told on IRC that: "documentation around developing plugins is sparse in general. the code is the best documentation that exists (right now)". I didn't get a reply when asking which code in particular could provide good examples either. In comparison, Puppet has excellent documentation on how to create custom types, functions and facts. That is definitely a turn-off for a new contributor, but at least my pull request was merged in and I can only hope that seasoned Ansible contributors expand on this critical piece of documentation eventually.

Misc As you can see, I'm all over the place, as usual. GitHub tells me I "Opened 13 other pull requests in 11 repositories" (emphasis mine), which I guess means on top of the "9 commits in 5 repositories" mentioned earlier. My profile probably tells a more detailed story that what would be useful to mention here. I should also mention how difficult it is to write those reports: I basically do a combination of looking into my GitHub and GitLab profiles, the last 30 days of emails and filesystem changes (!!). En vrac, a list of changes which may be of interest:

• font-large (and its alias, font-small): shortcut to send the right escape sequence to rxvt so it changes its font
• fix-acer: short script to hardcode the modeline (you remember those?!) for my screen which has a broken EDID pin (so autodetection fails, yay Xorg log files...)
• ikiwiki-pandoc-quickie: fake ikiwiki renderer that (ab)uses pandoc to generate a HTML file with the right stylesheet to preview Markdown as it may look in this blog (the basic template is missing still)
• git-annex-transfer: a command I've often been missing in git-annex, which is a way to transfer files between remotes without having to copy them locally (upstream feature request)
• I linked the graphics of the Debian archive software architecture in the Debian wiki in the hope more people notice it.
• I did some tweaks on my Taffybar to introduce a battery meter and hoping to have temperature sensors, which mostly failed. there's a pending pull request that may bring some sense into this, hopefully.
• I made two small patches in Monkeysign to fix gpg.conf handling and multiple email output, a dumb bug I cannot believe anyone noticed or reported just yet. Thanks Valerie for the bug report! The upload of this in Debian is pending a review from the release team.

system("ffmpeg -y -i $outputdir/$slug.ts -pass 1 -passlogfile ...");

package Object;
use Moose;
has 'time' => (
    is => 'rw',
    builder => 'read_time',
    lazy => 1,
);
sub read_time  
    return localtime();
 

package SReview::Video;
use Moose;
has 'url' => (
    is => 'rw',
)
has 'video_codec' => (
    is => 'rw',
    builder => '_probe_videocodec',
    lazy => 1,
);
has 'videodata' => (
    is => 'bare',
    reader => '_get_videodata',
    builder => '_probe_videodata',
    lazy => 1,
);
has 'probedata' => (
    is => 'bare',
    reader => '_get_probedata',
    builder => '_probe',
    lazy => 1,
);
sub _probe_videocodec  
    my $self = shift;
    return $self->_get_videodata-> codec_name ;
 
sub _probe_videodata  
    my $self = shift;
    if(!exists($self->_get_probedata-> streams ))  
        return  ;
     
    foreach my $stream(@ $self->_get_probedata-> streams )  
        if($stream-> codec_type  eq "video")  
            return $stream;
         
     
    return  ;
 
sub _probe  
    my $self = shift;
    open JSON, "ffprobe -print_format json -show_format -show_streams " . $self->url . " "
    my $json = "";
    while(<JSON>)  
        $json .= $_;
     
    close JSON;
    return decode_json($json);
 

• call ffprobe, and cache the (decoded) output for further use
• from that, extract the video stream data, and cache that for further use
• from that, extract the name of the used codec, cache it, and then return that name to the caller.

has 'reference' => (
    is => 'ro',
    isa => 'SReview::Video',
    predicate => 'has_reference'
);

sub _probe  
    my $self = shift;
    if($self->has_reference)  
        return $self->reference->_get_probedata;
     
    # original code remains here
 

my $video = SReview::Video->new(url => 'file.ts');
my $generated = SReview::Video->new(url => 'file2.ts', reference => $video);