Search Results: "js"

10 March 2016

Lunar: Reproducible builds: week 45 in Stretch cycle

What happened in the reproducible builds effort between February 28th and March 5th:

Toolchain fixes
  • Antonio Terceiro uploaded gem2deb/0.27 that forces generated gemspecs to use the date from debian/changelog.
  • Antonio Terceiro uploaded gem2deb/0.28 that forces generated gemspecs to have their contains file lists sorted.
  • Robert Luberda uploaded ispell/3.4.00-5 which make builds of hashes reproducible.
  • C dric Boutillier uploaded ruby-ronn/0.7.3-4 which will make the output locale agnostic. Original patch by Chris Lamb.
  • Markus Koschany uploaded spring/101.0+dfsg-1. Fixed by Alexandre Detiste.
Ximin Luo resubmitted the patch adding the --clamp-mtime option to Tar on Savannah's bug tracker. Lunar rebased our experimental dpkg on top of the current master branch. Changes in the test infrastructure are required before uploading a new version to our experimental repository. Reiner Herrmann rebased our custom texlive-bin against the latest uploaded version.

Packages fixed The following 77 packages have become reproducible due to changes in their build dependencies: asciidoctor, atig, fuel-astute, jekyll, libphone-ui-shr, linkchecker, maven-plugin-testing, node-iscroll, origami-pdf, plexus-digest, pry, python-avro, python-odf, rails, ruby-actionpack-xml-parser, ruby-active-model-serializers, ruby-activerecord-session-store, ruby-api-pagination, ruby-babosa, ruby-carrierwave, ruby-classifier-reborn, ruby-compass, ruby-concurrent, ruby-configurate, ruby-crack, ruby-css-parser, ruby-cucumber-rails, ruby-delorean, ruby-encryptor, ruby-fakeweb, ruby-flexmock, ruby-fog-vsphere, ruby-gemojione, ruby-git, ruby-grack, ruby-htmlentities, ruby-jekyll-feed, ruby-json-schema, ruby-listen, ruby-markerb, ruby-mathml, ruby-mini-magick, ruby-net-telnet, ruby-omniauth-azure-oauth2, ruby-omniauth-saml, ruby-org, ruby-origin, ruby-prawn, ruby-pygments.rb, ruby-raemon, ruby-rails-deprecated-sanitizer, ruby-raindrops, ruby-rbpdf, ruby-rbvmomi, ruby-recaptcha, ruby-ref, ruby-responders, ruby-rjb, ruby-rspec-rails, ruby-rspec, ruby-rufus-scheduler, ruby-sass-rails, ruby-sass, ruby-sentry-raven, ruby-sequel-pg, ruby-sequel, ruby-settingslogic, ruby-shoulda-matchers, ruby-slack-notifier, ruby-symboltable, ruby-timers, ruby-zip, ticgit, tmuxinator, vagrant, wagon, yard. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: Patches submitted which have not made their way to the archive yet:
  • #816209 on elog by Reiner Herrmann: use printf instead of echo which is shell-independent.
  • #816214 on python-pip by Reiner Herrmann: removes timestamp from generated Python scripts.
  • #816230 on rows by Reiner Herrmann: tell grep to always treat the input as text.
  • #816232 on eficas by Reiner Herrmann: use printf instead of echo which is shell-independent.
Florent Daigniere and bancfc reported that linux-grsec was currently built with GRKERNSEC_RANDSTRUCT which will prevent reproducible builds with the current packaging.

tests.reproducible-builds.org pbuilder has been updated to the last version to be able to support Build-Depends-Arch and Build-Conflicts-Arch. (Mattia Rizzolo, h01ger) New package sets have been added for Subgraph OS, which is based on Debian Stretch: packages and build dependencies. (h01ger) Two new armhf build nodes have been added (thanks Vagrant Cascadian) and integrated in our Jenkins setup with 8 new armhf builder jobs. (h01ger)

strip-nondeterminism development strip-nondeterminism version 0.016-1 was released on Sunday 28th. It will now normalize the POT-Creation-Date field in GNU Gettext .mo files. (Reiner Herrmann) Several improvements to the packages metadata have also been made. (h01ger, Ben Finney)

Package reviews 185 reviews have been removed, 91 added and 33 updated in the previous week. New issue: fileorder_in_gemspec_files_list. 43 FTBFS bugs were reported by Chris Lamb, Martin Michlmayr, and gregor herrmann.

Misc. After merging the patch from Dhiru Kholia adding support for SOURCE_DATE_EPOCH in rpm, Florian Festi opened a discussion on the rpm-ecosystem mailing list about reproducible builds. On March 4th, Lunar gave an overview of the general reproducible builds effort at the Internet Freedom Festival in Valencia.

Antoine Beaupr : Trying out Keybase

I was privileged enough to get access to the alpha preview of Keybase.io. Last year, the project raised 11M$ in venture capital and they seem to be attacking the hard problems, so it is a serious project, worth considering for any cryptography hacker like me. This was obviously discussed elsewhere previously but since Keybase has evolved a bit since then, I figured it would be worth adding my grain of salt.

The elite alpha problem My first problem with Keybase was of course that I wasn't in the elite club of invited people. Somehow, I finally got in but the problem remains for the everyone else. In my case, it was the Keybase filesystem announcement that brought my attention back to the project, although I have yet to test that itself - yet another invite-only problem (need to email chris@keybase.io to get access, I guess). Still, 10GB of storage might be interesting for some people, but considering the number of hurdles or "friction", as they say, that needs to be jumped over to get access, I am not sure how much traction this will get for now. The project has been in alpha for a while now (more than a year) with no announced public beta just yet, which is way slower than similarly challenging projects with less capital. Let's encrypt, as a comparison, was publicly announced in 2014 and by the end of 2015, the public beta was opened.

Private key leakage My second major concern is that Keybase does worrisome things to private key material. If you tell it to import your regular GPG keys, it will copy them into the keybase keyring, which is already a problem: now you have two places where you store sensitive secrets. But worse, Keybase offers you to upload your private key material on the server, without clearly and explicitely exposing the potential risks that means to the user. The key, presumably, is encrypted (at least that is what I can parse from the Keybase privacy policy). Yet this is still a concern: a determined attacker (e.g. the government or another malicious entity) could force Keybase to give up the encrypted private key material and install sniffers that would reveal the passphrase that would allow decryption and impersonation of the related identity. That is a huge deal. This is what caused Hushmail's demise in 2007: they had similar promises about how they had a "secure" email (in this case) service based on OpenPGP, but since they key was stored on the server, they only had to serve a malicious applet to victims and were able to give cleartext data to the US government (12 CDs!) under a Mutual Legal Assistance Treaty with Canada. In Keybase, When I generate a new key, it asks me if I want to upload the key to their servers, and "Yes" is chosen as a default, so it is clearly a policy that is encouraged. This is presumably to enable browser-based crypto, yet this could be done by storing keys within the browser, not on the server. The mere possibility that the client can meddle with secret key material in such a way is a problem. Even if uploading secret keys to the server is optional, the fact that it is a possibility widens the attack surface on the client significantly. GPG has been working on isolating private key operations to a different process (gpg-agent) or smart cards, SSH has privilege seperation. The idea of uploading secret key material online is so beyond current practices that it should bring the project to a full stop already. My private key material is pretty sensitive. I am a Debian developer: if someone gets access to that key, they can upload arbitrary code for thousands of packages, affecting thousands if not millions of users. That is a huge deal, and I am not ready to give that power to an arbitrary third party that I know little about.

Privacy policy issues Furthermore, the Keybase privacy policy explicitely states that "Logs of this information may persist for an indefinite period", which include, and I quote:
  • Your computer s IP address
  • Your preferences and settings (time zone, language, privacy preferences, application preferences, etc.)
  • The URL of the site that referred you to the Service
  • The buttons and controls you clicked on (if any) within the Service
  • How long you used the Service and which parts and features you used
  • Session times and lengths
For a company working to improve user's privacy, this is a pretty bad policy. It exposes Keybase users to undue surveillance, on a lot of private information that is not necessary to operate the service. I haven't actually read the remaining of the policy, as I was already scared as hell and figure I would just log off for now and try not to add any extra personal information in that already growing pile. Let's mention, while I'm here, that the keybase commandline client has this odd way of working that it forks a daemon in the background all the time, even to run just keybase status. One has to wonder if this thing is included in the surveillance data above, which would make the keybase client an amazing surveillance device in itself. To stop all this, you need: 
keybase logout
keybase ctl stop
Not quite obvious...

Usability issues Which brings me to a series of usability issues I have found while working with Keybase. Having worked on usability myself in the Monkeysphere project, I understand how hard those problems can be. But right now, the fact is that keybase is only a commandline client, with some web-based sugar sprinkled on top. I recognize the huge efforts that have been done to make the user experience (UX) as easy as possible through the commandline, but this is definitely not "grand public" material yet. Even as a OpenPGP hacker, Keybase can get confusing. Since it has its own separate keyring from GPG, things that used to be obvious before are suddenly new things to learn. For example, I originally generated a key for anarcat@debian.org in keybase, thinking it would be a separate thing from my current GPG identity. Well, that ended up creating a new key in my GPG keyring, duplicating my existing identity, with no way for me to tell this was a Keybase copy. This could have ended up on the keyservers and confuse anyone looking for my PGP key.

Revoking keys So I revoked that key, which, in itself, is not such an obvious process either. I first had to "drop" the key: 
keybase pgp drop '010143572d4e438f457a7447c9758804cc7be44c1ee2b7915c3904567d0a3fb5cf590a'
To find that magic number, I had to go on the website and click through the PGP key to end up on the revoke link which told me which magic string to put. The web UI was telling me I could use keybase status to get that information, but I couldn't find that in the output here: 
$ keybase status
Username:      anarcat
Logged in:     no
Device:
    name:      angela
    ID:        070f6a13f2a66afbb463f49dadfd4518
    status:    active
Keybase keys:  unlocked
Session:       anarcat [salt only]
KBFS:
    status:    not running
    version:
    log:       /home/anarcat/.cache/keybase/keybase.kbfs.log
Service:
    status:    running
    version:   1.0.14-1
    log:       /home/anarcat/.cache/keybase/keybase.service.log
Platform Information:
    OS:        linux
    Runtime:   go1.5.3
    Arch:      amd64
Client:
    version:   1.0.14-1
Desktop app:
    status:    not running
    version:
    log:       /home/anarcat/.cache/keybase/Keybase.app.log
Config path:   /home/anarcat/.config/keybase/config.json
Default user:  anarcat
Other users:
command-line client:
    keybase status [pid: 14562, version: 1.0.14-1]
So keybase drop "deletes" the key from the keybase client (and presumably the server). One problem here is that we have three different verbs to mean the same thing: in some places, the website says "deleted" (in the web graph), the commandline client uses "drop" and everyone else using PGP in the world uses "revoke". So what is it? Also note that just dropping that key is not enough to eradicate that key: you still need to generate and import a revocation certificate on GPG's side as well, for that key. This makes sense, because you may have imported that key and you may not want to destroy it when you remove it from keybase, but there could be a little better guessing and UX here as well.

Password usage Then another problem is that, when you signup on the commandline, it asks you to choose a passphrase. It's not clear at all from the UI what that passphrase is for. Is it to protect the private key material? To login the website? Both? It turns out it is actually to login to the website and the Keybase API. It could also be the key to the private key material, but I haven't quite figured that out yet. Similarly, the "paper devices" are a little confusing to me as well. The registration process is fairly insistent that I need to write a series of secret words down on a piece of paper and put it my wallet. This is based on the idea that you can use that key to recover your account and setup new devices. However, it is not made clear at all how those credentials should be protected or used. What if I lose my wallet? What do I do then? Is it okay if i write it on the back of my laptop instead? I know it sounds like stupid questions to crypto geek, but it's definitely stuff people will do, and education, at all levels, is necessary here.

The existing web of trust Finally, I found it surprisingly counter-intuitive to sign keys with keybase. I was assuming the whole point of the thing was to expand the web of trust, but it seems their mission lies somewhere else: there is no facility to sign keys directly in keybase. You sign statements on various social sites (Github, Twitter, Reddit...) but doing traditional key exchanges seems to be off limits. My obvious use case was to sign my existing key material with my newly generated anarcat@keybase.io key, so that keybase would become another way of verifying my identity. Well, this is not possible with keybase directly - you still have to go through the regular: 
gpg --sign-key DEADBEEF
Which is really unfortunate, because some craaaazy people like me insist on doing in person, actually secure key exchanges sometimes, and the tools to do this right now (mainly gpg, caff and monkeysign) really need a lot of love and help. Keybase could have helped a lot in pushing those initiatives forward and bring new ways of doing secure key exchanges, instead of only lowering the standards on cryptographic identity.

Usual proprietary startup rant I can't help but finish by noticing that the whole basic problem with Keybase is more fundamental than a few usability issues, which can all be fixed fairly easily. Keybase.IO is a web-based service that is entirely proprietary. The client is freely downloadable and free software - but the server side is not. This makes us rely on a central point of failure and the goodwill of those operators to not only keep the service running (for free?) indefinitely, but to protect us against all attackers, state-run or otherwise. This is definitely against basic free software principles and the open architecture of the internet federation, well embodied in the Franklin street statement. Online services should be decentralized and open like the OpenPGP keyservers have always been, period.

Conclusion I am curious to see where Keybase brings us, as a community. So far, I am concerned about centralization and devaluation of the web of trust as cryptographic system. I do not believe that trusting multiple corporate social networking sites bring much benefit to our security, although it does improve usability significantly and is certainly better than TOFU, so that part of the project is definitely interesting, especially since it allows leveraging classical internet protocols like DNS and HTTPS. Key exchange is a critical problem that still hasn't properly been resolved in the cryptographic community. Most efforts at building communication tools (say like Signal or Telegram) mostly ignore the problem and expect people to read up strings of numbers or rely on synchronous communications (see the Axolotl rachet for Signal's take on it). Keybase's attempts at fixing this are great, but needs much more work to actually resolve the PKI issues in a significant way. More fundamentally, the practice of storing keys on the servers should just stop: it is a definite no go for me, and a classic crypto mistake that has bitten way too many people. In my mind, it discredits the whole project. The point of OpenPGP is end to end encryption across devices: storing the keys on a server breaks that apart, and doesn't improve much on the existing HTTPS security systems already in place on the web. It is especially a problem given the new waves of attacks on cryptography from western governments, from the UK to California... Until those issues are resolved, I can't get myself to recommend Keybase to anyone just yet.

Post-scriptum Note to Keybase developers: I have considered filing issues regarding the above, but unfortunately, I was unable to filter through the gigantic github issues list for duplicates, and didn't have time to file detailed reports for everything. I apologize for bypassing those usual conventions. If you want to try out Keybase to make your own opinion, I do have 4 invites I can give away, but I will do so only if you need to test specific issues. Otherwise, I note that there is a Request For Package for the Debian package to become official, that Debian developers (or Keybase developers!) may be interested in completing.

Michal Čihař: Weblate 2.5

After almost six months of development Weblate 2.5 has been released. It brings lot of improvements and it's quite hard to point few ones. The most important ones include support for Python 3, reports generators, placeables highlighting, extended keyboard shortcuts, configurable dashboard or group based ACLs. Full list of changes for 2.5: If you are upgrading from older version, please follow our upgrading instructions. You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Aptoide, FreedomBox, Weblate itself and many other projects. Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure. Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: English Weblate 0 comments

8 March 2016

Michal Čihař: Weekly phpMyAdmin contributions 2016-W09

The last week was mostly spent on bugfixing and cleanup after security releases. Hopefully the amount of security reports will go down now. Most of the bug fixes were in the SQL parser which influences quite a lot of parts of phpMyAdmin. It is responsible for splitting queries on import, generating queries for export or linting the queries as users type them. Additionally Debian packages were also updated, for both unstable and testing and for Ubuntu PPA. Handled issues:

Filed under: Debian English phpMyAdmin 0 comments

7 March 2016

Hideki Yamane: Debian meeting in Tokyo (2016 March)

5th March, we've held Debian (Debian meeting) in Tokyo. (Cybozu, Inc.) kindly provides their office for meeting, thanks for folks in Cybozu.



This time, three talks were given:

    • Kentaro is "groonga"(an open-source fulltext search engine and column store) upstream author and package maintainer in Debian and Fedora. This talk is about his experience how to use Debian "porterbox" for non-DDs.
  • "Porting Debian to tilegx" by @wskoka
    • About his experience porting Debian to tilegx architecture, multicore processor family by Tilera. He is not porting expert ("I'm sale person", he said :), but did try&error and now apt is working on that machine.
  • "Introduction to Debian Ports" by John Paul Adrian Glaubitz
    • Adrian comes from Germany(!) and gave a talk about debian-ports.


During break time, did some discussion, GPG keysigning and enjoyed coffee and sweets provided by Cybozu, thanks!

And thanks for all participates!

5 March 2016

Antonio Terceiro: Debian Ruby Sprint 2016 - day 5: More Reproducible Builds, Retrospective, and A Little Bit of Tourism

Earlier today I was made aware by Holger of the results of our reproducibility efforts during the sprint. I would like to thank Lunar for pinging us about the issue, and Holger for pointing me to updated results. The figure below depicts a stacked area chart where the X axis is time and the green area is reproducible packages. Red is packages that fail to build, and Orange are unreproducible packages I was able to book accommodation for the sprint attendees very close to both my place and the sprint venue, what was very useful but also had this downside of them not being able to see much of city. As the final day of the sprint was getting closer, we decided to have a different lunch to allow them to see one of the most famous local landmarks, the botanical gardens. So we headed down to the botanical gardens, grabbed a few items for lunch at the park coffee shop, and set out to visit this very beautiful place. I have to say that there is the place were I usually take every visitor I have. We were joined by Gioavani who had just arrived for the the MiniDebconf on the following weekend. The final lists of accomplishments of the day was again very impressive
  • r10k 2.1.1-2
  • run massive update on team repositories
    • bump Standards-Version
    • fix Vcs-* fields
    • drop version in gem2deb build-dependency
    • set debhelper compatibility level to 9
    • update the default ruby-tests.rake
  • day 4 report
  • uploaded ruby-faraday-middleware-multi-json 0.0.6-2
  • uploaded ruby-powerpack 0.1.1-2
  • uploaded ruby-contracts 0.13.0-1
  • uploaded ruby-chef-config 12.7.2-1 (NEW)
  • uploaded ruby-foreigner #808530 and asked it removal from the NEW queue (was already ROMed)
  • filled for RM ruby-opengraph-parser (#816752)
  • new how-can-i-help version developed and uploaded
  • uploaded ruby-romkan to unstable (from exp)
  • uploaded ruby-rinku to unstable (from exp)
  • uploaded ruby-ole to unstable (from exp)
  • uploaded ruby-net-ldap to unstable (from exp)
  • uploaded ruby-rack-mobile-detect to unstable (from exp)
  • uploaded gem2deb 0.28 to help with reproducible builds: filenames are now sorted
  • uploaded rails 2:4.2.5.2-2 with packaging improvements
    • run unit tests during the build and on CI
    • apply upstream patch to fix ActiveRecord breakage under Ruby 2.3
  • pushed a ton of tags for existing uploads
  • merged improvements to the team master repository
    • review/cleanup the contents of the repository
    • improved helper scripts to automate the workflow (upload, build, new-upstream, etc)
  • followed up on ruby2.3 transition, filed #816698 against subversion because of ftbfs on mips, mipsel
  • put ruby-cocoon into a better state
  • uploaded ruby-plist
  • gem2deb: gem2tgz will now create foo.gemspec (easier to patch) instead of metadata.yml
  • gemwatch: ditto
  • close #794139 jekyll bug (unreproducible)
  • close #798934 ruby-ffi-rzmq bug (unreproducible)
  • closed ftbfs #816586 #800057 #784699 as unreproducible
  • reassigned #760952 #680297 to ruby2.3 (from ruby2.2)
  • investigated how to list packages with non-buildd-binary uploads
  • ScottK has removed ruby2.1 from unstable!
By the end of the afternoon I asked everyone to fill out a simple retrospective list, what we can use later to make future sprints better and better. Below are the results we got. What was good:
  • restricted room hours actually made for a nice rhythm (did not apply for a long time )
  • very good food
  • very cheap food!
  • longer period makes the effort of travel more worthwhile
  • many participants and longer sprint than usual allowing more work to be done
  • good preparation with clear goals, make the sprint usefull
  • patience with the less experienced participants
  • RFS very fast
  • Ant nio is an excellent host
  • You all are so helpful
  • great dinners
What could be better:
  • room too close to the street, too much vehicle noise, but sometimes nice music
  • more coffee ^W meat
  • could know more portugues so ordering food would have been easier
  • debian infra could have not been down during the sprint
The night ended at Bar do Alem o ( The German s Bar ). Both their beer and their food are very good, but I don t have enough elements to vouch for their authenticity. :) We were joined by Giovani (who we also met earlier in the botanic gardens), and by Paulo and Daniel who are organizing the MiniDebconf. And that is the end of this year s Debian Ruby team sprint. I hope we do it all over again next year.

Lunar: Reproducible builds: week 44 in Stretch cycle

What happened in the reproducible builds effort between February 21th and February 27th:

Toolchain fixes Didier Raboud uploaded pyppd/1.0.2-4 which makes PPD generation deterministic. Emmanuel Bourg uploaded plexus-maven-plugin/1.3.8-10 which sorts the components in the components.xml files generated by the plugin. Guillem Jover has implemented stable ordering for members of the control archives in .debs. Chris Lamb submitted another patch to improve reproducibility of files generated by cython.

Packages fixed The following packages have become reproducible due to changes in their build dependencies: dctrl-tools, debian-edu, dvdwizard, dymo-cups-drivers, ekg2, epson-inkjet-printer-escpr, expeyes, fades, foomatic-db, galternatives, gnuradio, gpodder, gutenprint icewm, invesalius, jodconverter-cli latex-mk, libiio, libimobiledevice, libmcrypt, libopendbx, lives, lttnganalyses, m2300w, microdc2, navit, po4a, ptouch-driver, pxljr, tasksel, tilda, vdr-plugin-infosatepg, xaos. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them:

tests.reproducible-builds.org The reproducibly tests for Debian now vary the provider of /bin/sh between bash and dash. (Reiner Herrmann)

diffoscope development diffoscope version 50 was released on February 27th. It adds a new comparator for PostScript files, makes the directory tests pass on slower hardware, and line ordering variations in .deb md5sums files will not be hidden anymore. Version 51 uploaded the next day re-added test data missing from the previous tarball. diffoscope is looking for a new primary maintainer.

Package reviews 87 reviews have been removed, 61 added and 43 updated in the previous week. New issues: captures_shell_variable_in_autofoo_script, varying_ordering_in_data_tar_gz_or_control_tar_gz. 30 new FTBFS have been reported by Chris Lamb, Antonio Terceiro, Aaron M. Ucko, Michael Tautschnig, and Tobias Frost.

Misc. The release team reported on their discussion about the topic of rebuilding all of Stretch to make it self-contained (in respect to reproducibility). Christian Boltz is hoping someone could talk about reproducible builds at the openSUSE conference happening June 22nd-26th in N rnberg, Germany.

4 March 2016

Daniel Pocock: Machine Learning for Hackers with Debian and Ubuntu

Data Science and Machine Learning are hot topics at the moment. Many people are considering how to extend their skills into these areas and many solutions have appeared, including full online degrees, free online courses combined with free software and for those who prefer hard copy, a staggering choice of books on the topic. One of those books is O'Reilly's Machine Learning for Hackers by John Myles White and Drew Conway. The book uses R to demonstrate a series of techniques for analysis and prediction. The book offers a great opportunity to simultaneously get an introduction to basic machine learning techniques and also an introduction to the increasingly popular R platform. On page 11 they list all the major R packages needed to run their examples (available on Github). I had a look over this list to see how many could be installed on a Debian system using apt-get and found that about half of them were already present. Five of them and one dependency, however, were not already available so I've whipped up packages for them and they are now in jessie-backports for all users of the current stable release. If you are following the exercises in this book, you can get all the software you need with one convenient command: 
$ sudo apt-get install -t jessie-backports \
  r-cran-ggplot2 r-cran-lme4 r-cran-rcurl \
  r-cran-reshape r-cran-xml r-cran-arm \
  r-cran-glmnet r-cran-igraph r-cran-lubridate \
  r-cran-rjsonio r-cran-tm
Thanks to all those who already packaged other parts of R and backported the relevant packages. Note that the RJSONIO package's authors have not provided a valid free software license so it is in non-free. It is there to support people using the book but I would encourage people to use RJSON for any new projects as it does have a valid license.

Enrico Zini: Praise of component reuse

I farm bits and pieces out to the guys who are much more brilliant than I am. I say, "build me a laser", this. "Design me a molecular analyzer", that. They do, and I just stick 'em together. (Seth Brundle, "The Fly")
When I decided to try and turn siterefactor into staticsite, I decided that I would go ahead only for as long as it could be done with minimal work, writing code in the most straightforward way on top of existing and stable components. I am pleased by how far that went. Python-Markdown It works fast enough, already comes with extensions for most of what I needed, and can be extended in several ways. One of the extension methods is a hook for manipulating the ElementTree of the rendered document before serializing it to HTML, which made it really easy to go and process internal links in all <a href= and <img src= attributes. To tell an internal link from an external link I just use the standard python urlparse and see if the link has a scheme or a netloc component. If it does not, and if it has a path, then it is an internal link. This also means that I do not need to invent new Markdown syntax for internal references, avoiding the need for remembering things like [text]( < relref "blog/post.md" > ) or [text]( filename /blog/post.md). In staticsite, it's just [text](/blog/post.md) or [text](post.md) if the post is nearby. This feels nicely clean to me: if I wanted to implement fancy markdown features, I could do it as Python-Markdown extensions and submit them upstream. If I wanted to implement fancy interlinking features, I could do it with a special url scheme in links. For example, it would be straigtforward to implement a ssite: url scheme that expanded the url with elements from staticsite's settings using a call to python's string.format (ssite: SETTING_NAME /bar maybe?), except I do not currently see any use cases for extending internal linking from what it is now. Jinja2 Jina2 is a template engine that I already knew, it is widely used, powerful and pleasant to use, both on the templating side and on the API's side. It is not HTML specific, so I can also use it to generate Atom, RSS2, "dynamic" site content, and even new site Markdown pages. Implementing RSS and Atom feeds was just a matter of writing and testing these Jinja2 macros and then reusing them anywhere. toml, yaml, json No need to implement my own front matter parsing. Also, reusing the same syntax as Hugo allows me to just link to its documentation. python-slugify I found python-slugify so I did not bother writing a slug-generating function. As a side effect, now things works better than I would even have thought to implement, including transliteration of non-ascii characters: 
$ ./ssite new example --noedit --title "Cos  parl  Enrico"
/enrico-dev/staticsite/example/site/blog/2016/cosi-parlo-enrico.md
(I just filed an RFP) python-livereload Implementing ssite serve which monitors the file system and autoreloads when content changes and renders everything on the fly, took about an hour. Most of that hour went into implementing rendering pages on demand. Then I discovered that it autoreloads even when I edit staticsite's source code. Then I discovered that it communicates with the browser and even automatically triggers a page refresh. I can keep vim on half my screen and a browser in the other half, and I get live preview for free every time I save, without ever leaving the editor. Bootstrap I already use Bootstrap at work, so creating the default theme templates with it took about 10 minutes. This morning I tried looking at my website using my mobile phone, and I pleasantly saw it automatically turning into a working mobile version of itself. Pygments Python-Markdown uses Pygments for syntax highlighting, and it can be themed just by loading a .css. So, without me really doing anything, even staticsite's syntax highligthing is themable, and there's even a nice page with a list of themes to choose from. Everything else... Command line parsing? Straight argparse. Logging? python's logging support. Copying static resource files? shutil.copy2. Parsing dates? dateutil.parser. Timing execution? time.perf_counter. Timezone handling? pytz. Building the command to run an editor? string.format. Matching site pages? fnmatch.translate. ...and then some. If I ever decide to implement incremental rendering, how do I implement tracking which source files have changed? Well, for example, how about just asking git?

Arturo Borrero Gonz lez: How are we training to climb Mulhacen

Alcazaba, Punta de la Cornisa and Mulhac n 3.479 m

I have been in some adventures with a group of friends for a couple of years now. We call ourselves Los Extraviaos (a quick translation to english is perhaps The Wanderers).

We like nature adventures. During summer we use to do water adventures (like rivers trekking, waterfall climbing, snorkeling, canyoning, rafting, kayaking and so on). During winter and spring, we use to do trekking, hiking and mountaineering (low altitudes, keep reading).

For this sport season, one of our main milestones is to climb the second most prominent mountain in Spain: Mulhacen, with 3.479 m over sea level, during summer (June).
From an alpinist point of view, this isn't a very big deal, since most of the mountain's snow and ice is completely melted and the altitude is not that high.
However, from our amateur mountaineering point of view we will be taking a lot of kilometres to walk and a lot of elevation gain uphill (almost 1.600 m), which is in fact a thing to take into account.
Anyway, Mulhacen is considered 'high mountaineering' and all cautions should be taken both during preparation/training and during the climbing of the mountain itself.

To train our bodies and minds we are climbing lower mountains. We have climbed 3 or 4 already, and will continue to climb more in upcoming moths until we go to Mulhacen.
Also, it's important to climb mountains in different weather conditions. We need to train how to overcome wind, snow, storm, cold, night situations. The way we learn most is facing new, challenging situations.

We started by going to the summit of 'El Aljibe', 1.094 m over sea level, with about 500 m elevation gain uphill during 13 km.
This time, the weather was very windy. A video of the summit moment:


A pic of the group:


Next was the 'Torrecilla' peak. The summit is at 1.919 m over sea level, with about 900 m elevation gain uphill during 15 km.
This time we faced a snowy trekking.
Here a music video of that day:


The Torrecilla peak is in fact challenging with snow, so we went a second time, with even more snow and wind:


This second time we faced winds of 70km/h which, due to wind-chill, resulted in a very cold day.

We also went to Sierra de Grazalema Natural Park, in a hard trekking under snowfall:


This time we walked about 15 km with about 500m elevation gain uphill, while enjoying this kind of environment:


What's next: more mountains! We plan to reach 2.069 m over sea level next week in 'La Maroma' and keep doing steep trekking in upcoming months until we go in June to Mulhacen.

It should be very funny to do some FLOSS contribution from the summit of a mountain. Do you know of someone who did that?

1 March 2016

Thorsten Alteholz: My Debian Activities in February 2016

FTP assistant This month I marked 364 package for accept and rejected 66. Due to the help of lamby, the length of the NEW queue dropped mostly below 50, so there is no need for complaints anymore :-) . I also sent 22 emails to maintainers asking questions. Squeeze LTS This was my twentieth month that I did some work for the Squeeze LTS initiative, started by Raphael Hertzog at Freexian. This month more people started to contribute and my workload dropped down to 11.25h. Altogether I uploaded those DLAs:
  • [DLA 424-1] didiwiki security update
  • [DLA 423-1] krb5 security update
  • [DLA 433-1] xerces-c security update
  • [DLA 444-1] php5 security update
This month I was also involved in embargoed uploads and could do an upload on my own (DLA 433-1). Now Squeeze LTS is officially done. I leave it with mixed feelings. On the one hand it became more and more difficult to backport patches for the latest version to the old software. On the other hand I could learn a lot of stuff about the methods other maintainers used some years ago. Yes, although not always visible at first sight, over the years there are lots of improvements on how packages can be handled in Debian. So, let us start with Wheezy now Other stuff On the way to pump.io, grunt and some other cool stuff, I uploaded:
  • node-abab
  • node-array-equal
  • node-array-flatten
  • node-array-unique
  • node-cors
  • node-deep-extend
  • node-original
  • node-simplesmtp
  • node-setimmediate
  • node-uglify-save-license
  • node-unpipe
Yes, sometimes this npm2deb makes it really easy to create a package. In order to fix FTBFSs, errors from DebCI or whatever might fail these days, I also uploaded new versions of:
  • node-array-equal
  • node-array-parallel
  • node-bufferjs
  • node-crc
  • node-css-what
  • node-eventsource
  • node-mime-types
  • node-mocks-http
  • node-rai
  • node-requires-port
  • node-url-parse
  • node-xoauth2
Today I could see the first fruits of my labor. Some packages, I did not touch, migrated to testing because some of their dependencies were finally able to migrate as well.

Ritesh Raj Sarraf: Linux Power Savings 2016

Having moved to a new place, now at times, I also have to deal with power outages. As heat increases, the power outages will be much longer and more frequent. So much, that UPS and Power Inverters run out. Such are ideal times to measure idle power consumption for my laptop. Here's what my default (and idle) OS looks like. It should be standard to most "typical" users. Some minor odds could be apport, dnsmasq, and maybe, tor. Also, thanks to systemd, there are some native services which are now converted to socket activation based ones. There's also Laptop Mode Tools for userspace power savings. But huge thanks to Intel's work in making the kernel more power efficient. With things like Runtime PM and Intel P-State, we can now define, default and aggressive power-savings modes, helping conserve more power. We still have challenges with things like binary-only drivers; and also stock GPL drivers but with an external firmware blob. There have been many instances when my previous machines (which had both) ran into severe power drain. The challenge with power savings is that it is not just the job of the kernel. All components, kernel and userspace, need to be made aware of, and work in unison. Otherwise, one ill behaving web browser tab can be the cause of power drain. P-State driver seems to already be doing some scale down of CPU resources. And with CGroups maturing moe I think we'll eventually have all Desktop Environments making use of it. 
systemd-+-ModemManager-+- gdbus 
                        - gmain 
         -NetworkManager-+- gdbus 
                          - gmain 
         -accounts-daemon-+- gdbus 
                           - gmain 
         -agetty
         -apport-notifyd
         -atd
         -avahi-daemon---avahi-daemon
         -bluetoothd
         -colord-+- gdbus 
                  - gmain 
         -cron
         -dbus-daemon
         -dnsmasq
         -evolution-+- book-client-dbu 
                     - dconf worker 
                     -7*[ evolution ]
                     - gdbus 
                     - gmain 
         -gdm3-+-gdm-session-wor-+-gdm-x-session-+-Xorg
                                                  -gnome-session-b-+-gnome-settings--+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                                      - pool 
                                                                                      - threaded-ml 
                                                                    -gnome-shell-+-ibus-daemon-+-ibus-dconf-+- dconf worker 
                                                                                                             - gdbus 
                                                                                                             - gmain 
                                                                                                -ibus-engine-sim-+- gdbus 
                                                                                                                  - gmain 
                                                                                                - gdbus 
                                                                                                - gmain 
                                                                                  - JS GC Helper 
                                                                                  - JS Sour~ Thread 
                                                                                  - dconf worker 
                                                                                  - gdbus 
                                                                                  - gmain 
                                                                                  - threaded-ml 
                                                                    - dconf worker 
                                                                    - gdbus 
                                                                    - gmain 
                                                  - gdbus 
                                                  - gmain 
                                  - gdbus 
                                  - gmain 
                -gdm-session-wor-+-gdm-x-session-+-Xorg
                                                  -gnome-session-b-+-evolution-alarm-+- cal-client-dbus 
                                                                                      - dconf worker 
                                                                                      - evolution-alarm 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                    -gnome-settings--+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                                      - pool 
                                                                                      - threaded-ml 
                                                                    -gnome-shell-+-2*[redshift-+- gdbus ]
                                                                                                - gmain ]
                                                                                  - JS GC Helper 
                                                                                  - JS Sour~ Thread 
                                                                                  - dconf worker 
                                                                                  - gdbus 
                                                                                  - gmain 
                                                                                  - threaded-ml 
                                                                    -parcimonie
                                                                    -parcimonie-appl-+- gdbus 
                                                                                      - gmain 
                                                                    -python3---libinput-debug-
                                                                    -ssh-agent
                                                                    -tracker-extract-+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                                      -10*[ pool ]
                                                                    -tracker-miner-a-+- gdbus 
                                                                                      - gmain 
                                                                    -tracker-miner-f-+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                    -tracker-miner-u-+- gdbus 
                                                                                      - gmain 
                                                                    - dconf worker 
                                                                    - gdbus 
                                                                    - gmain 
                                                  - gdbus 
                                                  - gmain 
                                  - gdbus 
                                  - gmain 
                - gdbus 
                - gmain 
         -geoclue-+- gdbus 
                   - gmain 
         -gnome-keyring-d-+- gdbus 
                           - gmain 
                           - timer 
         -gpg-agent
         -gsd-printer-+- gdbus 
                       - gmain 
         -ibus-daemon-+-ibus-dconf-+- dconf worker 
                                    - gdbus 
                                    - gmain 
                       -ibus-engine-sim-+- gdbus 
                                         - gmain 
                       - gdbus 
                       - gmain 
         -ibus-x11-+- gdbus 
                    - gmain 
         -iio-sensor-prox-+- gdbus 
                           - gmain 
         -inetd---leafnode
         -libvirtd---15*[ libvirtd ]
         -lvmetad
         -master-+-pickup
                  -qmgr
                  -tlsmgr
         -mcelog
         -minissdpd
         -mount.ntfs
         -onboard-+- dconf worker 
                   - gdbus 
                   - gmain 
         -packagekitd-+- gdbus 
                       - gmain 
         -polkitd-+- gdbus 
                   - gmain 
         -2*[pulseaudio-+-gconf-helper]
                         - alsa-sink-ALC23 ]
                         - alsa-sink-HDMI  ]
                         - alsa-sink-pcspe ]
                         - alsa-source-ALC ]
         -rtkit-daemon---2*[ rtkit-daemon ]
         -systemd-+-(sd-pam)
                   -at-spi-bus-laun-+-dbus-daemon
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -at-spi2-registr-+- gdbus 
                                     - gmain 
                   -dbus-daemon
                   -dconf-service-+- gdbus 
                                   - gmain 
                   -goa-daemon-+- dconf worker 
                                - gdbus 
                                - gmain 
                   -goa-identity-se-+- gdbus 
                                     - gmain 
                                     - pool 
                   -gvfs-afc-volume-+- gdbus 
                                     - gmain 
                                     - gvfs-afc-volume 
                   -gvfs-goa-volume-+- gdbus 
                                     - gmain 
                   -gvfs-gphoto2-vo-+- gdbus 
                                     - gmain 
                   -gvfs-mtp-volume-+- gdbus 
                                     - gmain 
                   -gvfs-udisks2-vo-+- gdbus 
                                     - gmain 
                   -gvfsd-+- gdbus 
                           - gmain 
                   -gvfsd-fuse-+- gdbus 
                                - gmain 
                                - gvfs-fuse-sub 
                                -2*[ gvfsd-fuse ]
                   -gvfsd-metadata-+- gdbus 
                                    - gmain 
                   -mission-control-+- dconf worker 
                                     - gdbus 
                                     - gmain 
         -systemd-+-(sd-pam)
                   -at-spi-bus-laun-+-dbus-daemon
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -at-spi2-registr-+- gdbus 
                                     - gmain 
                   -dbus-daemon
                   -gconfd-2
         -systemd-+-(sd-pam)
                   -at-spi-bus-laun-+-dbus-daemon
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -at-spi2-registr-+- gdbus 
                                     - gmain 
                   -dbus-daemon
                   -dconf-service-+- gdbus 
                                   - gmain 
                   -evolution-addre-+-2*[evolution-addre-+- dconf worker ]
                                                          - evolution-addre ]
                                                          - gdbus ]
                                                          - gmain ]
                                     - dconf worker 
                                     - evolution-addre 
                                     - gdbus 
                                     - gmain 
                   -evolution-calen-+-evolution-calen-+- dconf worker 
                                                       - evolution-calen 
                                                       - gdbus 
                                                       - gmain 
                                                       -6*[ pool ]
                                     -evolution-calen-+- book-client-dbu 
                                                       - dconf worker 
                                                       - evolution-calen 
                                                       - gdbus 
                                                       - gmain 
                                                       - pool 
                                     -2*[evolution-calen-+- dconf worker ]
                                                          - evolution-calen ]
                                                          - gdbus ]
                                                          - gmain ]
                                     - dconf worker 
                                     - evolution-calen 
                                     - gdbus 
                                     - gmain 
                   -evolution-sourc-+- dconf worker 
                                     - gdbus 
                                     - gmain 
                   -gconfd-2
                   -gnome-shell-cal-+- cal-client-dbus 
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                                     - gnome-shell-cal 
                   -gnome-terminal--+-bash---pstree
                                     -bash
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -goa-daemon-+- dconf worker 
                                - gdbus 
                                - gmain 
                   -goa-identity-se-+- gdbus 
                                     - gmain 
                                     - pool 
                   -gpaste-daemon-+- dconf worker 
                                   - gdbus 
                                   - gmain 
                   -gvfs-afc-volume-+- gdbus 
                                     - gmain 
                                     - gvfs-afc-volume 
                   -gvfs-goa-volume-+- gdbus 
                                     - gmain 
                   -gvfs-gphoto2-vo-+- gdbus 
                                     - gmain 
                   -gvfs-mtp-volume-+- gdbus 
                                     - gmain 
                   -gvfs-udisks2-vo-+- gdbus 
                                     - gmain 
                   -gvfsd-+- gdbus 
                           - gmain 
                   -gvfsd-burn-+- gdbus 
                                - gmain 
                   -gvfsd-dnssd-+- gdbus 
                                 - gmain 
                   -gvfsd-fuse-+- gdbus 
                                - gmain 
                                - gvfs-fuse-sub 
                                -2*[ gvfsd-fuse ]
                   -gvfsd-metadata-+- gdbus 
                                    - gmain 
                   -gvfsd-network-+- dconf worker 
                                   - gdbus 
                                   - gmain 
                   -gvfsd-trash-+- gdbus 
                                 - gmain 
                   -mission-control-+- dconf worker 
                                     - gdbus 
                                     - gmain 
                   -nautilus-+-gedit
                              - dconf worker 
                              - gdbus 
                              - gmain 
                   -tracker-store-+- dconf worker 
                                   - gdbus 
                                   - gmain 
                                   -4*[ pool ]
         -systemd-journal
         -systemd-logind
         -systemd-network
         -systemd-timesyn--- sd-resolve 
         -systemd-udevd
         -thermald--- thermald 
         -tor
         -udisksd-+- cleanup 
                   - gdbus 
                   - gmain 
                   - probing-thread 
         -upowerd-+- gdbus 
                   - gmain 
         -wpa_supplicant
So I used the standard power measurement tool, powertop. Below is a screenshot of PowerTop claiming 9 - 10 hours of possible battery backup. This is idle state. "Idle" would mean when the user is not interacting with the machine. When under basic usage (email, web), the actual battery backup I've sensed is somewhere around 6-7 hrs, which is still good from what we got a couple years ago. There's one oddity in the screenshot though. The estimated power consumption reported is 4.55W, where as, the power consumption of just the wifi card is reported 6.17W. That too, when the wifi was disconnected. This reminded me of the bug report against PowerTop, which is now more than 2 years old. We've still not concluded in that bug report, so if anyone can shed some light on PowerTop's reporting, please do share it in the bug report. Note: Please ignore the terminal's title in the screenshot. That is wrong and I've not bothered to figure out why it is displaying the title name constructed from an outdated session, which is already terminated.

Categories:

Keywords:

Like:

Stein Magnus Jodal: February contributions

The following is a short summary of my open source work in February, just like in previous months.

Debian

Mopidy
  • Released Mopidy-Spotify 2.3.1: Works around search being broken in libspotify by using the Spotify Web API for the search functionality, and nothing else.
  • Wrapped up and released Mopidy 2.0 release. This was a quite large release that ports to GStreamer 1.x and adds support for gapless playback. It was uploaded to Debian in time to be part of Ubuntu 16.04 LTS.
  • Released Mopidy-Spotify 3.0.0: A couple of small compatibility fixes to work with Mopidy 2.x and GStreamer 1.x.
  • Updates to packages not in Debian, only at apt.mopidy.com:
    • Uploaded mopidy-spotify-tunigo 1.0.0-0mopidy1. Far down the chain, it depends on libspotify, which is not in Debian.
    • Uploaded mopidy-spotify 2.0.0-0mopidy1.
  • Updated homebrew-mopidy with lots of new releases:
    • Mopidy 2.0.0
    • Mopidy-Dirble 1.3.0
    • Mopidy-SoundCloud 2.0.2
    • Mopidy-Spotify 2.3.0, 2.3.1 and 3.0.0
    • python-backports-abc 0.4
    • python-backports-ssl-match-hostname 3.5.0.1
    • python-certifi 2015.11.20.1
    • python-cffi 1.5.0 and 1.5.2
    • python-requests 2.9.1
    • python-singledispatch 3.4.0.3
    • python-six 1.10.0
    • python-tornado 4.3
  • Rebased my feature/py3-compat branch on top of Mopidy 2.0.
  • Planned breaking changes for Mopidy 3.0: unbundling of Mopidy.js, removal of deprecated core APIs, model APIs, audio APIs and removal of the http/static_dir config.
  • Planned breaking changes to modernize Mopidy.js: change default calling convention to by-position-or-by-name, replace When.js with ES6 Promise, replace Bane.js with EventEmitter, and replace Buster.js with Karma+Mocha.

Comics
  • Started working on the upgrade to Django 1.8. Most deps are upgraded. The major remaining pieces is to upgrade django-registration from 0.8 to 2.0 and to replace the years-old vendored copy of django-invitation (singular) with django-invitations (plural).

29 February 2016

Chris Lamb: Free software activities in February 2016

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):
  • Various improvements to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
    • Added explicit support to send messages asynchronously via the Celery distributed task queue. (#29)
    • Worked with Patrick Clope to add an escapeslack template filter to ensure the correct characters for Slack's API are escaped using Django's built-in safe is too invasive. (#36)
    • Corrected an issue where custom endpoints and channel names were incompatible. (#27)
    • Overhaul of the field/option/block parsing. (49ff3c4)
    • Moved away from using a django.template.Context instance, preventing a deprecation warning. (#31)
  • Added a create-folders subcommand to my tickle-me-email Getting Things Done (GTD) email toolbox to create numbered folders for the rotate "tickler" functionality. (#3)
  • Wrote and released a Django template tag to collapse multiple whitespace/newline characters in order to correctly format, for example, plaintext emails that make extensive use of for-loops. These would normally require careful and fragile placement of the % for .. % and % endfor % tags. (Repo)
  • Pushed a number of updates to my Strava Enhancement Suite, a Chrome extension that improves and fixes annoyances in the web interface of the Strava cycling and running tracker:
    • Fix unit conversion where the element was wrapped in an another HTML element. (#48)
    • Hide premium indicator from top-level navigation. (#45)
    • Correct matching of the "shop" link in top-level navigation and footer module. (#47 & #46)
    • Remove "premium" pages on Club pages. (#44)
  • Added a security-oriented warning to Ansible's documentation regarding the behaviour if an UFW firewall application profile is added and subsequently removed. (#1740)
  • Added support to my django-staticfiles-dotd "staticfiles" library which concatenates Javascript and CSS files from .d-style directories to support an arbitrary file rendering method. This allows the use of media pre-processors such as SASS within such directories. (#1)
  • Corrected my Chrome extension for the FastMail web interface to correctly hook into the internal "send" mechanism. (#2)
  • Moved my stravabot IRC bot to use dh-virtualenv and systemd, improving security and reliability. (1c48709)
  • Updated django-pedantic-http-methods a tool to raise an exception during development when attempting to perform side effects in GET and HEAD HTTP methods to support the latest version of Django. (#1)
Debian
  • Updated travis.debian.net a hosted script to easily test and build Debian packages on the Travis CI continuous integration platform to support:
    • Automatic bumping of the version number in debian/changelog based on TRAVIS_BUILD_NUMBER. (#14)
    • Security repositories. Thanks to Stefan Jenkner for the initial pull request. These are additionally now enabled by default. (#15)
    • The backports repositories. (#13)
  • Applied #812830 and #812830 from James Clark to the Debian Archive Kit to improve the interface of various webpages it generates.
  • Updated the SSL certificate for try.diffoscope.org, a hosted version of the diffoscope in-depth and content-aware diff utility. Thanks to Bytemark for sponsoring the hardware.
  • Worked on my slides for Reproducible Builds - fulfilling the original promise of free software, to be presented at FOSSASIA '16.
My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#40, #41, #42, #43)
LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:
  • "Frontdesk" duty for the week of 22nd 28th, triaging CVEs, etc.
  • Proofread announcements, etc. for the upcoming migration to wheezy-lts.
  • Issued DLA 417-1 for xdelta3 to fix a buffer overflow that allowed arbitrary code execution from input files.
  • Issued DLA 420-1 for libmatroska, correcting a heap information leak.
  • Issued DLA 428-1 for websvn fixing a cross-site scripting vulnerability.
  • Issued DLA 429-1 for pixman fixing a buffer overflow issue.
  • Issued DLA 430-1 & DLA 431-1 for libfcgi and libfcgi-perl respectfully, fixing a remote denial-of-service (DoS) vulnerability.
Uploads
  • redis (2:3.0.7-2) Correcting my SOURCE_DATE_EPOCH reproducibility patch as the conditional was accidentally inverted. Thanks to Reiner Herrmann (deki).
  • disque (1.0~rc1-5) Making the parallel SOURCE_DATE_EPOCH patch change and additionally tidying the packaging after introducing procps as a build-dependency.
RC bugs


I also filed 137 FTBFS bugs against aac-tactics, angular.js, astyle, bcftools, blacs-mpi, bogofilter, boxes, caldav-tester, ccdproc, ckeditor, coq-float, cqrlog, dasher, django-recurrence, dspdfviewer, eclipse-egit, ess, etcd, felix-latin, fio, flexml, funny-manpages, gap-atlasrep, garmin-plugin, gitlab, gnome-mines, graphicsmagick, haskell-nettle, healpy, hg-git, hunspell, hwloc, ijs, ipset, janest-core-extended, jpathwatch, kcompletion, kcompletion, keyrings.alt, kodi-pvr-hts, kodi-pvr-vdr-vnsi, libcommons-compress-java, libgnome2-wnck-perl, libkate, liblrdf, libm4ri, libnet-server-mail-perl, libsis-jhdf5-java, libspectre, libteam, libwnck, libwnckmm, libxkbcommon, lombok, lombok-patcher, mako, maven-dependency-analyzer, mopidy-mpris, mricron, multcomp, netty-3.9, numexpr, ocaml-textutils, openimageio, openttd-openmsx, osmcoastline, osmium-tool, php-guzzle, php-net-smartirc, plexus-component-metadata, polari, profitbricks-client, pyentropy, pynn, pyorbital, pypuppetdb, python-aioeventlet, python-certifi, python-hglib, python-kdcproxy, python-matplotlib-venn, python-mne, python-mpop, python-multipletau, python-pbh5tools, python-positional, python-pydot-ng, python-pysam, python-snuggs, python-tasklib, r-cran-arm, r-cran-httpuv, r-cran-tm, rjava, ros-geometry-experimental, ros-image-common, ros-pluginlib, ros-ros-comm, rows, rr, ruby-albino, ruby-awesome-print, ruby-default-value-for, ruby-fast-gettext, ruby-github-linguist, ruby-gruff, ruby-hipchat, ruby-omniauth-crowd, ruby-packetfu, ruby-termios, ruby-thinking-sphinx, ruby-tinder, ruby-versionomy, ruby-zentest, sbsigntool, scikit-learn, scolasync, sdl-image1.2, signon-ui, sisu-guice, sofa-framework, spykeutils, ssreflect, sunpy, tomcat-maven-plugin, topmenu-gtk, trocla, trocla, tzdata, verbiste, wcsaxes, whitedune, wikidiff2, wmaker, xmlbeans, xserver-xorg-input-aiptek & zeroc-icee-java.
FTP Team

As a Debian FTP assistant I ACCEPTed 107 packages: androguard, android-platform-dalvik, android-platform-development, android-platform-frameworks-base, android-platform-frameworks-native, android-platform-libnativehelper, android-platform-system-core, android-platform-system-extras, android-platform-tools-base, android-sdk-meta, apktool, armci-mpi, assertj-core, bart, bind9, caja, caldav-tester, clamav, class.js, diamond, diffoscope, django-webpack-loader, djangocms-admin-style, dnsvi, esptool, fuel-astute, gcc-6-cross, gcc-6-cross-ports, gdal, giella-core, gnupg, golang-github-go-ini-ini, golang-github-tarm-serial, gplaycli, gradle-jflex-plugin, haskell-mountpoints, haskell-simple, hurd, iceweasel, insubstantial, intellij-annotations, jetty9, juce, keyrings.alt, leptonlib, libclamunrar, libdate-pregnancy-perl, libgpg-error, libhtml5parser-java, libica, libvoikko, linux, llvm-toolchain-3.8, lombok-patcher, mate-dock-applet, mate-polkit, mono-reference-assemblies, mxt-app, node-abab, node-array-equal, node-array-flatten, node-array-unique, node-bufferjs, node-cors, node-deep-extend, node-original, node-setimmediate, node-simplesmtp, node-uglify-save-license, node-unpipe, oar, openjdk-8, openjdk-9, pg8000, phantomjs, php-defaults, php-random-compat, php-symfony-polyfill, pnetcdf, postgresql-debversion, pulseaudio-dlna, pyconfigure, pyomo, pysatellites, python-fuelclient, python-m3u8, python-pbh5tools, python-qtpy, python-shellescape, python-tunigo, pyutilib, qhull, r-cran-rjsonio, r-cran-tm, reapr, ruby-fog-dynect, scummvm-tools, symfony, talloc, tesseract, twextpy, unattended-upgrades, uwsgi, vim-command-t, win-iconv, xkcdpass & xserver-xorg-video-ast. I additionally REJECTed 4 packages.

Norbert Preining: Debian/TeX Live full update 20160223

Only one week has passed since the last Debian/TeX Live update, and there is already a new one? The explanation is simple: consolidation of packages and preparation for 2016. The update comprises of uploads of texlive-base, texlive-lang, texlive-extra, texlive-bin, musixtex, pmx, m-tx, xmltex, jadetex. Besides the usual changes it brings a merge back in of several hitherto separately packaged packages. Debian - TeX Live 2015 The details are as follows:
  • Music related packages (musixtex, pmx, m-tx) have been merged into texlive-music, and temporary upgrade packages for the old packages have been uploaded
  • HTML/SGML/XML related packages (jadetex, xmltex) have been merged into the new package texlive-htmlxml, and temporary upgrade packages for the old ones have been uploaded
Furthermore, since some packages ship binaries, texlive-bin has been updated to build the new binaries, too. Lots of changes, but in the long run I see less work and better upgrades. Especially keeping the music-related packages uptodate was a bit a pain, and using the standard TeX Live update mechanisms looks much more promising. And, not to forget, a few more new and updated packages: New packages bibletext, cochineal, hyphen-occitan, libertinegc, lroundrect, miama, mparrows, scrlttr2copy, visualpstricks. Updated packages animate, babel-friulan, babel-russian, bxjscls, caption, celtic, cmtiup, crimson, crossrefware, droit-fr, dvips, fibeamer, fira, fithesis, forest, geschichtsfrkl, gost, hobby, hyperxmp, hyphen-base, hyph-utf8, inconsolata, ipaex-type1, mcf2graph, media9, reledmac, roundrect, showhyphens, spath3, splitindex, unicode-data. Enjoy.

13 February 2016

Mart n Ferrari: Why I am not touching node.js

Dear node.js/node-webkit people, what's the matter with you? I wanted to try out some stuff that requires node-webkit. So I try to use npm to download, build and install it, like CPAN would do. But then I see that the nodewebkit package is just a stub that downloads a 37MB file (using HTTP without TLS) containing pre-compiled binaries. Are you guys out of your minds? This is enough for me to never again get close to node.js and friends. I had already heard some awful stories, but this is just insane. Update: This was a rage post, and not really saying anything substantial, but I want to quote here the first comment reply from Christian as I think it is much more interesting:
I see a recurring pattern here: developers create something that they think is going to simplify distributing things by quite a bit, because what other people have been doing is way too complicated[tm]. In the initial iteration it usually turns out to have three significant problems:
  • a nightmare from a security perspective
  • there's no concern for stability, because the people working on it are devs and not people who have certain reliability requirements
  • bundling other software is often considered a good idea, because it makes life so much easier[tm]
Given a couple of years they start to catch up where the rest of the world (e.g. GNU/Linux distributions) is - at least to some extent. And then their solution is actually of a similar complexity compared to what is in use by other people, because they slowly realize that what they're trying to do is actually not that simple and that other people who have done things before were not stupid and the complexity is actually necessary... Just give node.js 4 or 5 more years or so, then you won't be so disappointed. ;-) I've seen this pattern over and over again:
  • the Ruby ecosystem was a catastrophe for quite some time (just ask e.g. the packages of Ruby software in distributions 4 or 5 years ago), and only now would I actually consider using things written in Ruby for anything productive
  • docker and vagrant were initially not designed with security in mind when it came to downloading images - only in the last one or two years have there actually been solutions available that even do the most basic cryptographic verification
  • the entire node.js ecosystem mess you're currently describing
Then the next new fad comes along in the development world and everything starts over again. The only thing that's unclear is what the next hype following this pattern is going to be.
And I also want to quote myself with what I think are some things you could do to improve this situation:
  1. You want to make sure your build is reproducible (in the most basic sense of being re-buildable from scratch), that you are not building upon scraps of code that nobody knows where they came from, or which version they are. If possible, at the package level don't vendor dependencies, depend on the user having the other dependencies pre-installed. Building should be a mainly automatic task. Automation tools then can take care of that (cpan, pip, npm).
  2. By doing this you are playing well with distributions, your software becomes available to people that can not live on the bleeding edge, and need to trust the traceability of the source code, stability, patching of security issues, etc.
  3. If you must download a binary blob, for example what Debian non-free does for Adobe Flashplayer, then for the love of all that is sacred, use TLS and verify checksums!
8 comments

9 February 2016

Michal &#268;iha&#345;: Weekly phpMyAdmin contributions 2016-W05

Last week was really focused on code cleanups. The biggest change was removal of PmaAbsoluteUri configuration directive, which has caused quite some pain in past and is not really needed these days (when browsers support relative paths in the Location HTTP header). This lead to cleanup in other parts as well - support for dead Mozilla Prism is gone, used HTTPS for OpenStreetMap tiles (the map layer now works on HTTPS as well), removed ForceSSL configuration directive as this is something what really needs to be handled at web server level. To improve test coverage, several tests no longer require runkit as the header() call is wrapped within Response class and can be overridden for testing without using runkit. The list of handled issues is not that impressive this week:

Filed under: English phpMyAdmin 0 comments

7 February 2016

Steve Kemp: Redesigning my clustered website

I'm slowly planning the redesign of the cluster which powers the Debian Administration website. Currently the design is simple, and looks like this:
In brief there is a load-balancer that handles SSL-termination and then proxies to one of four Apache servers. These talk back and forth to a MySQL database. Nothing too shocking, or unusual. (In truth there are two database servers, and rather than a single installation of HAProxy it runs upon each of the webservers - One is the master which is handled via ucarp. Logically though traffic routes through HAProxy to a number of Apache instances. I can lose half of the servers and things still keep running.) When I setup the site it all ran on one host, it was simpler, it was less highly available. It also struggled to cope with the load. Half the reason for writing/hosting the site in the first place was to document learning experiences though, so when it came to time to make it scale I figured why not learn something and do it neatly? Having it run on cheap and reliable virtual hosts was a good excuse to bump the server-count and the design has been stable for the past few years. Recently though I've begun planning how it will be deployed in the future and I have a new design:
Rather than having the Apache instances talk to the database I'll indirect through an API-server. The API server will handle requests like these:
  • POST /users/login
    • POST a username/password and return 200 if valid. If bogus details return 403. If the user doesn't exist return 404.
  • GET /users/Steve
    • Return a JSON hash of user-information.
    • Return 404 on invalid user.
I expect to have four API handler endpoints: /articles, /comments, /users & /weblogs. Again we'll use a floating IP and a HAProxy instance to route to multiple API-servers. Each of which will use local caching to cache articles, etc. This should turn the middle layer, running on Apache, into simpler things, and increase throughput. I suspect, but haven't confirmed, that making a single HTTP-request to fetch a (formatted) article body will be cheaper than making N-database queries. Anyway that's what I'm slowly pondering and working on at the moment. I wrote a proof of concept API-server based CMS two years ago, and my recollection of that time is that it was fast to develop, and easy to scale.

31 January 2016

Chris Lamb: Free software activities in January 2016

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):
  • Changed Django's project/app templates to use a py-tpl suffix to workaround the 1.9 release series shipping invalid .py files that are used as templates. (#5735)
  • Pushed a number of updates to my Strava Enhancement Suite Chrome extension:
    • Added the ability to sort starred segments first. (#42)
    • Added an option to display hidden segments by default. (#34)
    • Also hide "Yearly Goals" as part of hiding upcoming data. (#41)
    • Removed more "premium" badges. (#43)
    • Fixed an issue where removing feed entries didn't correctly cleanup subsequently-empty date headers. (commit)
  • Released a work-in-progress django.contrib.staticfiles library to recursively transparently concatenate Javascript and CSS files using "Debian-style" .d directories. I had previously been doing this manually via the various release processes and bespoke views during development as the idea pre-dated the existence of the staticfiles framework. (Repo)
  • Updated travis.debian.net, a hosted script to easily test and build Debian packages on Travis CI, to support the wheezy distribution and to improve error-handling in general. (Repo)
  • Ensured that try.diffoscope.org, a hosted version of the diffoscope in-depth and content aware diff utility, periodically cleans up containers.
  • Moved my personal music library to use dh-virtualenv and to deploy to its own server via Ansible. I also updated the codebase to the latest version of Django at the same time, taking advantage of a large number of new features and recommendations.
  • Fixed a strange issue with empty IMAP messages in my tickle-me-email GTD email toolbox.
  • It was also a month of significant bug reports and feature requests being sent to my private email.
Debian
  • Had a talk proposal accepted (Reproducible Builds - fulfilling the original promise of free software) at FOSSASIA 16.
My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#35, #36, #37, #38, #39)
LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:
  • Sevend days of "frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 386-1 for cacti to patch an SQL injection vulnerability.
  • Issued DLA 388-1 for dwarfutils fixing a NULL deference issue.
  • Issued DLA 391-1 for prosody correcting the use of a weak pseudo-random number generator.
  • Issued DLA 404-1 for nginx to prevent against an invalid pointer deference.
Uploads
  • redis (2:3.0.7-1) New upstream stable release, also ensure that test processes are cleaned up and replacing an existing reproducibility patch with a SOURCE_DATE_EPOCH solution.
  • python-django (1.9.1-1) New upstream release.
  • disque (1.0~rc1-4) Make the build reproducible via SOURCE_DATE_EPOCH, ensure that test processes are cleaned up and that the nocheck flag is correctly honoured.
  • gunicorn (19.4.5-1) New upstream release.
  • redis (2:3.2~rc3-1) New upstream RC release (to experimental).
RC bugs


I also filed 100 FTBFS bugs against apache-log4j2, awscli, binutils, brian, ccbuild, coala, commons-beanutils, commons-vfs, composer, cyrus-sasl2, debiandoc-sgml-doc-pt-br, dfvfs, dillo, django-compat, dulwich, git-annex, grpc, hdf-eos5, hovercraft, ideviceinstaller, ircp-tray, isomd5sum, javamail, jhdf, jsonpickle, kivy, klog, libcloud, libcommons-jexl2-java, libdata-objectdriver-perl, libdbd-sqlite3-perl, libpam-krb5, libproc-waitstat-perl, libslf4j-java, libvmime, linuxdcpp, lsh-utils, mailutils, mdp, menulibre, mercurial, mimeo, molds, mugshot, nose, obex-data-server, obexfs, obexftp, orafce, p4vasp, pa-test, pgespresso, pgpool2, pgsql-asn1oid, php-doctrine-cache-bundle, php-net-ldap2, plv8, pngtools, postgresql-mysql-fdw, pyfftw, pylint-common, pylint-django, pylint-django, python-ase, python-axiom, python-biopython, python-dcos, python-falcon, python-instagram, python-markdown, python-pysam, python-requests-toolbelt, python-ruffus, pytsk, pyviennacl, ros-class-loader, ros-ros-comm, ros-roscpp-core, roxterm, ruby-celluloid-extras, ruby-celluloid-fsm, ruby-celluloid-supervision, ruby-eye, ruby-net-scp, ruby-net-ssh, ruby-sidekiq, ruby-sidekiq-cron, ruby-sinatra-contrib, seaview, smc, spatial4j-0.4, swift-plugin-s3, tilecache, typecatcher, ucommon, undertaker, urdfdom, ussp-push, xserver-xorg-video-intel & yt.
FTP Team

As a Debian FTP assistant I ACCEPTed 201 packages: abi-tracker, android-platform-build, android-platform-frameworks-native, android-platform-libcore, android-platform-system-core, animate.css, apitrace, argon2, autosize.js, bagel, betamax, bittorrent, bls-standalone, btfs, caja-dropbox, cegui-mk2, complexity, corebird, courier-authlib, cpopen, ctop, dh-haskell, django-python3-ldap, e2fsprogs1.41, emacs-async, epl, fast5, fastkml, flask-restful, flask-silk, gcc-6, gitlab, golang-github-kolo-xmlrpc, golang-github-kr-fs, golang-github-pkg-sftp, golang-github-prometheus-common, google-auth-library-php, h5py, haskell-aeson-compat, haskell-userid, heroes, hugo, ioprocess, iptables, ivy-debian-helper, ivyplusplus, jquery-timer.js, klaus, kpatch, lazarus, libatteanx-store-sparql-perl, libbrowserlauncher-java, libcgi-test-perl, libdata-sah-normalize-perl, libfsntfs, libjs-fuzzaldrin-plus, libjung-free-java, libmongoc, libmygpo-qt, libnet-nessus-rest-perl, liborcus, libperinci-sub-util-propertymodule-perl, libpodofo, librep, libsodium, libx11-xcb-perl, linux, linux-grsec-base, list.js, lombok, lua-mediator, luajit, maven-script-interpreter, midicsv, mimeo, miniasm, mlpack, mom, mosquitto-auth-plugin, moxie.js, msgpuck, nanopolish, neovim, netcdf, network-manager-applet, network-manager-ssh, node-esprima-fb, node-mocks-http, node-schlock, nomacs, ns3, openalpr, openimageio, openmpi, openms, orafce, pbsim, pd-iemutils, pd-nusmuk, pd-puremapping, pd-purest-json, pg-partman, pg-rage-terminator, pgfincore, pgmemcache, pgsql-asn1oid, php-defaults, php-jwt, php-mf2, php-redis, pkg-info-el, plr, pnmixer, postgresql-multicorn, postgresql-mysql-fdw, powa-archivist, previsat, pylint-flask, pyotherside, python-caldav, python-cookies, python-dcos, python-flaky, python-flickrapi, python-frozendict, python-genty, python-git, python-greenlet, python-instagram, python-ironic-inspector-client, python-manilaclient, python-neutronclient, python-openstackclient, python-openstackdocstheme, python-prometheus-client, python-pymzml, python-pysolr, python-reno, python-requests-toolbelt, python-scales, python-socketio-client, qdox2, qgis, r-cran-biasedurn, rebar.js, repmgr, rfcdiff, rhythmbox-plugin-alternative-toolbar, ripe-atlas-cousteau, ripe-atlas-sagan, ripe-atlas-tools, ros-image-common, ruby-acts-as-list, ruby-allocations, ruby-appraiser, ruby-appraiser-reek, ruby-appraiser-rubocop, ruby-babosa, ruby-combustion, ruby-did-you-mean, ruby-fixwhich, ruby-fog-xenserver, ruby-hamster, ruby-jeweler, ruby-mime-types-data, ruby-monkey-lib, ruby-net-telnet, ruby-omniauth-azure-oauth2, ruby-omniauth-cas3, ruby-puppet-forge, ruby-racc, ruby-reek, ruby-rubinius-debugger, ruby-rubysl, ruby-rubysl-test-unit, ruby-sidekiq-cron, ruby-threach, ruby-wavefile, ruby-websocket-driver, ruby-xmlhash, rustc, s-nail, scrm, select2.js, senlin, skytools3, slurm-llnl, sphinx-argparse, sptk, sunpy, swauth, swift, tdiary, three.js, tiny-initramfs, tlsh, ublock-origin, vagrant-cachier, xapian-core, xmltooling, & yp-tools. I additionally REJECTed 29 packages.

30 January 2016

Simon McVittie: GNOME Developer Experience hackfest: xdg-app + Debian

Over the last few days I've been at the GNOME Developer Experience hackfest in Brussels, looking into xdg-app and how best to use it in Debian and Debian derivatives. xdg-app is basically a way to run "non-core" software on Linux distributions, analogous to apps on Android and iOS. It doesn't replace distributions like Debian or packaging systems, but it adds a layer above them. It's mostly aimed towards third-party apps obtained from somewhere that isn't your distribution vendor, aiming to address a few long-standing problems in that space:
  • There's no single ABI that can be called "a standard Linux system" in the same way there would be for Windows or OS X or Android or whatever, apart from LSB which is rather limited. Testing that a third-party app "works on Linux", or even "works on stable Linux releases from 2015", involves a combinatorial explosion of different distributions, desktop environments and local configurations. Steam uses the Steam Runtime, a chroot environment closely resembling Ubuntu 12.04 LTS; other vendors tend to test on a vaguely recent Ubuntu LTS and leave it at that.
  • There's no widely-supported mechanism for installing third-party applications as an ordinary user. gog.com used to distribute Ubuntu- and Debian-compatible .deb files, but installing a .deb involves running arbitrary vendor-supplied scripts as root, which should worry anyone who wants any sort of privilege-separation. (They have now switched to executable self-extracting installers, which involve running arbitrary vendor-supplied scripts as an ordinary user... better, but not perfect.)
  • Relatedly, the third-party application itself runs with the user's full privileges: a malicious or security-buggy third-party application can do more or less anything, unless you either switch to a different uid to run third-party apps, or use a carefully-written, app-specific AppArmor profile or equivalent.
To address the first point, each application uses a specified "runtime", which is available as /usr inside its sandbox. This can be used to run application bundles with multiple, potentially incompatible sets of dependencies within the same desktop environment. A runtime can be updated within its branch - for instance, if an application uses the "GNOME 3.18" runtime (consisting of a basic Linux system, the GNOME 3.18 libraries, other related libraries like Mesa, and their recursive dependencies like libjpeg), it can expect to see minor-version updates from GNOME 3.18.x (including any security updates that might be necessary for the bundled libraries), but not a jump to GNOME 3.20. To address the second issue, the plan is for application bundles to be available as a single file, containing metadata (such as the runtime to use), the app itself, and any dependencies that are not available in the runtime (which the app vendor is responsible for updating if necessary). However, the primary way to distribute and upgrade runtimes and applications is to package them as OSTree repositories, which provide a git-like content-addressed filesystem, with efficient updates using binary deltas. The resulting files are hard-linked into place. To address the last point, application bundles run partially isolated from the wider system, using containerization techniques such as namespaces to prevent direct access to system resources. Resources from outside the sandbox can be accessed via "portal" services, which are responsible for access control; for example, the Documents portal (the only one, so far) displays an "Open" dialog outside the sandbox, then allows the application to access only the selected file. xdg-app for Debian One thing I've been doing at this hackfest is improving the existing Debian/Ubuntu packaging for xdg-app (and its dependencies ostree and libgsystem), aiming to get it into a state where I can upload it to Debian experimental. Because xdg-app aims to be a general freedesktop project, I'm currently intending to make it part of the "Utopia" packaging team alongside projects like D-Bus and polkit, but I'm open to suggestions if people want to co-maintain it elsewhere. In the process of updating xdg-app, I sent various patches to Alex, mostly fixing build and test issues, which are in the new 0.4.8 release. I'd appreciate co-maintainers and further testing for this stuff, particularly ostree: ostree is primarily a whole-OS deployment technology, which isn't a use-case that I've tested, and in particular ostree-grub2 probably doesn't work yet. Source code: Binaries (no trust path, so only use these if you have a test VM):
  • deb https://people.debian.org/~smcv/xdg-app xdg-app main
The "Hello, World" of xdg-apps Another thing I set out to do here was to make a runtime and an app out of Debian packages. Most of the test applications in and around GNOME use the "freedesktop" or "GNOME" runtimes, which consist of a Yocto base system and lots of RPMs, are rebuilt from first principles on-demand, and are extensive and capable enough that they make it somewhat non-obvious what's in an app or a runtime. So, here's a step-by-step route through xdg-app, first using typical GNOME instructions, but then using the simplest GUI app I could find - xvt, a small xterm clone. I'm using a Debian testing (stretch) x86_64 virtual machine for all this. xdg-app currently requires systemd-logind to put users and apps in cgroups, either with systemd as pid 1 (systemd-sysv) or systemd-shim and cgmanager; I used the default systemd-sysv. In principle it could work with plain cgmanager, but nobody has contributed that support yet. Demonstrating an existing xdg-app Debian's kernel is currently patched to be able to allow unprivileged users to create user namespaces, but make it runtime-configurable, because there have been various security issues in that feature, making it a security risk for a typical machine (and particularly a server). Hopefully unprivileged user namespaces will soon be secure enough that we can enable them by default, but for now, we have to do one of three things to let xdg-app use them:
  • enable unprivileged user namespaces via sysctl: 
    sudo sysctl kernel.unprivileged_userns_clone=1
  • make xdg-app root-privileged (it will keep CAP_SYS_ADMIN and drop the rest): 
    sudo dpkg-statoverride --update --add root root 04755 /usr/bin/xdg-app-helper
  • make xdg-app slightly less privileged: 
    sudo setcap cap_sys_admin+ep /usr/bin/xdg-app-helper
First, we'll need a runtime. The standard xdg-app tutorial would tell you to download the "GNOME Platform" version 3.18. To do that, you'd add a remote, which is a bit like a git remote, and a bit like an apt repository: 
$ wget http://sdk.gnome.org/keys/gnome-sdk.gpg
$ xdg-app remote-add --user --gpg-import=gnome-sdk.gpg gnome \
    http://sdk.gnome.org/repo/
(I'm ignoring considerations like trust paths and security here, for brevity; in real life, you'd want to obtain the signing key via https and/or have a trust path to it, just like you would for a secure-apt signing key.) You can list what's available in a remote: 
$ xdg-app remote-ls --user gnome
...
org.freedesktop.Platform
...
org.freedesktop.Platform.Locale.cy
...
org.freedesktop.Sdk
...
org.gnome.Platform
...
The Platform runtimes are what we want here: they are collections of runtime libraries with which you can run an application. The Sdk runtimes add development tools, header files, etc. to be able to compile apps that will be compatible with the Platform. For now, all we want is the GNOME 3.18 platform: 
$ xdg-app install --user gnome org.gnome.Platform 3.18
Next, we can install an app that uses it, from Alex Larsson's nightly builds of a subset of GNOME. The server they're on doesn't have a great deal of bandwidth, so be nice :-) 
$ wget http://209.132.179.2/keys/nightly.gpg
$ xdg-app remote-add --user --gpg-import=nightly.gpg nightly \
    http://209.132.179.2/repo/
$ xdg-app install --user nightly org.mypaint.MypaintDevel
We now have one app, and the runtime it needs: 
$ xdg-app list
org.mypaint.MypaintDevel
$ xdg-app run org.mypaint.MypaintDevel
[you see a GUI window]
Digression: what's in a runtime? Behind the scenes, xdg-app runtimes and apps are both OSTree trees. This means the ostree tool, from the package of the same name, can be used to inspect them. 
$ sudo apt install ostree
$ ostree refs --repo ~/.local/share/xdg-app/repo
gnome:runtime/org.gnome.Platform/x86_64/3.18
nightly:app/org.mypaint.MypaintDevel/x86_64/master
A "ref" has roughly the same meaning as in git: something like a branch or a tag. ostree can list the directory tree that it represents: 
$ ostree ls --repo ~/.local/share/xdg-app/repo \
    runtime/org.gnome.Platform/x86_64/3.18
d00755 0 0      0 /
-00644 0 0    493 /metadata
d00755 0 0      0 /files
$ ostree ls --repo ~/.local/share/xdg-app/repo \
    runtime/org.gnome.Platform/x86_64/3.18 /files
d00755 0 0      0 /files
l00777 0 0      0 /files/local -> ../var/usrlocal
l00777 0 0      0 /files/sbin -> bin
d00755 0 0      0 /files/bin
d00755 0 0      0 /files/cache
d00755 0 0      0 /files/etc
d00755 0 0      0 /files/games
d00755 0 0      0 /files/include
d00755 0 0      0 /files/lib
d00755 0 0      0 /files/lib64
d00755 0 0      0 /files/libexec
d00755 0 0      0 /files/share
d00755 0 0      0 /files/src
You can see that /files in a runtime is basically a copy of /usr. This is not coincidental: the runtime's /files gets mounted at /usr inside the xdg-app container. There is also some metadata, which is in the ini-like syntax seen in .desktop files: 
$ ostree cat --repo ~/.local/share/xdg-app/repo \
    runtime/org.gnome.Platform/x86_64/3.18 /metadata
[Runtime]
name=org.gnome.Platform/x86_64/3.16
runtime=org.gnome.Platform/x86_64/3.16
sdk=org.gnome.Sdk/x86_64/3.16
[Extension org.freedesktop.Platform.GL]
version=1.2
directory=lib/GL
[Extension org.freedesktop.Platform.Timezones]
version=1.2
directory=share/zoneinfo
[Extension org.gnome.Platform.Locale]
directory=share/runtime/locale
subdirectories=true
[Environment]
GI_TYPELIB_PATH=/app/lib/girepository-1.0
GST_PLUGIN_PATH=/app/lib/gstreamer-1.0
LD_LIBRARY_PATH=/app/lib:/usr/lib/GL
Looking at an app, the situation is fairly similar: 
$ ostree ls --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master
d00755 0 0      0 /
-00644 0 0    258 /metadata
d00755 0 0      0 /export
d00755 0 0      0 /files
This time, /files maps to what will become /app for the application, which was compiled with --prefix=/app: 
$ ostree ls --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master /files
d00755 0 0      0 /files
-00644 0 0   4599 /files/manifest.json
d00755 0 0      0 /files/bin
d00755 0 0      0 /files/lib
d00755 0 0      0 /files/share
There is also a /export directory, which is made visible to the host system so that the contained app can appear as a "first-class citizen" in menus: 
$ ostree ls --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master /export
d00755 0 0      0 /export
d00755 0 0      0 /export/share
user@debian:~$ ostree ls --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master /export/share
d00755 0 0      0 /export/share
d00755 0 0      0 /export/share/app-info
d00755 0 0      0 /export/share/applications
d00755 0 0      0 /export/share/icons
user@debian:~$ ostree ls --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master /export/share/applications
d00755 0 0      0 /export/share/applications
-00644 0 0    715 /export/share/applications/org.mypaint.MypaintDevel.desktop
$ ostree cat --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master \
    /export/share/applications/org.mypaint.MypaintDevel.desktop
[Desktop Entry]
Version=1.0
Name=(Nightly) MyPaint
TryExec=mypaint
Exec=mypaint %f
Comment=Painting program for digital artists
...
Comment[zh_HK]= 
GenericName=Raster Graphics Editor
GenericName[fr]= diteur d'Image Matricielle
MimeType=image/openraster;image/png;image/jpeg;
Type=Application
Icon=org.mypaint.MypaintDevel
StartupNotify=true
Categories=Graphics;GTK;2DGraphics;RasterGraphics;
Terminal=false
Again, there's some metadata: 
$ ostree cat --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master /metadata
[Application]
name=org.mypaint.MypaintDevel
runtime=org.gnome.Platform/x86_64/3.18
sdk=org.gnome.Sdk/x86_64/3.18
command=mypaint
[Context]
shared=ipc;
sockets=x11;pulseaudio;
filesystems=host;
[Extension org.mypaint.MypaintDevel.Debug]
directory=lib/debug
Building a runtime, probably the wrong way The way in which the reference/demo runtimes and containers are generated is... involved. As far as I can tell, there's a base OS built using Yocto, and the actual GNOME bits come from RPMs. However, we don't need to go that far to get a working runtime. In preparing this runtime I'm probably completely ignoring some best-practices and tools - but it works, so it's good enough. First we'll need a repository: 
$ sudo install -d -o$(id -nu) /srv/xdg-apps
$ ostree init --repo /srv/xdg-apps
I'm just keeping this local for this demonstration, but you could rsync it to a web server's exported directory or something - a lot like a git repository, it's just a collection of files. We want everything in /usr because that's what xdg-app expects, hence usrmerge: 
$ sudo mount -t tmpfs -o mode=0755 tmpfs /mnt
$ sudo debootstrap --arch=amd64 --include=libx11-6,usrmerge \
    --variant=minbase stretch /mnt http://192.168.122.1:3142/debian
$ sudo mkdir /mnt/runtime
$ sudo mv /mnt/usr /mnt/runtime/files
This obviously has a lot of stuff in it that we don't need - most obviously init, apt and dpkg - but it's Good Enough . We will also need some metadata. This is sufficient: 
$ sudo sh -c 'cat > /mnt/runtime/metadata'
[Runtime]
name=org.debian.Debootstrap/x86_64/8.20160130
runtime=org.debian.Debootstrap/x86_64/8.20160130
That's a runtime. We can commit it to ostree, and generate xdg-app metadata: 
$ ostree commit --repo /srv/xdg-apps \
    --branch runtime/org.debian.Debootstrap/x86_64/8.20160130 \
    /mnt/runtime
$ fakeroot ostree commit --repo /srv/xdg-apps \
    --branch runtime/org.debian.Debootstrap/x86_64/8.20160130
$ fakeroot xdg-app build-update-repo /srv/xdg-apps
(I'm not sure why ostree and xdg-app report "Operation not permitted" when we aren't root or fakeroot - feedback welcome.) build-update-repo would presumably also be the right place to GPG-sign your repository, if you were doing that. We can add that as another xdg-app remote: 
$ xdg-app remote-add --user --no-gpg-verify local file:///srv/xdg-apps
$ xdg-app remote-ls --user local
org.debian.Debootstrap
Building an app, probably the wrong way The right way to build an app is to build a "SDK" runtime - similar to that platform runtime, but with development files and tools - and recompile the app and any missing libraries with ./configure --prefix=/app && make && make install. I'm not going to do that, because simplicity is nice, and I'm reasonably sure xvt doesn't actually hard-code /usr into the binary: 
$ install -d xvt-app/files/bin
$ sudo apt-get --download-only install xvt
$ dpkg-deb --fsys-tarfile /var/cache/apt/archives/xvt_2.1-20.1_amd64.deb \
      tar -xvf - ./usr/bin/xvt
./usr/
./usr/bin/
./usr/bin/xvt
...
$ mv usr xvt-app/files
Again, we'll need metadata, and it's much simpler than the more production-quality GNOME nightly builds: 
$ cat > xvt-app/metadata
[Application]
name=org.debian.packages.xvt
runtime=org.debian.Debootstrap/x86_64/8.20160130
command=xvt
[Context]
sockets=x11;
$ fakeroot ostree commit --repo /srv/xdg-apps \
    --branch app/org.debian.packages.xvt/x86_64/2.1-20.1 xvt-app
$ fakeroot xdg-app build-update-repo /srv/xdg-apps
Updating appstream branch
No appstream data for runtime/org.debian.Debootstrap/x86_64/8.20160130
No appstream data for app/org.debian.packages.xvt/x86_64/2.1-20.1
Updating summary
$ xdg-app remote-ls --user local
org.debian.Debootstrap
org.debian.packages.xvt
The obligatory screenshot OK, good, now we can install it: 
$ xdg-app install --user local org.debian.Debootstrap 8.20160130
$ xdg-app install --user local org.debian.packages.xvt 2.1-20.1
$ xdg-app run --branch=2.1-20.1 org.debian.packages.xvt
xvt in a container and you can play around with the shell in the xvt and see what you can and can't do in the container. I'm sure there were better ways to do most of this, but I think there's value in having such a simplistic demo to go alongside the various GNOMEish apps.
Acknowledgements: Thanks to all those!

Next.

Previous.