Search Results: "pm"

17 September 2015

Julien Danjou: My interview in le Journal du Hacker

A few days ago, the French equivalent of Hacker News, called "Le Journal du Hacker", interviewed me about my work on OpenStack, my job at Red Hat and my self-published book The Hacker's Guide to Python. I've spent some time translating it into English so you can read it if you don't understand French! I hope you'll enjoy it.
Hi Julien, and thanks for participating in this interview for the Journal du Hacker. For our readers who don't know you, can you introduce you briefly?
You're welcome! My name is Julien, I'm 31 years old, and I live in Paris. I now have been developing free software for around fifteen years. I had the pleasure to work (among other things) on Debian, Emacs and awesome these last years, and more recently on OpenStack. Since a few months now, I work at Red Hat, as a Principal Software Engineer on OpenStack. I am in charge of doing upstream development for that cloud-computing platform, mainly around the Ceilometer, Aodh and Gnocchi projects.
Being myself a system architect, I follow your work in OpenStack since a while. It's uncommon to have the point of view of someone as implied as you are. Can you give us a summary of the state of the project, and then detail your activities in this project?
The OpenStack project has grown and changed a lot since I started 4 years ago. It started as a few projects providing the basics, like Nova (compute), Swift (object storage), Cinder (volume), Keystone (identity) or even Neutron (network) who are basis for a cloud-computing platform, and finally became composed of a lot more projects. For a while, the inclusion of projects was the subject of a strict review from the technical committee. But since a few months, the rules have been relaxed, and we see a lot more projects connected to cloud-computing joining us. As far as I'm concerned, I've started with a few others people the Ceilometer project in 2012, devoted to handling metrics of OpenStack platforms. Our goal is to be able to collect all the metrics and record them to analyze them later. We also have a module providing the ability to trigger actions on threshold crossing (alarm). The project grew in a monolithic way, and in a linear way for the number of contributors, during the first two years. I was the PTL (Project Technical Leader) for a year. This leader position asks for a lot of time for bureaucratic things and people management, so I decided to leave my spot in order to be able to spend more time solving the technical challenges that Ceilometer offered. I've started the Gnocchi project in 2014. The first stable version (1.0.0) was released a few months ago. It's a timeseries database offering a REST API and a strong ability to scale. It was a necessary development to solve the problems tied to the large amount of metrics created by a cloud-computing platform, where tens of thousands of virtual machines have to be metered as often as possible. This project works as a standalone deployment or with the rest of OpenStack. More recently, I've started Aodh, the result of moving out the code and features of Ceilometer related to threshold action triggering (alarming). That's the logical suite to what we started with Gnocchi. It means Ceilometer is to be split into independent modules that can work together with or without OpenStack. It seems to me that the features provided by Ceilometer, Aodh and Gnocchi can also be interesting for operators running more classical infrastructures. That's why I've pushed the projects into that direction, and also to have a more service-oriented architecture (SOA)
I'd like to stop for a moment on Ceilometer. I think that this solution was very expected, especially by the cloud-computing providers using OpenStack for billing resources sold to their customers. I remember reading a blog post where you were talking about the high-speed construction of this brick, and features that were not supposed to be there. Nowadays, with Gnocchi and Aodh, what is the quality of the brick Ceilometer and the programs it relies on?
Indeed, one of the first use-case for Ceilometer was tied to the ability to get metrics to feed a billing tool. That's now a reached goal since we have billing tools for OpenStack using Ceilometer, such as CloudKitty. However, other use-cases appeared rapidly, such as the ability to trigger alarms. This feature was necessary, for example, to implement the auto-scaling feature that Heat needed. At the time, for technical and political reasons, it was not possible to implement this feature in a new project, and the functionality ended up in Ceilometer, since it was using the metrics collected and stored by Ceilometer itself. Though, like I said, this feature is now in its own project, Aodh. The alarm feature is used since a few cycles in production, and the Aodh project brings new features on the table. It allows to trigger threshold actions and is one of the few solutions able to work at high scale with several thousands of alarms. It's impossible to make Nagios run with millions of instances to fetch metrics and triggers alarms. Ceilometer and Aodh can do that easily on a few tens of nodes automatically. On the other side, Ceilometer has been for a long time painted as slow and complicated to use, because its metrics storage system was by default using MongoDB. Clearly, the data structure model picked was not optimal for what the users were doing with the data. That's why I started Gnocchi last year, which is perfectly designed for this use case. It allows linear access time to metrics (O(1) complexity) and fast access time to the resources data via an index. Today, with 3 projects having their own perimeter of features defined and which can work together Ceilometer, Aodh and Gnocchi finally erased the biggest problems and defects of the initial project.
To end with OpenStack, one last question. You're a Python developer for a long time and a fervent user of software testing and test-driven development. Several of your blogs posts point how important their usage are. Can you tell us more about the usage of tests in OpenStack, and the test prerequisites to contribute to OpenStack?
I don't know any project that is as tested on every layer as OpenStack is. At the start of the project, there was a vague test coverage, made of a few unit tests. For each release, a bunch of new features were provided, and you had to keep your fingers crossed to have them working. That's already almost unacceptable. But the big issue was that there was also a lot of regressions, et things that were working were not anymore. It was often corner cases that developers forgot about that stopped working. Then the project decided to change its policy and started to refuse all patches new features or bug fix that would not implement a minimal set of unit tests, proving the patch would work. Quickly, regressions were history, and the number of bugs largely reduced months after months. Then came the functional tests, with the Tempest project, which runs a test battery on a complete OpenStack deployment. OpenStack now possesses a complete test infrastructure, with operators hired full-time to maintain them. The developers have to write the test, and the operators maintain an architecture based on Gerrit, Zuul, and Jenkins, which runs the test battery of each project for each patch sent. Indeed, for each version of a patch sent, a full OpenStack is deployed into a virtual machine, and a battery of thousands of unit and functional tests is run to check that no regressions are possible. To contribute to OpenStack, you need to know how to write a unit test the policy on functional tests is laxer. The tools used are standard Python tools, unittest for the framework and tox to run a virtual environment (venv) and run them. It's also possible to use DevStack to deploy an OpenStack platform on a virtual machine and run functional tests. However, since the project infrastructure also do that when a patch is submitted, it's not mandatory to do that yourself locally.
The tools and tests you write for OpenStack are written in Python, a language which is very popular today. You seem to like it more than you have to, since you wrote a book about it, The Hacker's Guide to Python, that I really enjoyed. Can you explain what brought you to Python, the main strong points you attribute to this language (quickly) and how you went from developer to author?
I stumbled upon Python by chance, around 2005. I don't remember how I hear about it, but I bought a first book to discover it and started toying with that language. At that time, I didn't find any project to contribute to or to start. My first project with Python was rebuildd for Debian in 2007, a bit later. I like Python for its simplicity, its object orientation rather clean, its easiness to be deployed and its rich open source ecosystem. Once you get the basics, it's very easy to evolve and to use it for anything, because the ecosystem makes it easy to find libraries to solve any kind of problem. I became an author by chance, writing blog posts from time to time about Python. I finally realized that after a few years studying Python internals (CPython), I learned a lot of things. While writing a post about the differences between method types in Python which is still one of the most read post on my blog I realized that a lot of things that seemed obvious to me where not for other developers. I wrote that initial post after thousands of hours spent doing code reviews on OpenStack. I, therefore, decided to note all the developers pain points and to write a book about that. A compilation of what years of experience taught me and taught to the other developers I decided to interview in the book.
I've been very interested by the publication of your book, for the subject itself, but also the process you chose. You self-published the book, which seems very relevant nowadays. Is that a choice from the start? Did you look for an editor? Can you tell use more about that?
I've been lucky to find out about others self-published authors, such as Nathan Barry who even wrote a book on that subject, called Authority. That's what convinced me it was possible and gave me hints for that project. I've started to write in August 2013, and I ran the firs interviews with other developers at that time. I started to write the table of contents and then filled the pages with what I knew and what I wanted to share. I manage to finish the book around January 2014. The proof-reading took more time than I expected, so the book was only released in March 2014. I wrote a complete report on that on my blog, where I explain the full process in detail, from writing to launching. I did not look for editors though I've been proposed some. The idea of self-publishing really convince me, so I decided to go on my own, and I have no regret. It's true that you have to wear two hats at the same time and handle a lot more things, but with a minimal audience and some help from the Internet, anything's possible! I've been reached by two editors since then, a Chinese and Korean one. I gave them rights to translate and publish the books in their countries, so you can buy the Chinese and Korean version of the first edition of the book out there. Seeing how successful it was, I decided to launch a second edition in Mai 2015, and it's likely that a third edition will be released in 2016.
Nowadays, you work for Red Hat, a company that represents the success of using Free Software as a commercial business model. This company fascinates a lot in our community. What can you say about your employer from your point of view?
It only has been a year since I joined Red Hat (when they bought eNovance), so my experience is quite recent. Though, Red Hat is really a special company on every level. It's hard to see from the outside how open it is, and how it works. It's really close to and it really looks like an open source project. For more details, you should read The Open Organization, a book wrote by Jim Whitehurst (CEO of Red Hat), which he just published. It describes perfectly how Red Hat works. To summarize, meritocracy and the lack of organization in silos is what makes Red Hat a strong organization and puts them as one of the most innovative company. In the end, I'm lucky enough to be autonomous for the project I work on with my team around OpenStack, and I can spend 100% working upstream and enhance the Python ecosystem.

Norbert Preining: Debian/TeX Live 2015.20150917-1

Usual regular update of the TeX Live packages. Now that Debian/unstable is back to full development speed thanks to the end of the bit gcc transition, probably this TeX Live update will be lost between the hundreds of others. Anyway a few interesting changes are still in there. Debian - TeX Live 2015 Most importantly the fix for unicode-math that was broken for a short time, and for the Japanese users it seems that the LaTeX3 packages have gained support for upTeX. Good to hear. Of course, these are not the only updates and newcomers, see the list below. Updated packages abc, acro, alegreya, animate, arabxetex, babel-bosnian, babel-french, babel-greek, babel-latin, babel-welsh, beamer-FUBerlin, bidi, bigfoot, blochsphere, bxjscls, chemformula, chemmacros, chet, classicthesis, crossrefware, csplain, ctable, ctanify, datetime2, datetime2-basque, detlev-cm, disser, doclicense, drm, dtxgen, dtxtut, dvips, ecclesiastic, eledmac, embrac, etex-pkg, etoc, factura, fbb, fithesis, font-change, fontname, glossaries, gothic, greek-fontenc, IEEEtran, l3build, l3experimental, l3kernel, l3packages, latexconfig, lettrine, luatexja, mathastext, mcf2graph, media9, metrix, mhequ, minted, moderntimeline, newtx, ocgx2, pagecolor, pdftex, pgf, pstricks, reledmac, resphilosophica, roboto, shdoc, siunitx, suftesi, tetex, tex4ht, texlive-scripts, tikz-bayesnet, translations, ucharcat, udesoftec, ulthese, unicode-math, ut-thesis, withargs, xint. New packages checklistings, cleanthesis, easyreview, fei, medstarbeamer, mfirstuc, nevelok, old-arrows, proofread, uassign, xebaposter. Enjoy.

15 September 2015

Norbert Preining: In Memoriam Adrian Frutiger

Just a few days ago another great font designer has passed away, Adrian Frutiger, 24 May 1928 10 September 2015. Famous in particular for his fonts Frutiger, Avenir, and the Univers family. His fonts are used throughout the worlds for visual communication like road signs. Much has been written about him: Linotype s excellent obituary, Adam Twardoch, David Airey, New York Times article. For an interview conducted with Frutiger in 1999, see this Eye Magazine article. An interesting article on Frutiger s typefaces by Charles Bigelow.
frutiger1 Here I want to recall a different quality of Adrian Frutiger his interest in the visual language of symbols, their development and interaction. His book Der Mensch und seine Zeichen (Signs and Symbols) is a profound study on the development, history, and use of all kind of symbols. In this book, translated into many languages, Frutiger explores the depth and breath of symbols, but the most important part for him, easily to be seen from the German title, is the human part ( Der Mensch ). Symbols are created, changed, and used by and for humans. His studies exhibit connections between various cultures when it comes to sign usage and design.
frutiger5 He also explores signets as modern sign language, and their importance for visual communcation and identity building.
frutiger4 The book also discusses the development of the Roman alphabet, and of course passes the Univers font family.
frutiger2 My favorite piece is this overlay of a s from different fonts, to show that there is a common pattern in these fonts.
frutiger3 He concludes this book with a very wise statement:
Zum Festhalten des Gedankens, zur Vermittlung der Aussage gen gen seit langem schon die Alphabet-Zeichen nicht mehr allein. Orientierung und Kommunikation sind heute unm glich ohne Schemata, Zeichen und Signale. Der geschriebene Ausdruck wird durch die Bildvermittlung notwendig erg nzt.
SInce long the alphabet letters alone do not suffice to take hold of thoughts and to transmit statements. Nowadays, orientation and communication are impossible without schemata, signs and signals. The written expression is necessarily enhanced through image transmission.
(my translation) After Hermann Zapf, this is now the second loss of the typographic world in this year. We will miss both of them.

14 September 2015

Julien Danjou: Visualize your OpenStack cloud: Gnocchi & Grafana

We've been hard working with the Gnocchi team these last months to store your metrics, and I guess it's time to show off a bit. So far Gnocchi offers scalable metric storage and resource indexation, especially for OpenStack cloud but not only, we're generic. It's cool to store metrics, but it can be even better to have a way to visualize them! Prototyping We very soon started to build a little HTML interface. Being REST-friendly guys, we enabled it on the same endpoints that were being used to retrieve information and measures about metric, sending back text/html instead of application/json if you were requesting those pages from a Web browser. But let's face it: we are back-end developers, we suck at any kind front-end development. CSS, HTML, JavaScript? Bwah! So what we built was a starting point, hoping some magical Web developer would jump in and finish the job. Obviously it never happened. Ok, so what's out there? It turns out there are back-end agnostic solutions out there, and we decided to pick Grafana. Grafana is a complete graphing dashboard solution that can be plugged on top of any back-end. It already supports timeseries databases such as Graphite, InfluxDB and OpenTSDB. That was largely enough for that my fellow developer Mehdi Abaakouk to jump in and start writing a Gnocchi plugin for Grafana! Consequently, there is now a basic but solid and working back-end for Grafana that lies in the grafana-plugins repository.
With that plugin, you can graph anything that is stored in Gnocchi, from raw metrics to metrics tied to resources. You can use templating, but no annotation yet. The back-end supports Gnocchi with or without Keystone involved, and any type of authentication (basic auth or Keystone token). So yes, it even works if you're not running Gnocchi with the rest of OpenStack.
It also supports advanced queries, so you can search for resources based on some criterion and graphs their metrics. I want to try it! If you want to deploy it, all you need to do is to install Grafana and its plugins, and create a new datasource pointing to Gnocchi. It is that simple. There's some CORS middleware configuration involved if you're planning on using Keystone authentication, but it's pretty straightforward just set the cors.allowed_origin option to the URL of your Grafana dashboard. We added support of Grafana directly in Gnocchi devstack plugin. If you're running DevStack you can follow the instructions which are basically adding the line enable_service gnocchi-grafana. Moving to Grafana core Mehdi just opened a pull request a few days ago to merge the plugin into Grafana core. It's actually one of the most unit-tested plugin in Grafana so far, so it should be on a good path to be merged in the future and have support of Gnocchi directly into Grafana without any plugin involved.

Francois Marier: Setting up a network scanner using SANE

Sharing a scanner over the network using SANE is fairly straightforward. Here's how I shared a scanner on a server (running Debian jessie) with a client (running Ubuntu trusty).

Install SANE The packages you need on both the client and the server are: You should check whether or your scanner is supported by the latest stable release or by the latest development version. In my case, I needed to get a Canon LiDE 220 working so I had to grab the libsane 1.0.25+git20150528-1 package from Debian experimental.

Test the scanner locally Once you have SANE installed, you can test it out locally to confirm that it detects your scanner: 
scanimage -L
This should give you output similar to this: 
device  genesys:libusb:001:006' is a Canon LiDE 220 flatbed scanner
If that doesn't work, make sure that the scanner is actually detected by the USB stack: 
$ lsusb   grep Canon
Bus 001 Device 006: ID 04a9:190f Canon, Inc.
and that its USB ID shows up in the SANE backend it needs: 
$ grep 190f /etc/sane.d/genesys.conf 
usb 0x04a9 0x190f
To do a test scan, simply run: 
scanimage > test.ppm
and then take a look at the (greyscale) image it produced (test.ppm).

Configure the server With the scanner working locally, it's time to expose it to network clients by adding the client IP addresses to /etc/sane.d/saned.conf: 
## Access list
192.168.1.3
and then opening the appropriate port on your firewall (typically /etc/network/iptables in Debian): 
-A INPUT -s 192.168.1.3 -p tcp --dport 6566 -j ACCEPT
Then you need to ensure that the SANE server is running by setting the following in /etc/default/saned: 
RUN=yes
if you're using the sysv init system, or by running this command: 
systemctl enable saned.socket
if using systemd. I actually had to reboot to make saned visible to systemd, so if you still run into these errors: 
$ service saned start
Failed to start saned.service: Unit saned.service is masked.
you're probably just one reboot away from getting it to work.

Configure the client On the client, all you need to do is add the following to /etc/sane.d/net.conf: 
connect_timeout = 60
myserver
where myserver is the hostname or IP address of the server running saned.

Test the scanner remotely With everything in place, you should be able to see the scanner from the client computer: 
$ scanimage -L
device  net:myserver:genesys:libusb:001:006' is a Canon LiDE 220 flatbed scanner
and successfully perform a test scan using this command: 
scanimage > test.ppm

12 September 2015

Ritesh Raj Sarraf: Google Take Action - Encryption

So I got an email yesterday from Google's Jess Hemerly, talking about how they care about encryption To quote a small snippet of the email: 
We use locks to keep our homes and our possessions safe offline. But how can we protect our digital things, like photos on a smartphone or email traveling across the web? The answer is encryption.
To change, it has to start from home. That means, add encryption support to Gmail. Add encryption to Google Photos. To every service you provide. So that, if tomorrow, your service is compromised, you don't public our data. What all the service providers need to realize, is that it is not the tool that needs encryption, but the data. You could claim that newer Androids are safer because encryption is enabled by default. But the data that you are backing up on your cloud, is not encrypted. It only makes sense to have an end-to-end encryption, or rather, encryption at source. Had you just stacked the interface to have write() transparently call encrypt() for any and all data, there'd be no worry about privacy. But wait, then how'd you munch.

Categories:

Keywords:

Like:

Michal Čihař: Weblate for translating everything

Weblate is not only useful for translating software, it can help in translating any content. Let's look where our users are using it. Software translation is the most usual use case. This is actually where Weblate was used for first time and still provides great support for that. As an example (and oldest project hosted in Weblate) you can look at phpMyAdmin, where Weblate also helps to keep in sync translation for different maintenance branches. It can also help you in using same terminology in command line utility and graphical one like it is done in Gammu and Wammu translations. Weblate also supports localization of mobile application. Android support was originally implemented for OsmAnd project, but now they also translate iOS application using Weblate. Another use case is for translating documentation. You need to have some infrastructure to do, but there are several free software project to help here as well. For phpMyAdmin documentation or documentation for python module Odorik the Sphinx documentation generator is used, which has support for translation of the documentation. I've recently published quick howto for this setup. Another great tool for translating documentation is Publican, it is Docbook based tool and is used for translating The Debian Administrator's Handbook. Last but not least, you can use Weblate to translate your website. For example website for Weblate itself is translated this way. Did you miss your use case for Weblate? Please let me know. PS: I've heard that one Linux distribution will start using Weblate soon.

Filed under: English phpMyAdmin SUSE Weblate 0 comments

11 September 2015

Matthias Klumpp: Deploying Limba packages: Ideas & current status

LimbaHubThe Limba project does not only have the goal to allow developers to deploy their applications directly on multiple Linux distributions while reducing duplication of shared resources, it should also make it easy for developers to build software for Limba. Limba is worth nothing without good tooling to make it fun to use. That s why I am working on that too, and I want to share some ideas of how things could work in future and which services I would like to have running. I will also show what is working today already (and that s quite something!). This time I look at things from a developer s perspective (since the last posts on Limba were more end-user centric). If you read on, you will also find a nice video of the developer workflow 1. Creating metadata and building the software To make building Limba packages as simple as possible, Limba reuses already existing metadata, like AppStream metadata to find information about the software you want to create your package for. To ensure upstreams can build their software in a clean environment, Limba makes using one as simple as possible: The limba-build CLI tool creates a clean chroot environment quickly by using an environment created by debootstrap (or a comparable tool suitable for the Linux distribution), and then using OverlayFS to have all changes to the environment done during the build process land in a separate directory. To define build instructions, limba-build uses the same YAML format TravisCI uses as well for continuous integration. So there is a chance this data is already present as well (if not, it s trivial to write). In case upstream projects don t want to use these tools, e.g. because they have well-working CI already, then all commands needed to build a Limba package can be called individually as well (ideally, building a Limba package is just one call to lipkgen). I am currently planning DeveloperIPK packages containing resources needed to develop against another Limba package. With that in place and integrated with the automatic build-environment creation, upstream developers can be sure the application they just built is built against the right libraries as present in the package they depend on. The build tool could even fetch the build-dependencies automatically from a central repository. 2. Uploading the software to a repository While everyone can set up their own Limba repository, and the limba-build repo command will help with that, there are lots of benefits in having a central place where upstream developers can upload their software to. I am currently developing a service like that, called LimbaHub . LimbaHub will contain different repositories distributors can make available to their users by default, e.g. there will be one with only free software, and one for proprietary software. It will also later allow upstreams to create private repositories, e.g. for beta-releases. 3. Security in LimbaHub Every Limba package is signed with they key of its creator anyway, so in order to get a package into LimbaHub, one needs to get their OpenPGP key accepted by the service first. Additionally, the Hub service works with a per-package permission system. This means I can e.g. allow the Mozilla release team members to upload a package with the component-ID org.mozilla.firefox.desktop or even allow those user(s) to own the whole org.mozilla.* namespace. This should prevent people hijacking other people s uploads accidentally or on purpose. 4. QA with LimbaHub LimbaHub should also act as guardian over ABI stability and general quality of the software. We could for example warn upstreams that they broke ABI without declaring that in the package information, or even reject the package then. We could validate .desktop files and AppStream metadata, or even check if a package was built using hardening flags. This should help both developers to improve their software as well as users who benefit from that effort. In case something really bad gets submitted to LimbaHub, we always have the ability to remove the package from the repositories as a last resort (which might trigger Limba to issue a warning for the user that he won t receive updates anymore). What works Limba, LimbaHub and the tools around it are developing nicely, so far no big issues have been encountered yet. That s why I made a video showing how Limba and LimbaHub work together at time: Still, there is a lot of room for improvement Limba has not yet received enough testing, and LimbaHub is merely a proof-of-concept at time. Also, lots of high-priority features are not yet implemented. LimbaHub and Limba need help! At time I am developing LimbaHub and Limba alone with only occasional contributions from others (which are amazing and highly welcomed!). So, if you like Python and Flask, and want to help developing LimbaHub, please contact me the LimbaHub software could benefit from a more experienced Python web developer than I am (and maybe having a designer look over the frontend later makes sense as well). If you are not afraid of C and GLib, and like to chase bugs or play with building Limba packages, consider helping Limba development :-)

Russell Coker: Running a Shell in a Daemon Domain

allow unconfined_t logrotate_t:process transition;
allow logrotate_t shell_exec_t bin_t :file entrypoint;
allow logrotate_t unconfined_t:fd use;
allow logrotate_t unconfined_t:process sigchld; I recently had a problem with SE Linux policy related to logrotate. To test it out I decided to run a shell in the domain logrotate_t to interactively perform some of the operations that logrotate performs when run from cron. I used the above policy to allow unconfined_t (the default domain for a sysadmin shell) to enter the daemon domain. Then I used the command runcon -r system_r -t logrotate_t bash to run a shell in the domain logrotate_t. The utility runcon will attempt to run a program in any SE Linux context you specify, but to succeed the system has to be in permissive mode or you need policy to permit it. I could have written policy to allow the logrotate_t domain to be in the role unconfined_r but it was easier to just use runcon to change roles. Then I had a shell in the logrotate_t command to test out the post-rotate scripts. It turned out that I didn t really need to do this (I had misread the output of an earlier sesearch command). But this technique can be used for debugging other SE Linux related problems so it seemed worth blogging about.
Related posts:
  1. Xen CPU use per Domain The command xm list displays the number of seconds of...
  2. SE Linux Lenny Status Update I previously described four levels of SE Linux support on...
  3. UBAC and SE Linux in Debian A recent development in SE Linux policy is the concept...

6 September 2015

Lunar: Reproducible builds: week 19 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Dmitry Shachnev uploaded sphinx/1.3.1-6 with improved patches from Val Lorentz. Chris Lamb submitted a patch for ibus-table which makes the output of ibus-table-createdb deterministic. Niko Tyni wrote a patch to make libmodule-build-perl linking order deterministic. Santiago Vila has been leading discussions on the best way to fix timestamps coming from Gettext POT files. Packages fixed The following 35 packages became reproducible due to changes in their build dependencies: apache-log4j2, dctrl-tools, dms, gitit, gnubik, isrcsubmit, mailutils, normaliz, oaklisp, octave-fpl, octave-specfun, octave-vrml, opencolorio, openvdb, pescetti, php-guzzlehttp, proofgeneral, pyblosxom, pyopencl, pyqi, python-expyriment, python-flask-httpauth, python-mzml, python-simpy, python-tidylib, reactive-streams, scmxx, shared-mime-info, sikuli, siproxd, srtp, tachyon, tcltk-defaults, urjtag, velvet. The following packages became reproducible after getting fixed: The package is not in yet in unstable, but linux/4.2-1~exp1 is now reproducible! Kudos to Ben Hutchings, and most fixes are already merged upstream. Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet:
  • #797432 on torus-trooper by Reiner Herrmann: set locale to C when sorting source file list.
  • #797437 on flow-tools by Chris Lamb: use pre-defined hostname and date of the latest debian/changelog entry in build string.
  • #797505 on cloudprint by Chris Lamb: remove embedded .pyc files.
  • #797506 on comix by Chris Lamb: remove embedded .pyc files.
  • #797508 on litl by Chris Lamb: remove date from documentation generated with LaTeX.
  • #797518 on gyoto by Chris Lamb: set date in documentation generated with LaTeX to the latest debian/changelog entry.
  • #797539 on cadabra by Chris Lamb: use date of the latest debian/changelog entry as build time.
  • #797543 on xotcl by Chris Lamb: sort source list in Makefile.
  • #797579 on ferret-vis by Chris Lamb: use date of the latest debian/changelog entry as build time.
  • #797711 on libkinosearch1-perl by Niko Tyni: sort source list in Build.PL.
  • #797871 on xbae by Chris Lamb: use date of the latest debian/changelog entry as build time.
reproducible.debian.net Some bugs that prevented packages to build successfully in the remote builders have been fixed. (h01ger) Two more amd64 build jobs have been removed from the Jenkins host in favor of six more on the new remote nodes. (h01ger) The munin graphs currently looks fine, so more amd64 jobs will probably be added in the next week. diffoscope development Version 32 of diffoscope has been released on September 3rd with the following new features:
  • A new --fuzzy-threshold option to specify the TLSH score used as cut-off for fuzzy matching. Specifying 0 will disable fuzzy-matching entirely. Suggested by Jakub Wilk.
  • A new --new-file option to treat absent files as empty. This make diffoscope a great tool to look at the content of an archive at once by comparing it with a non-existent file (example). Suggested by Jakub Wilk.
  • Comparisons of symlinks and devices given on the command line is now possible.
  • Default values are displayed in --help.
It also fixes many bugs. Head over to the changelog for the full list. Version 33 was released the day after to fix a bug introduced in the packaging. Documentation update Chris Lamb blessed the SOURCE_DATE_EPOCH specification with the version number 1.0 . Lunar documented how the .file assembler directive can help with random filenames in debug symbols. Package reviews 235 reviews have been removed, 84 added and 277 updated this week. 29 new FTBFS bugs were filled by Chris Lamb, Chris West (Faux), Daniel Stender, and Niko Tyni. New issues identified this week: random_order_in_ibus_table_createdb_output, random_order_in_antlr_output, nondetermistic_link_order_in_module_build, and timestamps_in_tex_documents. Misc. Thanks to Dhole and Thomas Vincent, the talk held at DebConf15 now has subtitles! Void Linux started to merge changes to make packages produced by xbps reproducible.

5 September 2015

Lisandro Damián Nicanor Pérez Meyer: Important Akonadi fix in today's Debian Jessie's update (aka 8.2)

Todays Debian Jessie's update brings a fix in Akonadi that you certainly want in your system.

There was a bug in Akonadi that made it leak files. And if you use Kmail you will certainly want to keep reading: most of us who tested it before pushing it to testing (and now to stable) removed more than 4 GiB of useless data from our homes.

The bug that makes Akonadi leak files gets solved with the latest stable update (and has been in testing for a couple of months already). But you need to purge the leaked files. It's pretty easy: with your normal user account just run:

akonadictl fsck

That's all. After a while you will get back a lot of disk space. Note that you don't need the Akonadi fix in order to run this tool and recover your space. The fix just makes sure this won't happen again.

4 September 2015

Guido G nther: Debian work in August 2015

Debian LTS August was the fourth month I contributed to Debian LTS under the Freexian umbrella. In total I spent four hours working on:
  • pykerberors: Fixed a regression in DLA-265-1 affecting the default parameters of checkPassword(). This resulted in DLA-265-2.
  • wordpress: Craig Small updated wordpress to fix CVE-2015-2213 CVE-2015-5622 CVE-2015-5731 CVE-2015-5732 and CVE-2015-5734. I did some testing and released DLA-294-1.
Besides that I did CVE triaging of 9 CVEs to check if and how they affect oldoldstable security as part of my LTS front desk work. Debconf 15 was a great opportunity to meet some of the other LTS contributors in person and to work on some of my packages: Git-buildpackage git-buildpackage gained buildpackage-rpm based on the work by Markus Lehtonnen and merging of mock support is hopefully around the corner. Debconf had two gbp skill shares hosted by dkg and a BoF by myself. A summary is here. Integration with dgit as (discussed with Ian) looks doable and I have parts of that on my todo list as well. Among other things gbp import-orig gained a --merge-mode option so you can replace the upstream branches verbatim on your packaging branch but keep the contents of the debian/ directory. Libvirt I prepared an update for libvirt in Jessie fixing a crasher bug, QEMU error reporting. apparmor support now works out of the box in Jessie (thanks to intrigeri and Felix Geyer for that). Speaking of apparmor I learned enough at Debconf to use this now by default so we hopefully see less breackage in this area when new libvirt versions hit the archive. The bug count wen't down quiet a bit and we have a new version of virt-manager in unstable now as well. As usual I prepared the RC candidates of libvirt 1.2.19 in experimental and 1.2.19 final is now in unstable.

1 September 2015

Lunar: Reproducible builds: week 18 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes
  • Bdale Garbee uploaded tar/1.28-1 which includes the --clamp-mtime option. Patch by Lunar.
Aur lien Jarno uploaded glibc/2.21-0experimental1 which will fix the issue were locales-all did not behave exactly like locales despite having it in the Provides field. Lunar rebased the pu/reproducible_builds branch for dpkg on top of the released 1.18.2. This made visible an issue with udebs and automatically generated debug packages. The summary from the meeting at DebConf15 between ftpmasters, dpkg mainatainers and reproducible builds folks has been posted to the revelant mailing lists. Packages fixed The following 70 packages became reproducible due to changes in their build dependencies: activemq-activeio, async-http-client, classworlds, clirr, compress-lzf, dbus-c++, felix-bundlerepository, felix-framework, felix-gogo-command, felix-gogo-runtime, felix-gogo-shell, felix-main, felix-shell-tui, felix-shell, findbugs-bcel, gco, gdebi, gecode, geronimo-ejb-3.2-spec, git-repair, gmetric4j, gs-collections, hawtbuf, hawtdispatch, jack-tools, jackson-dataformat-cbor, jackson-dataformat-yaml, jackson-module-jaxb-annotations, jmxetric, json-simple, kryo-serializers, lhapdf, libccrtp, libclaw, libcommoncpp2, libftdi1, libjboss-marshalling-java, libmimic, libphysfs, libxstream-java, limereg, maven-debian-helper, maven-filtering, maven-invoker, mochiweb, mongo-java-driver, mqtt-client, netty-3.9, openhft-chronicle-queue, openhft-compiler, openhft-lang, pavucontrol, plexus-ant-factory, plexus-archiver, plexus-bsh-factory, plexus-cdc, plexus-classworlds2, plexus-component-metadata, plexus-container-default, plexus-io, pytone, scolasync, sisu-ioc, snappy-java, spatial4j-0.4, tika, treeline, wss4j, xtalk, zshdb. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet:
  • #797027 on zyne by Chris Lamb: switch to pybuild to get rid of .pyc files.
  • #797180 on python-doit by Chris Lamb: sort output when creating completion script for bash and zsh.
  • #797211 on apt-dater by Chris Lamb: fix implementation of SOURCE_DATE_EPOCH.
  • #797215 on getdns by Chris Lamb: fix call to dpkg-parsechangelog in debian/rules.
  • #797254 on splint by Chris Lamb: support SOURCE_DATE_EPOCH for version string.
  • #797296 on shiro by Chris Lamb: remove username from build string.
  • #797408 on splitpatch by Reiner Herrmann: use SOURCE_DATE_EPOCH to set manpage date.
  • #797410 on eigenbase-farrago by Reiner Herrmann: sets the comment style to scm-safe which tells ResourceGen that no timestamps should be included.
  • #797415 on apparmor by Reiner Herrmann: sorting with the locale set to C. CAPABILITIES
  • #797419 on resiprocate by Reiner Herrmann: set the embedded hostname to a static value.
  • #797427 on jam by Reiner Herrmann: sorting with the locale set to C.
  • #797430 on ii-esu by Reiner Herrmann: sort source list using C locale.
  • #797431 on tatan by Reiner Herrmann: sort source list using C locale.
Chris Lamb also noticed that binaries shipped with libsilo-bin did not work. Documentation update Chris Lamb and Ximin Luo assembled a proper specification for SOURCE_DATE_EPOCH in the hope to convince more upstreams to adopt it. Thanks to Holger it is published under a non-Debian domain name. Lunar documented easiest way to solve issues with file ordering and timestamps in tarballs that came with tar/1.28-1. Some examples on how to use SOURCE_DATE_EPOCH have been improved to support systems without GNU date. reproducible.debian.net armhf is finally being tested, which also means the remote building of Debian packages finally works! This paves the way to perform the tests on even more architectures and doing variations on CPU and date. Some packages even produce the same binary Arch:all packages on different architectures (1, 2). (h01ger) Tests for FreeBSD are finally running. (h01ger) As it seems the gcc5 transition has cooled off, we schedule sid more often than testing again on amd64. (h01ger) disorderfs has been built and installed on all build nodes (amd64 and armhf). One issue related to permissions for root and unpriviliged users needs to be solved before disorderfs can be used on reproducible.debian.net. (h01ger) strip-nondeterminism Version 0.011-1 has been released on August 29th. The new version updates dh_strip_nondeterminism to match recent changes in debhelper. (Andrew Ayer) disorderfs disorderfs, the new FUSE filesystem to ease testing of filesystem-related variations, is now almost ready to be used. Version 0.2.0 adds support for extended attributes. Since then Andrew Ayer also added support to reverse directory entries instead of shuffling them, and arbitrary padding to the number of blocks used by files. Package reviews 142 reviews have been removed, 48 added and 259 updated this week. Santiago Vila renamed the not_using_dh_builddeb issue into varying_mtimes_in_data_tar_gz_or_control_tar_gz to align better with other tag names. New issue identified this week: random_order_in_python_doit_completion. 37 FTBFS issues have been reported by Chris West (Faux) and Chris Lamb. Misc. h01ger gave a talk at FrOSCon on August 23rd. Recordings are already online. These reports are being reviewed and enhanced every week by many people hanging out on #debian-reproducible. Huge thanks!

Russ Allbery: Review: Kanban

Review: Kanban, by David J. Anderson
Publisher: Blue Hole
Copyright: 2010
ISBN: 0-9845214-0-2
Format: Trade paperback
Pages: 240
Another belated review, this time of a borrowed book. Which I can now finally return! A delay in the review of this book might be a feature if I had actually used it for, as the subtitle puts it, successful evolutionary change in my technology business. Sadly, I haven't, so it's just late. So, my background: I've done a lot of variations of traditional project management for IT projects (both development and more operational ones), both as a participant and as a project manager. (Although I've never done the latter as a full-time job, and have no desire to do so.) A while back at Stanford, my team adopted Agile, specifically Scrum, so I did a fair bit of research about Scrum including a couple of training courses. Since then, at Dropbox, I've used a few different variations on a vaguely Agile-inspired planning process, although it's not particularly consistent with any one system. I've been hearing about Kanban for a while and have friends who swear by it, but I only had a vague idea of how it worked. That seemed like a good excuse to read a book. And Anderson's book is a good one if, like me, you're looking for a basic introduction. It opens with a basic description and definition, talks about motivation and the expected benefits, and then provides a detailed description of Kanban as a system. The tone is on the theory side, written using the terminology of (I presume) management theory and operations management, areas about which I know almost nothing, but the terminology wasn't so heavy as to make the book hard to read. Anderson goes into lots of detail, and I thought he did a good job of distinguishing between basic principles, optional techniques, and variations that may be appropriate for particular environments. If you're also not familiar, the basic concept of Kanban is to organize work using an in-progress queue. It eschews time-bounded planning entirely in favor of staging work at the start of a sequence of queues and letting the people working on each queue pull from the previous queue when they're ready to take on a new task. As you might guess from that layout, Kanban was originally invented for assembly-line manufacturing (at Toyota). That was one of the problems that I had with it, or at least the presentation in this book: most of my software development doesn't involve finishing one part of something and handing it off to someone else, which made it hard to identify with the pipeline model. Anderson has clearly spent a lot of time working with large-scale programming shops, including outsourced development firms, with dedicated QA and operations roles. This is not at all how Silicon Valley agile development works, so parts of this book felt like missives from a foreign country. That said, the key point of Kanban is not the assembly line but the work limit. One of the defining characteristics of Kanban, at least as Anderson presents it, is that one does not try to estimate work and tile it based on estimates, not even to the extent that Scrum and other Agile methodologies do within the sprint. Instead, each person takes as long as they take on the things they're working on, and pulls a new task when they have free capacity. The system instead puts a limit on how many things they can have in progress at a time. The problem of pipeline stalls is dealt with both via continuous improvement and "swarming" of a problem to unblock the line (since other teams may not be able to work until the block is fixed), and with being careful about the sizing of work items (I'm used to calling them stories) that go in the front end. Predictability, which Scrum uses story sizing and team velocity analysis to try to achieve, is basically statistical. One uses a small number of buckets of sizes of stories, and the whole pipeline will finish some number of work items per unit time, with a variance. The promise made to clients and other teams is that some percentage of the work items will be finished within some time frame from when they enter the system. Most importantly, these are all descriptive properties, determined statistically after the fact, rather than planned properties worked out through story sizing and extensive team discussion. If you, like me, are pretty thoroughly sick of two-hour sprint planning meetings and endless sizing exercises, this is rather appealing. My problem with most work planning systems is that I think they overplan and put too much weight our ability to estimate how long work will take. Kanban is very appealing when viewed through that lens: it gives up on things we're bad at in favor of simple measurement and building a system with enough slack that it can handle work of various different sizes. As mentioned at the top, I haven't had a chance to try it (and I'm not sure it's a good fit with the inter-group planning methods in use at my current workplace), but I came away from this book wanting to do so. If, like me, your experience is mostly with small combined teams or individual programming work, Anderson's examples may seem rather large, rather formal, and bureaucratic. But this is a solid introduction and well worth reading if your only experience with Agile is sprint planning, story writing and sizing, and fast iterations. Rating: 7 out of 10

31 August 2015

Russ Allbery: Review: Through Struggle, The Stars

Review: Through Struggle, The Stars, by John J. Lumpkin
Series: Human Reach #1
Publisher: John J. Lumpkin
Copyright: July 2011
ISBN: 1-4611-9544-6
Format: Kindle
Pages: 429
Never let it be said that I don't read military SF. However, it can be said that I read books and then get hellishly busy and don't review them for months. So we'll see if I can remember this well enough to review it properly. In Lumpkin's future world, mankind has spread to the stars using gate technology, colonizing multiple worlds. However, unlike most science fiction futures of this type, it's not all about the United States, or even the United States and Russia. The great human powers are China and Japan, with the United States relegated to a distant third. The US mostly maintains its independence from either, and joins the other lesser nations and UN coalitions to try to pick up a few colonies of its own. That's the context in which Neil and Rand join the armed services: the former as a pilot in training, and the latter as an army grunt. This is military SF, so of course a war breaks out. But it's a war between Japan and China: improved starship technology and the most sophisticated manufacturing in the world against a much larger economy with more resources and a larger starting military. For reasons that are not immediately clear, and become a plot point later on, the United States president immediately takes an aggressive tone towards China and pushes the country towards joining the war on the side of Japan. Most of this book is told from Neil's perspective, following his military career during the outbreak of war. His plans to become a pilot get derailed as he gets entangled with US intelligence agents (and a bad supervisor). The US forces are not in a good place against China, struggling when they get into ship-to-ship combat, and Neil's ship goes on a covert mission to attempt to complicate the war with political factors. Meanwhile, Rand tries to help fight off a Chinese invasion of one of the rare US colony worlds. Through Struggle, The Stars does not move quickly. It's over 400 pages, and it's a bit surprising how little happens. What it does instead is focus on how the military world and the war feels to Neil: the psychological experience of wanting to serve your country but questioning its decisions, the struggle of working with people who aren't always competent but who you can't just make go away, the complexities of choosing a career path when all the choices are fraught with politics that you didn't expect to be involved in, and the sheer amount of luck and random events involved in the progression of one's career. I found this surprisingly engrossing despite the slow pace, mostly because of how true to life it feels. War is not a never-ending set of battles. Life in a military ship has moments when everything is happening far too quickly, but other moments when not much happens for a long time. Lumpkin does a great job of reflecting that. Unfortunately, I thought there were two significant flaws, one of which means I probably won't seek out further books in this series. First, one middle portion of the book switches away from Neil to follow Rand instead. The first part of that involves the details of fighting orbiting ships with ground-based lasers, which was moderately interesting. (All the technological details of space combat are interesting and thoughtfully handled, although I'm not the sort of reader who will notice more subtle flaws. But David Weber this isn't, thankfully.) But then it turns into a fairly generic armed resistance story, which I found rather boring. It also ties into the second and more serious flaw: the villain. The overall story is constructed such that it doesn't need a personal villain. It's about the intersection of the military and politics, and a war that may be ill-conceived but that is being fought anyway. That's plenty of conflict for the story, at least in my opinion. But Lumpkin chose to introduce a specific, named Chinese character in the villain role, and the characterization is... well. After he's humiliated early in the story by the protagonists, Li Xiao develops an obsession with killing them, for his honor, and then pursues them throughout the book in ways that are sometimes destructive to the Chinese war efforts. It's badly unrealistic compared to the tone of realism taken by the rest of the story. Even if someone did become this deranged, it's bizarre that a professional military (and China's forces are otherwise portrayed as fairly professional) would allow this. Li reads like pure caricature, and despite being moderately competent apart from his inexplicable (but constantly repeated) obsession, is cast in a mustache-twirling role of personal villainy. This is weirdly out of place in the novel, and started irritating me enough that it took me out of the story. Through Struggle, The Stars is the first book of a series, and does not resolve much by the end of the novel. That plus its length makes the story somewhat unsatisfying. I liked Neil's development, and liked him as a character, and those who like the details of combat mixed with front-lines speculation about politics may enjoy this. But a badly-simplified mustache-twirling victim and some extended, uninteresting bits mar the book enough that I probably won't seek out the sequels. Followed by The Desert of Stars. Rating: 6 out of 10

30 August 2015

Antonio Terceiro: DebConf15, testing debian packages, and packaging the free software web

This is my August update, and by the far the coolest thing in it is Debconf. Debconf15 I don t get tired of saying it is the best conference I ever attended. First it s a mix of meeting both new people and old friends, having the chance to chat with people whose work you admire but never had a chance to meet before. Second, it s always quality time: an informal environment, interesting and constructive presentations and discussions. This year the venue was again very nice. Another thing that was very nice was having so many kids and families. This was no coincidence, since this was the first DebConf in which there was organized childcare. As the community gets older, this a very good way of keeping those who start having kids from being alienated from the community. Of course, not being a parent yet I have no idea how actually hard is to bring small kids to a conference like DebConf. ;-) I presented two talks:
  • Tutorial: Functional Testing of Debian packages, where I introduced the basic concepts of DEP-8/autopkgtest, and went over several examples from my packages giving tips and tricks on how to write functional tests for Debian packages.
  • Packaging the Free Software Web for the end user, where I presented the motivation for, and the current state of shak, a project I am working on to make it trivial for end users to install server side applications in Debian. I spent quite some hacking time during DebConf finishing a prototype of the shak web interface, which was demonstrated live in the talk (of course, as usual with live demos, not everything worked :-)).
There was also the now traditional Ruby BoF, where discussed the state and future of the Ruby ecosystem in Debian; and an in promptu Ruby packaging workshop where we introduced the basics of packaging in general, and Ruby packaging specifically. Besides shak, I was able to hack on a few cool things during DebConf:
  • debci has been updated with a first version of the code to produce britney hints files that block packages that fail their tests from migrating to testing. There are some issues to be sorted out together with the release team to make sure we don t block packages unecessarily, e.g. we don t want to block packages that never passed their test suite most the test suite, and not the package, is broken.
  • while hacking I ended up updating jquery to the newest version in the 1.x series, and in fact adopting it I guess. This allowed emeto drop the embedded jquery copy I used to have in the shak repository, and since then I was able to improve the build to produce an output that is identical, except for a build timestamp inside a comment and a few empty lines, to the one produced by upstream, without using grunt (.
Miscellaneous updates

Sven Hoexter: 1960 SubjectAlternativeNames on one certificate

tl;dr; You can add 1960+ SubjectAlternativeNames on one certificate and at least Firefox and Chrome are working fine with that. Internet Explorer failed but I did not investigate why. So why would you want to have close to 2K SANs on one certificate? While we're working on adopting a more dynamic development workflow at my workplace we're currently bound to a central development system. From there we serve a classic virtual hosting setup with "projectname.username.devel.ourdomain.example" mapped on "/web/username/projectname/". That is 100% dynamic with wildcard DNS entries and you can just add a new project to your folder and use it directly. All of that is served from just a single VirtualHost. Now our developers started to go through all our active projects to make them fit for serving via HTTPS. While we can verify the proper usage of https on our staging system where we've validating certificates, that's not the way you'd like to work. So someone approached me to look into a solution for our development system. Obvious choices like wildcard certificates do not work here because we've two dynamic components in the FQDN. So we would've to buy a wildcard certificate for every developer and we would've to create a VirtualHost entry for every new developer. That's expensive and we don't want all that additional work. So I started to search for documented limits on the number of SANs you can have on a certificate. The good news: there are none. The RFC does not define a limit. So much about the theory. ;) Following Ivans excellent documentation I setup an internal CA and an ugly "find ... sed ... tr ..." one-liner later I had a properly formated openssl config file to generate a CSR with all 1960 "projectname.username..." SAN combinations found on the development system. Two openssl invocations (CSR generation and signing) later I had a signed certificate with 1960 SANs on it. I imported the internal CA I created in Firefox and Chrome, and to my surprise it worked. Noteworthy: To sign with "openssl ca" without interactive prompts you've to use the "-batch" option. I'm thinking about regenerating the certificate every morning so our developers just have to create a new project directory and within 24h serving via HTTPS would be enabled. The only thing I'm currently pondering about is how to properly run the CA in a corporate Windows world. We could of course ask the Windows guys to include it for everyone but then we would've to really invest time in properly running the CA. I'd like to avoid that hassle. So I'd guess we just stick to providing the CA for those developers who need it. This all or nothing model is a constant PITA, and you really do not want to get owned via your own badly managed CA. :( Regarding Internet Explorer it jumped in my face with a strange error message that recommended to enable TLS 1.0, 1.1 and 1.2 in the options menu. Of course that's already enable. I'll try to take a look at the handshake next week, but I bet we've to accept for the moment that IE will not work with so many SANs. Would be interesting to try out Windows 10 with Spartan, but well I'm not that interested in Windows to invest more time on that front. Other TLS implementations, like Java, would be also interesting to test.

29 August 2015

Francois Marier: Letting someone ssh into your laptop using Pagekite

In order to investigate a bug I was running into, I recently had to give my colleague ssh access to my laptop behind a firewall. The easiest way I found to do this was to create an account for him on my laptop and setup a pagekite frontend on my Linode server and a pagekite backend on my laptop.

Frontend setup Setting up my Linode server in order to make the ssh service accessible and proxy the traffic to my laptop was fairly straightforward. First, I had to install the pagekite package (already in Debian and Ubuntu) and open up a port on my firewall by adding the following to both /etc/network/iptables.up.rules and /etc/network/ip6tables.up.rules: 
-A INPUT -p tcp --dport 10022 -j ACCEPT
Then I created a new CNAME for my server in DNS: 
pagekite.fmarier.org.   3600    IN  CNAME   fmarier.org.
With that in place, I started the pagekite frontend using this command: 
pagekite --clean --isfrontend --rawports=virtual --ports=10022 --domain=raw:pagekite.fmarier.org:Password1

Backend setup After installing the pagekite and openssh-server packages on my laptop and creating a new user account: 
adduser roc
I used this command to connect my laptop to the pagekite frontend: 
pagekite --clean --frontend=pagekite.fmarier.org:10022 --service_on=raw/22:pagekite.fmarier.org:localhost:22:Password1

Client setup Finally, my colleague needed to add the folowing entry to ~/.ssh/config: 
Host pagekite.fmarier.org
  CheckHostIP no
  ProxyCommand /bin/nc -X connect -x %h:10022 %h %p
and install the netcat-openbsd package since other versions of netcat don't work. On Fedora, we used netcat-openbsd-1.89 successfully, but this newer package may also work. He was then able to ssh into my laptop via ssh roc@pagekite.fmarier.org.

Making settings permanent I was quite happy settings things up temporarily on the command-line, but it's also possible to persist these settings and to make both the pagekite frontend and backend start up automatically at boot. See the documentation for how to do this on Debian and Fedora.

25 August 2015

Lunar: Reproducible builds: week 17 in Stretch cycle

A good amount of the Debian reproducible builds team had the chance to enjoy face-to-face interactions during DebConf15.
Names in red and blue were all present at DebConf15
Picture of the reproducible builds talk during DebConf15
Hugging people with whom one has been working tirelessly for months gives a lot of warm-fuzzy feelings. Several recorded and hallway discussions paved the way to solve the remaining issues to get reproducible builds part of Debian proper. Both talks from the Debian Project Leader and the release team mentioned the effort as important for the future of Debian. A forty-five minutes talk presented the state of the reproducible builds effort. It was then followed by an hour long roundtable to discuss current blockers regarding dpkg, .buildinfo and their integration in the archive. Picture of the reproducible builds roundtable during DebConf15 Toolchain fixes
  • Kenneth J. Pronovici uploaded epydoc/3.0.1+dfsg-12 which makes class and modules ordering predictable (#795835) and fixes __repr__ so memory addresses don't appear in docs (#795826). Patches by Val Lorentz.
  • Sergei Golovan uploaded erlang/1:18.0-dfsg-2 which adds support for SOURCE_DATE_EPOCH to erlc. Patch by Chris West (Faux) and Chris Lamb.
  • Dmitry Shachnev uploaded sphinx/1.3.1-5 which make grammar, inventory, and JavaScript locales generation deterministic. Original patch by Val Lorentz.
  • St phane Glondu uploaded ocaml/4.02.3-2 to experimental, making startup files and native packed libraries deterministic. The patch adds deterministic .file to the assembler output.
  • Enrico Tassi uploaded lua-ldoc/1.4.3-3 which now pass the -d option to txt2man and add the --date option to override the current date.
Reiner Herrmann submitted a patch to make rdfind sort the processed files before doing any operation. Chris Lamb proposed a new patch for wheel implementing support for SOURCE_DATE_EPOCH instead of the custom WHEEL_FORCE_TIMESTAMP. akira sent one making man2html SOURCE_DATE_EPOCH aware. St phane Glondu reported that dpkg-source would not respect tarball permissions when unpacking under a umask of 002. After hours of iterative testing during the DebConf workshop, Sandro Knau created a test case showing how pdflatex output can be non-deterministic with some PNG files. Packages fixed The following 65 packages became reproducible due to changes in their build dependencies: alacarte, arbtt, bullet, ccfits, commons-daemon, crack-attack, d-conf, ejabberd-contrib, erlang-bear, erlang-cherly, erlang-cowlib, erlang-folsom, erlang-goldrush, erlang-ibrowse, erlang-jiffy, erlang-lager, erlang-lhttpc, erlang-meck, erlang-p1-cache-tab, erlang-p1-iconv, erlang-p1-logger, erlang-p1-mysql, erlang-p1-pam, erlang-p1-pgsql, erlang-p1-sip, erlang-p1-stringprep, erlang-p1-stun, erlang-p1-tls, erlang-p1-utils, erlang-p1-xml, erlang-p1-yaml, erlang-p1-zlib, erlang-ranch, erlang-redis-client, erlang-uuid, freecontact, givaro, glade, gnome-shell, gupnp, gvfs, htseq, jags, jana, knot, libconfig, libkolab, libmatio, libvsqlitepp, mpmath, octave-zenity, openigtlink, paman, pisa, pynifti, qof, ruby-blankslate, ruby-xml-simple, timingframework, trace-cmd, tsung, wings3d, xdg-user-dirs, xz-utils, zpspell. The following packages became reproducible after getting fixed: Uploads that might have fixed reproducibility issues: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet:
  • #795861 on fakeroot by Val Lorentz: set the mtime of all files to the time of the last debian/changelog entry.
  • #795870 on fatresize by Chris Lamb: set build date to the time of the latest debian/changelog entry.
  • #795945 on projectl by Reiner Herrmann: sort with LC_ALL set to C.
  • #795977 on dahdi-tools by Dhole: set the timezone to UTC before calling asciidoc.
  • #795981 on x11proto-input by Dhole: set the timezone to UTC before calling asciidoc.
  • #795983 on dbusada by Dhole: set the timezone to UTC before calling asciidoc.
  • #795984 on postgresql-plproxy by Dhole: set the timezone to UTC before calling asciidoc.
  • #795985 on xorg by Dhole: set the timezone to UTC before calling asciidoc.
  • #795987 on pngcheck by Dhole: set the date in the man pages to the latest debian/changelog entry.
  • #795997 on python-babel by Val Lorentz: make build timestamp independent from the timezone and remove the name of the build system locale from the documentation.
  • #796092 on a7xpg by Reiner Herrmann: sort with LC_ALL set to C.
  • #796212 on bittornado by Chris Lamb: remove umask-varying permissions.
  • #796251 on liblucy-perl by Niko Tyni: generate lib/Lucy.xs in a deterministic order.
  • #796271 on tcsh by Reiner Herrmann: sort with LC_ALL set to C.
  • #796275 on hspell by Reiner Herrmann: remove timestamp from aff files generated by mk_he_affix.
  • #796324 on fftw3 by Reiner Herrmann: remove date from documentation files.
  • #796335 on nasm by Val Lorentz: remove extra timestamps from the build system.
  • #796360 on libical by Chris Lamb: removes randomess caused Perl in generated icalderivedvalue.c.
  • #796375 on wcd by Dhole: set the date in the man pages to the latest debian/changelog entry.
  • #796376 on mapivi by Dhole: set the date in the man pages to the latest debian/changelog entry.
  • #796527 on vserver-debiantools by Dhole: set the date in the man pages to the latest debian/changelog entry.
St phane Glondu reported two issues regarding embedded build date in omake and cduce. Aur lien Jarno submitted a fix for the breakage of make-dfsg test suite. As binutils now creates deterministic libraries by default, Aur lien's patch makes use of a wrapper to give the U flag to ar. Reiner Herrmann reported an issue with pound which embeds random dhparams in its code during the build. Better solutions are yet to be found. reproducible.debian.net Package pages on reproducible.debian.net now have a new layout improving readability designed by Mattia Rizzolo, h01ger, and Ulrike. The navigation is now on the left as vertical space is more valuable nowadays. armhf is now enabled on all pages except the dashboard. Actual tests on armhf are expected to start shortly. (Mattia Rizzolo, h01ger) The limit on how many packages people can schedule using the reschedule script on Alioth has been bumped to 200. (h01ger) mod_rewrite is now used instead of JavaScript for the form in the dashboard. (h01ger) Following the rename of the software, debbindiff has mostly been replaced by either diffoscope or differences in generated HTML and IRC notification output. Connections to UDD have been made more robust. (Mattia Rizzolo) diffoscope development diffoscope version 31 was released on August 21st. This version improves fuzzy-matching by using the tlsh algorithm instead of ssdeep. New command line options are available: --max-diff-input-lines and --max-diff-block-lines to override limits on diff input and output (Reiner Herrmann), --debugger to dump the user into pdb in case of crashes (Mattia Rizzolo). jar archives should now be detected properly (Reiner Herrman). Several general code cleanups were also done by Chris Lamb. strip-nondeterminism development Andrew Ayer released strip-nondeterminism version 0.010-1. Java properties file in jar should now be detected more accurately. A missing dependency spotted by St phane Glondu has been added. Testing directory ordering issues: disorderfs During the reproducible builds workshop at DebConf, participants identified that we were still short of a good way to test variations on filesystem behaviors (e.g. file ordering or disk usage). Andrew Ayer took a couple of hours to create disorderfs. Based on FUSE, disorderfs in an overlay filesystem that will mount the content of a directory at another location. For this first version, it will make the order in which files appear in a directory random. Documentation update Dhole documented how to implement support for SOURCE_DATE_EPOCH in Python, bash, Makefiles, CMake, and C. Chris Lamb started to convert the wiki page describing SOURCE_DATE_EPOCH into a Freedesktop-like specification in the hope that it will convince more upstream to adopt it. Package reviews 44 reviews have been removed, 192 added and 77 updated this week. New issues identified this week: locale_dependent_order_in_devlibs_depends, randomness_in_ocaml_startup_files, randomness_in_ocaml_packed_libraries, randomness_in_ocaml_custom_executables, undeterministic_symlinking_by_rdfind, random_build_path_by_golang_compiler, and images_in_pdf_generated_by_latex. 117 new FTBFS bugs have been reported by Chris Lamb, Chris West (Faux), and Niko Tyni. Misc. Some reproducibility issues might face us very late. Chris Lamb noticed that the test suite for python-pykmip was now failing because its test certificates have expired. Let's hope no packages are hiding a certificate valid for 10 years somewhere in their source! Pictures courtesy and copyright of Debian's own paparazzi: Aigars Mahinovs.

23 August 2015

Norbert Preining: Debian/TeX Live complete update

Triggered by all the bugs around font problems I spent my weekend instead of mountaineering with crawling through the TeX Live history for changes and fixes to dvipdfm-x. Thanks to for his hints, I have pulled out the changes necessary to fix Type1 support in dvipdfm-x and have reincluded them into the Debian texlive-bin package. The uploaded binaries (version 2015.20150524.37493-6) are already compiled against the new C++ ABI, see the Debian transition, so most systems will still need to wait for the update to be installable. Debian - TeX Live 2015 At the same time I did an update to the whole set of arch: all packages (texlive-base, texlive-lang, texlive-extra (version 2015.20150823-1). This was triggered by bug that seems to be caused by bad interplay between fontspec and l3 packages. Furthermore, I needed to remove the activation for fontconfig of the URW++ Base35 fonts, to ensure that fontconfig returns always the ones from the gsfonts package, instead of a mixture between TeX Live and gsfonts. Unrelated bug fix: libpaper intergration has been fixed and should work again. So for now all the bugs are now hopefully settled and we are back to normal. What remains is trying to fix jessie which is also broken in some respects. Updated packages acro, animate, babel-bosnian, babel-french, babel-latin, beamer-FUBerlin, beebe, breqn, chemformula, chet, cnltx, crossrefware, dantelogo, datetime2-it-fulltext, disser, drm, dvipdfmx-def, ecclesiastic, eledform, gradstudentresume, idxcmds, l3build, mcf2graph, media9, pageslts, pdfpages, reledmac, siunitx, tcolorbox, tex4ht, texlive-docindex, texlive-scripts, udesoftec, upmethodology, xindy. New packages blochsphere, e-french, fitbox, nar Enjoy.

Next.

Previous.