Search Results: "pm"

4 April 2016

Michal Čihař: Ukolovnik 1.5

There were quite some translation and code fixes in the repository, so I guess it's time to release it as well. So here comes Ukolovnik 1.5, released almost four years since last release. Full list of changes: PS: I don't plan any further development of this tool, this is just to bring all (few) users to current code in git.

Filed under: English Ukolovnik 0 comments

Ian Wienand: Image building for OpenStack CI -- Minimal images, big effort

A large part of OpenStack Infrastructure teams recent efforts has been focused on moving towards more stable and maintainable CI environments for testing.
OpenStack CI Overview Before getting into details, it's a good idea to get a basic big-picture conceptual model of how OpenStack CI testing works. If you look at the following diagram and follow the numbers with the explanation below, hopefully you'll have all the context you need. Overview of OpenStack CI

  1. The developer uploads their code to gerrit via the git-review tool. They wait.

  2. Gerrit provides a JSON-encoded "firehose" output of everything happening to it. New reviews, votes, updates and more all get sent out over this pipe. Zuul is the overall scheduler that subscribes itself to this information and is responsible for managing the CI jobs appropriate for each change.

  3. Zuul has a configuration that tells it what jobs to run for what projects. Zuul can do lots of interesting things, but for the purposes of this discussion we just consider that it puts the jobs it wants run into gearman for a Jenkins host to consume. gearman is a job-server; as they explain it "[gearman] provides a generic application framework to farm out work to other machines or processes that are better suited to do the work".

  4. A group of Jenkins hosts are subscribed to gearman as workers. It is these Jenkins hosts that will consume the job requests from the queue and actually get the tests running. Jenkins needs two things to be able to run a job -- a job definition (what to actually do) and a slave node (somewhere to do it). The first part -- what to do -- is provided by job-definitions stored in external YAML files and processed by Jenkins Job Builder (jjb) in to job configurations for Jenkins. Thus each Jenkins instance knows about all the jobs it might need to run. Zuul also knows about these job definitions, so you can see how we now have a mapping where Zuul can put a job into gearman saying "run test foo-bar-baz" and a Jenkins host can consume that request and know what to do. The second part -- somewhere to run the test -- takes some more explaining. To the next point...

  5. Several cloud companies donate capacity in their clouds for OpenStack to run CI tests. Overall, this capacity is managed by nodepool -- a customised orchestration tool. Nodepool watches the gearman queue and sees what requests are coming out of Zuul, and decides what type of capacity to provide and in what clouds to satisfy the outstanding job queue. Nodepool will start-up virtual-machines as required, and register those nodes to the Jenkins instances.

  6. At this point, Jenkins has what it needs to actually get jobs started. When nodepool registers a host to Jenkins as a slave, the Jenkins host can now advertise its ability to consume jobs. For example, if a ubuntu-trusty node is provided to the Jenkins instance by nodepool, Jenkins can now consume a job from the queue intended to run on an ubuntu-trusty host. It will run the job as defined in the job-definition -- ssh-ing into the host, running scripts, copying the logs and waiting for the result. (It is a gross oversimplification, but but Jenkins is pretty much a glorified ssh/scp wrapper to OpenStack CI. Zuul Version 3, under development, is working to remove the need for Jenkins to be involved at all).

  7. Eventually, the test will finish. Jenkins will put the result back into gearman, which Zuul will consume. The slave will be released back to nodepool, which destroys it and starts all over again (slaves are not reused and also have no sensitive details on them, as they are essentially publicly accessible). Zuul will wait for the results of all jobs and post the result back to Gerrit and give either a positive vote or the dreaded negative vote if required jobs failed (it also handles merges to git, but we'll ignore that bit for now).

In a nutshell, that is the CI work-flow that happens thousands-upon-thousands of times a day keeping OpenStack humming along.
Image builds There is, however, another more asynchronous part of the process that hides a lot of details the rest of the system relies on. Illustrated in step 8 above, this is the management of the images that tests are being run upon. Above we we said that a test runs on a ubuntu-trusty, centos, fedora or some other type of node, but glossed over where these images come from. Firstly, what are these images, and why build them at all? These images are where the "rubber hits the road" -- where DevStack, functional testing or whatever else someone might want to test is actually run for real. Caching is a big part of the role of these images. With thousands of jobs going on every day, an occasional network blip is not a minor annoyance, but creates constant and difficult to debug CI failures. We want the images that CI runs on to rely on as few external resources as possible so test runs are as stable as possible. This means caching all the git trees tests might use, VM images consumed by various tests and other common bits and pieces. Obviously a cache is only as useful as the data in it, so we build these images up every day to keep them fresh.
Provider images If you log into almost any cloud-provider's interface, they almost certainly have a range of pre-canned images of common distributions for you to use. At first, the base images for OpenStack CI testing came from what the cloud-providers had as their public image types. However, over time, there are a number of issues that emerge:
  1. No two images, even for the same distribution or platform, are the same. Every provider seems to do something "helpful" to the images which requires some sort of workaround.
  2. Providers rarely leave these images alone. One day you would boot the image to find a bunch of Python libraries pip-installed, or a mount-point moved, or base packages removed (all happened).
  3. Even if the changes are helpful, it does not make for consistent and reproducible testing if every time you run, you're on a slightly different base system.
  4. Providers don't have some images you want (like a latest Fedora), or have different versions, or different point releases. All update asynchronously whenever they get around to it.
So the original incarnations of building images was that nodepool would start one of these provider images, run a bunch of scripts on it to make a base-image (do the caching, setup keys, etc), snapshot it and then start putting VM's based on these images into the pool for testing (if you hear someone talking about a "snapshot image" in OpenStack CI context, that's what it means). The first problem you hit here is that the number of images being built starts to explode when you take into account multiple providers and multiple regions. Even with just Rackspace and the (now defunct) HP Cloud we would end up snapshot images for 4 or 5 platforms across a total of about 8 regions -- meaning anywhere up to 40 separate image builds happening. It was almost a fait accompli that some images would fail every day -- nodepool can deal with this by reusing old snapshots; but this leads to a inconsistent and heterogeneous testing environment. OpenStack is like a gigantic Jenga tower, with a full DevStack deployment resulting in hundreds of libraries and millions of lines of code all being exercised at once. The testing images are right at the bottom of all this, and it doesn't take much to make the whole thing fall over (see points about providers not leaving images alone). This leads to constant fire-firefighting and everyone annoyed as all CI stops. Naturally there was a desire for something much more consistent -- a single image that could run across multiple providers in a much more tightly controlled manner.
Upstream-based builds Upstream distributions do provide their "cloud-images", which are usually pre-canned .qcow2 format files suitable for uploading to your average cloud. So the diskimage-builder tool was put into use creating images for nodepool, based on these upstream-provided images. In essence, diskimage-builder uses a series of elements (each, as the name suggests, designed to do one thing) that allow you to build a completely customised image. It handles all the messy bits of laying out the image file, tries to be smart about caching large downloads and final things like conversion to qcow2 or vhd or whatever your cloud requires. nodepool has used diskimage-builder to create customised images based upon the upstream releases for some time. These are better, but still have some issues for the CI environment:
  1. You still really have no control over what does or does not go into the upstream base images. You don't notice a change until you deploy a new image based on an updated version and things break.
  2. The images still have a fair amount of "stuff" on them. For example cloud-init is a rather large Python program and has a fair few dependencies. These dependencies can both conflict with parts of OpenStack or end up tacitly hiding real test requirements (the test doesn't specify it, but the package is there as part of another base dependency -- things break when the base dependencies change). The whole idea of the CI is that (as much as possible) you're not making any assumptions about what is required to run your tests -- you want everything explicitly included. (If you were starting this whole thing again, things like Docker to keep "systems within systems" might come into play. Indeed they may be in the future. But don't forget that DevStack, the major CI deployment mechanism, was started before Docker existed. And there's tricky stuff with networking and Neutron etc going on).
  3. An image that "works everywhere" across multiple cloud-providers is quite a chore. cloud-init hasn't always had support for config-drive and Rackspace's DHCP-less environment, for example. Providers all have their various different networking schemes or configuration methods which needs to be handled consistently.
Minimal builds To this end, diskimage-builder now has a serial of "minimal" builds that are really that -- systems with essentially nothing on them. For Debian and Ubuntu, this is achieved via debootstrap, for Fedora and CentOS we replicate this with manual installs of base packages into a clean chroot environment. We add-on a range of important elements that make the image useful; for example, for networking, we have simple-init which brings up the network consistently across all our providers but has no dependencies to mess with the base system. If you check the elements provided by project-config you can see a range of specific elements that OpenStack Infra runs at each image build (these are actually specified by in arguments to nodepool, see the config file, particularly diskimages section). These custom elements do things like caching, using puppet to install the right authorized_keys files and setup a few needed things to connect to the host. In general, you can see the logs of an image build provided by nodepool for each daily build. So now, each day at 14:00 UTC nodepool builds the daily images that will be used for CI testing. We have one image of each type that (theoretically) works across all our providers. After it finishes building, nodepool uploads the image to all providers (p.s. the process of doing this is so insanely terrible it spawned shade; this deserves many posts of its own) at which point it will start being used for CI jobs. If you wish to replicate this entire process, the build-image.sh script, run on an Ubuntu Trusty host in a virtualenv with diskimage-builder will get you pretty close (let us know of any issues!).
Dependencies But guess what, there's more! Along the way, OpenStack CI ended up with the concept of bare nodes and devstack nodes. A bare node was one that was used for functional testing; i.e. what you do when you type tox -e py27 in basically any OpenStack project. The problem here is that tox has plenty of information about installing required Python packages into the virtualenv for testing; but it doesn't know anything about the system packages required to build the Python libraries. This means things like gcc and -devel packages which many Python libraries use to build library bindings. In contrast to this, DevStack has always been able to bootstrap itself from a blank system to a working OpenStack deployment, ensuring it has the right libraries, etc, installed to get everything working. If you look at all the job definitions, anything running DevStack has a dsvm in the job name; which referred to to "DevStack Virtual Machine"; or basically specifying what was installed on the host (and yes, the bare node had more installed, so it wasn't that bare at all!). Thus we don't want packages pre-installed for DevStack, because it hides actual devstack dependencies that we want explicitly defined. But the bare nodes, used for functional testing, were different -- there was an every-growing and not well-defined list of packages that were pre-installed on those nodes to make sure functional testing worked. In general, you don't want jobs relying on something like this; we want to be sure if jobs have a dependency, they require it explicitly. This is where a tool called bindep comes in. OpenStack has the concept of global requirements -- those Python dependencies that are common across all projects so version skew becomes somewhat manageable. This now has some extra information in the other-requirements.txt file, which lists the system-packages required to build the Python-packages in the requirements list. bindep knows how to look at this and get the right packages for the platform it is running on. Indeed -- remember how it was previously mentioned we want to minimise dependencies on external resources at runtime? Well we can pre-cache all of these packages onto the images, knowing that they are likely to be required by packages. How do we get the packages installed? The way we really should be doing it -- as part of the CI job. There is a macro called install-distro-packages which uses bindep to install those packages as required by the global-requirements list. The result -- no more need for the bare node type to run functional tests! In all cases we can start with essentially a blank image and all the dependencies to run the job are expressed by and within the job -- leading to a consistent and reproducible environment. Several things have broken as part of removing bare nodes -- this is actually a good thing because it means we have revealed areas where we were making assumptions in jobs about what the underlying platform provides; issues that get fixed by thinking about and ensuring we have correct dependencies bringing up jobs. There's a few other macros there that do things like provide MySQL/Postgres instances or setup other common job requirements. By splitting these out we also improve the performance of jobs who now only bring in the dependencies they need -- we don't waste time doing things like setting up databases for jobs that don't need it.
Conclusion While dealing with multiple providers, image-types and dependency chains has been a great effort for the infra team, to everyone's credit I don't think the project has really noticed much going on underneath. OpenStack CI has transitioned to a situation where there is a single image type for each platform we test that deploys unmodified across all our providers. We have better insight into our dependencies and better tools to manage them. This leads to greatly decreased maintenance burden, better consistency and better performance; all great things to bring to OpenStack CI!

1 April 2016

Norbert Preining: Searching for a job

Yesterday my contract with my university JAIST ended. I was working there as Associate Professor for 6.5 years, but the university decided not to extend my contract (financial problems, foreigner in Japan). That means, I am now out on the job market again not the greatest fun for an academic at my age. Thanks to JAIST for circumventing the ominous 5-year clause of permanent employment.
shuushoku My prime interests are academic work, that is research and teaching, where research is in a variety of fields, mostly mathematical logic, theoretical computer science, and formal methods. Recently I am working on an extension of the Curry-Howard correspondence to cover different calculi, in particular Hyper Sequent Calculi, and relate them to some kind of process algebra. For more details see the logic dedicated page. My other focal point is programming and software development, where I have contributed to many open source projects, and act as main developer of the TeX Live infrastructure and distribution system (tlmgr). Main languages are Perl, PHP, C, but also Python, Vala, Lisp, CafeOBJ, and whatever comes around. For more details see the software page. I recently became father, so I would prefer either an academic position in Hokuriku, Japan, or a software developer position where I can work (most of the time) remotely. In case one of the readers here has some hints, please drop me a line. In the meantime I enjoy my unemployed life with my baby!

31 March 2016

Antoine Beaupr : My free software activities, march 2016

Debian Long Term Support (LTS) This is my 4th month working on Debian LTS, started by Raphael Hertzog at Freexian. I spent half of the month away on a vacation so little work was done, especially since I tried to tackle rather large uploads like NSS and Xen. I also worked on the frontdesk shift last week.

Frontdesk That work mainly consisted of figuring out how to best help the security team with the last uploads to the Wheezy release. For those who don't know, Debian 7 Wheezy, or "oldstable", is going to be unsupported by the security team starting end of april, and the Debian 6 Squeeze (the previous LTS) is now unsupported. The PGP signatures on the archived release have started yielding expiration errors which can be ignored but that are really a strong reminder that it is really time to upgrade. So the LTS team is now working towards backporting a few security issues from squeeze to wheezy, and this is what I focused on during triage work. I have identified the following high priority packages I will work on after I complete my work on the Xen and NSS packages (detailed below):
  • libidn
  • icu
  • phpmyadmin
  • tomcat6
  • optipng
  • srtp
  • dhcpcd
  • python-tornado

Updates to NSS and Xen I have spent a lot of time testing and building packages for NSS and Xen. To be fair, Brian May did most of the work on the Xen packages, and I merely did some work to test the packages on Koumbit's infrastructure, something which I will continue doing in the next month. For NSS, wheezy and jessie are in this weird state where patches were provided to the security team all the way back in November yet were never tested. Since then, yet more issues came up and I worked hard to review and port patches for those new security issues to wheezy. I'll followup on both packages in the following month.

Other free software work

Android TL;DR: there's an even longer version of this with the step-by-step procedures and that I will update as time goes on in my wiki. I somehow inherited an Android phone recently, on a loan from a friend because the phone broke one too many times and she got a new one from her provider. This phone is a HTC One S "Ville", which is fairly old, but good enough to play with and give me a mobile computing platform to listen to podcasts, play music, access maps and create GPS traces. I was previously doing this with my N900, but that device is really showing its age: very little development is happening on it, the SDK is closed source and the device itself is fairly big, compared to the "Ville". Plus, the SIM card actually works on the Ville so, even though I do not have an actual contract with a cell phone provider (too expensive, too invasive on my privacy), I can still make emergency phone calls (911)! Plus, since there is good wifi on the machine, I can use it to connect to the phone system with the built-in SIP client, send text messages through SMS (thanks to VoIP.ms SMS support) or Jabber. I have also played around with LibreSignal, the free software replacement for Signal, which uses proprietary google services. Yes, the VoIP.ms SMS app also uses GCM, but hopefully that can be fixed. (While I was writing this, another Debian Developer wrote a good review of Signal so I am happy to skip that step. Go read that.)
See also my apps list for a more complete list of the apps I have installed on the phone. I welcome recommendations on cool free software apps I should use!
I have replaced the stock firmware on the phone with Cyanogenmod 12.1, which was a fairly painful experience, partly because of the difficult ambiance on the #cyanogenmod channel on Freenode, where I had extreme experiences: a brave soul helped me through the first flashing process for around 2 hours, nicely holding my hand at every step. Other times, I have seen flames and obtuse comments from people being vulgar, brutal, obnoxious, if not sometimes downright homophobic and sexist from other users. It is clearly a community that needs to fix their attitude. I have documented everything I could in details in this wiki page, in case others want to resuscitate their old phones, but also because I ended up reinstalling the freaking phone about 4 times, and was getting tired of forgetting how to do it every time. I am somewhat fascinated by Android: here is the Linux-based device that should save us all from the proprietary Apple nightmares of fenced in gardens and censorship. Yet, public Android downloads are hidden behind license agreements, even though the code itself is free, which has led fellow Debian developers to work on making libre rebuilds of Androids to workaround this insanity. But worse: all phones are basically proprietary devices down to the core. You need custom firmware to be loaded on the machine for it to boot at all, from the bootloader all the way down to the GSM baseband and Wifi drivers. It is a minefield of closed source software, and trying to run free software on there is a bit of a delusion, especially since the baseband has so much power over the phone. Still, I think it is really interesting to run free software on those machines, and help people that are stuck with cell phones get familiar with software freedom. It seems especially important to me to make Android developers aware of software freedom considering how many apps are available for free yet on which it is not possible to contribute significantly because the source code is not published at all, or published only on the Google Store, instead of the more open and public F-Droid repository which publishes only free software. So I did contribute. This month, I am happy to announce that I contributed to the following free software projects on Android: I have also reviewed the literature surrounding Telegram, a popular messaging app rival to Signal and Whatsapp. Oddly enough, my contributions to Wikipedia on that subject were promptly reverted which made me bring up the subject on the page's Talk page. This lead to an interesting response from the article's main editors which at least added that the "its security features have been contested by security researchers and cryptography experts". Considering the history of Telegram, I would keep people away from it and direct people to use Signal instead, even though Signal has similar metadata issues, mostly because Telegram's lack of response to the security issues that were outlined by fellow security researchers. Both systems suffer from a lack of federation as well, which is a shame in this era of increasing centralization. I am not sure I will put much more work in developing for Android for now. It seems like a fairly hostile platform to work on, and unless I have specific pain points I want to fix, it feels so much better to work on my own stuff in Debian. Which brings me to my usual plethora of free software projects I got stuck in this month.

IRC projects irssi-plugin-otr had a bunch of idiotic bugs lying around, and I had a patch that I hadn't submitted upstream from the Debian package, which needed a rebuild because the irssi version changed, which is a major annoyance. The version in sid is now a snapshot because upstream needs to make a new release but at least it should fix things for my friends running unstable and testing. Hopefully those silly uploads won't be necessary in the future. That's for the client side. On the server side, I have worked on updating the Atheme-services package to the latest version, which actually failed because the upstream libmowgli is missing release tags, which means the Debian package for it is not up-to-date either. Still, it is nice to have a somewhat newer version, even though it is not the latest and some bugs were fixed. I have also looked at making atheme reproducible but was surprised at the hostility of the upstream. In the end, it looks like they are still interested in patches, but they will be a little harder to deploy than for Charybdis, so this could take some time. Hopefully I will find time in the coming weeks to test the new atheme services daemon on the IRC network I operate.

Syncthing I have also re-discovered Syncthing, a file synchronization software. Amazingly, I was having trouble transferring a single file between two phones. I couldn't use Bluetooth (not sure why), the "Wifi sharing" app was available only on one phone (and is proprietary, and has a limit of 4MB files), and everything else requires an account, the cloud, or cabling. So. Just heading to f-droid, install syncthing, flash a few qr-codes around and voil : files are copied over! Pretty amazing: the files were actually copied over the local network, using IPv6 link-local addresses, encryption and the DHT. Which is a real geeky way of saying it's completely fast, secure and fast. Now, I found a few usability issues, so much that I wrote a whole usability story for the developers, which were really appreciative of my review. Some of the issues were already fixed, others were pretty minor. Syncthing has a great community, and it seems like a great project I encourage everyone to get familiar with.

Battery stats The battery-status project I mentionned previously has been merged with the battery-stats project (yes, the names are almost the same, which is confusing) and so I had to do some work to fix my Python graph script, which was accepted upstream and will be part of Debian officially from now on, which is cool. The previous package was unofficial. I have also noticed that my battery has a significantly than when I wrote the script. Whereas it was basically full back then, it seems now it has lost almost 15% of its capacity in about 6 months. According to the calculations of the script:
this battery will reach end of life (5%) in 935 days, 19:07:58.336480, on 2018-10-23 12:06:07.270290
Which is, to be fair, a good life: it will somewhat work for about three more years.

Playlist, git-annex and MPD in Python On top of my previously mentioned photos-import script, I have worked on two more small programs. One is called get-playlist and is an extension to git-annex to easily copy to the local git-annex repository all files present in a given M3U playlist. This is useful for me because my phone cannot possibly fit my whole MP3 collection, and I use playlists in GMPC to tag certain files, particularly the Favorites list which is populated by the "star" button in the UI. I had a lot of fun writing this script. I started using elpy as an IDE in Emacs. (Notice how Emacs got a new webpage, which is a huge improvement was had been basically unchanged since the original version, now almost 20 years old, and probably written by RMS himself.) I wonder how I managed to stay away from Elpy for so long, as it glues together key components of Emacs in an elegant and functional bundle:
  • Company: the "vim-like" completion mode i had been waiting for forever
  • Jedi: context-sensitive autocompletion for Python
  • Flymake: to outline style and syntax errors (unfortunately not the more modern Flycheck)
  • inline documentation...
In short, it's amazing and makes everything so much easier to work with that I wrote another script. The first program wouldn't work very well because some songs in the playlists had been moved, so I made another program, this time to repair playlists which refer to missing files. The script is simply called fix-playlists, and can operate transparently on multiple playlists. It has a bunch of heuristics to find files and uses a MPD server as a directory to search into. It can edit files in place or just act as a filter.

Useful snippets Writing so many scripts, so often, I figured I needed to stop wasting time always writing the same boilerplate stuff on top of every file, so I started publishing Yasnippet-compatible file snippets, in my snippets repository. For example, this report is based on the humble lts snippet. I also have a base license snippet which slaps the AGPLv3 license on top of a Python file. But the most interesting snippet, for me, is this simple script snippet which is a basic scaffolding for a commandline script that includes argument processing, logging and filtering of files, something which I was always copy-pasting around.

Other projects And finally, a list of interesting issues en vrac:
  • there's a new Bootstrap 4 theme for Ikiwiki. It was delivering content over CDNs, which is bad for privacy issues, so I filed an issue which was almost immediately fixed by the author!
  • I found out about BitHub, a tool to make payments with Bitcoins for patches submitted on OWS projects. It wasn't clear to me where to find the demo, but when I did, I quickly filed a PR to fix the documentation. Given the number of open PRs there and the lack of activity, I wonder if the model is working at all...
  • a fellow Debian Developer shared his photos workflow, which was interesting to me because I have my own peculiar script, which I never mentionned here, but which I ended up sharing with the author

Chris Lamb: Free software activities in March 2016

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):
Debian
  • Presented Reproducible Builds - fulfilling the original promise of free software at FOSSASIA '16.
  • Uploaded libfiu (0.94-4), adding a patch from Logan Rose to fix a FTBFS with ld --as-needed.
My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#44, #45, #46, #47).
LTS

This month I have been paid to work 7 hours on Debian Long Term Support (LTS). Whilst the LTS team will take over support from the Security Team on April 26, 2016, in the meantime I did the following:
  • Archived the squeeze distribution (via the FTPteam).
  • Assisted in preparing updates for python-django.
  • Helping end-users migrate to wheezy now that squeeze LTS has reached end-of-life.
FTP Team

As a Debian FTP assistant I ACCEPTed 143 packages: acme-tiny, berkshelf-api, circlator, cloud-utils, corsix-th, cronic, diaspora-installer, dub, dumb-init, firehol, firetools, flask-bcrypt, flask-oldsessions, flycheck, ganeti, geany-plugins, git-build-recipe, git-phab, gnome-shell-extension-caffeine, gnome-shell-extension-mediaplayer, golang-github-cheggaaa-pb, golang-github-coreos-ioprogress, golang-github-cyberdelia-go-metrics-graphite, golang-github-cznic-ql, golang-github-elazarl-goproxy, golang-github-hashicorp-hil, golang-github-mitchellh-go-wordwrap, golang-github-mvdan-xurls, golang-github-paulrosania-go-charset, golang-github-xeipuuv-gojsonreference, golang-github-xeipuuv-gojsonschema, grilo-plugins, gtk3-nocsd, herisvm, identity4c, lemonldap-ng, libisal, libmath-gsl-perl, libmemcached-libmemcached-perl, libplack-middleware-logany-perl, libplack-middleware-logwarn-perl, libpng1.6, libqmi, librdf-generator-http-perl, libtime-moment-perl, libvirt-php, libxml-compile-soap-perl, libxml-compile-wsdl11-perl, linux, linux-tools, mdk-doc, mesa, mpdecimal, msi-keyboard, nauty, node-addressparser, node-ansi-regex, node-argparse, node-array-find-index, node-base62, node-co, node-component-consoler, node-crypto-cacerts, node-decamelize, node-delve, node-for-in, node-function-bind, node-generator-supported, node-invert-kv, node-json-localizer, node-normalize-git-url, node-nth-check, node-obj-util, node-read-file, node-require-dir, node-require-main-filename, node-seq, node-starttls, node-through, node-uid-number, node-uri-path, node-url-join, node-xmlhttprequest-ssl, ocrmypdf, octave-netcdf, open-infrastructure-container-tools, osmose-emulator, pdal, pep8, pg-backup-ctl, php-guzzle, printrun, pydocstyle, pysynphot, python-antlr3, python-biom-format, python-brainstorm, python-django-adminsortable, python-feather-format, python-gevent, python-lxc, python-mongoengine, python-nameparser, python-pdal, python-pefile, python-phabricator, python-pika-pool, python-pynlpl, python-qtawesome, python-requests-unixsocket, python-saharaclient, python-stringtemplate3, r-cran-adegraphics, r-cran-assertthat, r-cran-bold, r-cran-curl, r-cran-data.table, r-cran-htmltools, r-cran-httr, r-cran-lazyeval, r-cran-mcmc, r-cran-openssl, r-cran-pbdzmq, r-cran-rncl, r-cran-uuid, rawtran, reel, ruby-certificate-authority, ruby-rspec-pending-for, ruby-ruby-engine, ruby-ruby-version, scribus-ng, specutils, symfony, tandem-mass, tdb, thrift, udfclient, vala, why3, wmaker, xdg-app & xiccd.

30 March 2016

Ian Wienand: Durable photo workflow

Ever since my kids were born I have accumulated thousands of digital happy-snaps and I have finally gotten to a point where I'm quite happy with my work-flow. I have always been extremely dubious of using any sort of external all-in-one solution to managing my photos; so many things seem to shut-down, cease development or disappear, all leaving you to have to figure out how to migrate to the next latest thing (e.g. Picasa shutting down). So while there is nothing complicated or even generic about them, there are a few things in my photo-scripts repo that might help others who like to keep a self-contained archive. Firstly I have a simple script to copy the latest photos from the SD card (i.e. those new since the last copy -- this is obviously very camera specific). I then split by date so I have a simple flat directory layout with each week's photos in it. With the price of SD cards and my rate of filling them up, I don't even bother wiping them at this point, but just keep them in the safe as a backup. For some reason I have a bit of a thing about geotagging all the photos so I know where I took them. Certainly some cameras do this today, but mine does not. I have a two-progned approach; I have a geotag script and then a small website easygeotag.info which quickly lets met translate a point on Google maps to exiv2 command-line syntax. Since I take a lot of photos in the same place, the script can store points by name in a small file sourced by the script. Adding comments to the photos is done with perhaps the lesser-known cousin of EXIF -- IPTC. Some time ago I wrote python bindings for libiptcdata and it has been working just fine ever since. Debian's python-iptcdata comes with a inbuilt script to set title and caption, which is easily wrapped. What I like about this is that my photos are in a simple directory layout, with all metadata embedded within the actual image files in very standarised formats that should be readable by anywhere I choose to host them. For sharing, I then upload to Flickr. I used to have a command-line script for this, but have found the web uploader works even better these days. It reads the IPTC data for titles and comments, and gets the geotag info for nice map displays. I manually coralle them into albums, and the Flickr "guest pass" is perfect for then sharing albums to friends and family without making them jump through hoops to register on a site to get access to the photos, or worse, host them myself. I consider Flickr a cache, because (even though I pay) I expect it to shut-down or turn evil at any time. Interestingly, their AI tagging is often quite accurate, and I imagine will only get better. This is nice extra metadata that you don't have to spend time on yourself. The last piece has always been the "hit by a bus" component of all this. Can anyone figure out access to all these photos if I suddenly disappear? I've tried many things here -- at one point I was using rdiff-backup to sync encrypted bundles up to AWS for example; but I very clearly found the problem in that when I forgot the keep the key safe and couldn't unencrypt any of my backups (let alone anyone else figuring all this out). Finally Google Nearline seems to be just what I want. It's off-site, redundant and the price is right; but more importantly I can very easily give access to the backup bucket to anyone with a Google address, who can then just hit a website to download the originals from the bucket (I left the link with my other "hit by a bus" bits and pieces). Of course what they then do with this data is their problem, but at least I feel like they have a chance. This even has an rsync like interface in the client, so I can quickly upload the new stuff from my home NAS (where I keep the photos in a RAID0). I've been doing this now for 350 weeks and have worked through some 25,000 photos. I used to get an album up every week, but as the kids get older and we're closer to family I now do it in batches about once a month. I do wonder if my kids will ever be interested in tagged and commented photos with pretty much their exact location from their childhood ... I doubt it, but it's nice to feel like I have a good chance of still having them if they do.

Bits from Debian: DebConf16 welcomes its first nine sponsors!

DebConf16 logo DebConf16 will take place in Cape Town, South Africa in July 2016. We strive to provide an intense working environment and enable good progress for Debian and for Free Software in general. We extend an invitation to everyone to join us and to support this event. As a volunteer-run non-profit conference, we depend on our sponsors. Nine companies have already committed to sponsor DebConf16! Let's introduce them: Our first Platinum sponsor is Hewlett Packard Enterprise (HPE). HPE is one of the largest computer companies in the world, providing a wide range of products and services, such as servers, storage, networking, consulting and support, software, and financial services. HPE is also a development partner of Debian, and provides hardware for port development, Debian mirrors, and other Debian services (hardware donations are listed in the Debian machines page). Our first Gold sponsor is Valve, a company developing games, social entertainment platform, and game engine technologies. Our second Gold sponsor is Google, the technology company specialized in Internet-related services as online advertising and search engine. Rusbitech (developers of the Astra Linux Debian derivative), credativ (a service-oriented company focusing on open-source software and also a Debian development partner), Catalyst (a company offering IT solutions using open source software), the Bern University of Applied Sciences (with over 7,000 students enrolled, located in the Swiss capital), and Texas Instruments (the global semiconductor company) are our four Silver sponsors. And last but not least, the open source company Univention has agreed to support us as Bronze-level. Become a sponsor too! Would you like to become a sponsor? Do you know of or work in a company or organization that may consider sponsorship? Please have a look at our sponsorship brochure (or a summarized flyer), in which we outline all the details and describe the sponsor benefits. For further details, feel free to contact us through sponsors@debconf.org, and visit the DebConf16 website at https://debconf16.debconf.org.

29 March 2016

Michal Čihař: Weekly phpMyAdmin contributions 2016-W12

Last week we've finally released phpMyAdmin 4.6.0. That also means that 4.5.5 is out of maintenance now (however 4.4.15 and 4.0.10 are still supported if you're stuck with older PHP or MySQL versions for some reason). The biggest focus was spent on handling incoming issues and checking if they are still present in 4.6.0 what in most cases proven not to be true. Handled issues:

Filed under: English phpMyAdmin 0 comments

28 March 2016

Lars Wirzenius: Obnam user survey, 2016

In January and February of 2016 I ran an Obnam user survey. I'm not a statistician, but here is my analysis of the results. Executive summary: Obnam is slow, buggy, and the name is bad. But they'd like to buy stickers and t-shirts. Method I wrote up a long list of questions about things I felt were of interest to me. I used Google Forms to collect responses, and exported them as a CSV file, and analysed based on that. I used Google Forms, even though it is not free software, as it was the easiest service I got to work that also seemed it'd be nice for people to use. I could have run the survey using Ikiwiki, but it wouldn't have been nearly as nice. I could have found and hosted some free software for this, but that would have been much more work. Most questions had free form text responses, and this was both good and bad. It was good, because many of the responses included things I could never have expected. It was bad, because it took me a lot more time and effort to process those. I think next time I'll keep the number of free text responses down. For some of the questions, I hand-processed the responses to a more or less systematic form, in order to count things with a bit of code. For others, I did not, and show the full list of responses (I'm lazy, we don't need a survey to determine that). The responses See http://code.liw.fi/obnam/survey-2016.html for the responses, after hand-processing. For the questions for which it makes sense, a script has tabulated the various responses and calculated percentages. I haven't produced graphs, as I don't know how to do that easily. (Maybe next time I'll enlist the help of statisticians.) Conclusions

26 March 2016

Jonathan McDowell: Dr Stoll: Or how I learned to stop worrying and love the GPL

[I wrote this as part of BelFOSS but I think it s worth posting here.] My Free Software journey starts with The Cuckoo s Egg. Back in the early 90s a family friend suggested I might enjoy reading it. He was right; I was fascinated by the world of interconnected machines it introduced me to. That helped start my involvement in FidoNet, but it also got me interested in Unix. So when I saw a Linux book at the Queen s University bookshop (sadly no longer with us) with a Slackware CD in the back I had to have it. The motivation at this point was to have a low cost version of Unix I could run on the PC hardware I already owned. I had no knowledge of the GNU Project before this point, and as I wasn t a C programmer I had no interest in looking at the source code. I spent some time futzing around with it and that partition (I was dual booting with DOS 6.22) fell into disuse. It wasn t until I d learnt some C and turned up to university, which provided me with an internet connection and others who were either already using Linux or interested in doing so, that I started running a Linux box full time. Once I was doing that I became a lot more interested in the Open Source side of the equation. Rather than running a closed operating system that even the API for wasn t properly specified (or I wouldn t have needed my copy of Undocumented DOS) I had the complete source to both the underlying OS and all the utilities that it was using. For someone doing a computer science degree this was invaluable. Minix may have been the OS discussed in the OS Design module I studied, but Linux was a much more feature complete option that I was running on my desktop and could also peer under the hood of. In my professional career I ve always welcomed the opportunities to work with Open Source. A long time ago I experienced a particularly annoying issue when writing a device driver under QNX. The documentation didn t seem to match the observed behaviour of the subsystem I was interfacing with. However due to licensing issues only a small number of people in the organisation were able to actually look at the QNX source. So I ended up wasting a much more senior engineer s time with queries like I think it s actually doing x, y and z instead of a, b and c; can you confirm? . Instances where I can look directly at the source code myself make me much more productive. Commercial development also started to make me more understanding of the Free Software nature of the code I was running. It wasn t just the ability to look at the code which was useful, but also the fact there was no need to reinvent the wheel. Need a base OS to build an appliance on? Debian ensures that the main component is Free for all usage. No need to worry about rolling your own compilers, base libraries etc. From a commercial perspective that allows you to concentrate on the actual product. And when you hit problems, the source is available and you can potentially fix it yourself or at least more easily find out if there s been a fix for that issue released (being able to see code development in version control systems rather than getting a new upstream release with a whole heap on unrelated fixes in it really helps with that). I had thus progressed from using FLOSS because it was free-as-in-beer, to appreciating the benefits of Open Source in my own learning and employment experiences, to a deeper understanding of the free-as-in-speech benefits that could be gained. However at this point I was still thinking very much from a developer mindset. Even my thoughts about how users can benefit from Free Software were in the context of businesses being able to easily switch suppliers or continue to maintain legacy software because they had the source to their systems available. One of the major factors that has helped me to see beyond this is the expansion of the Internet of Things (IoT). With desktop or server software there is by and large a choice about what to use. This is not the case with appliances. While manufacturers will often produce a few revisions of software for their devices, usually eventually there is a newer and shiny model and the old one is abandoned. This is problematic for many reasons. For example, historically TVs have been long lived devices (I had one I bought second hand that happily lasted me 7+ years). However the smart capabilities of the TV I purchased in 2012 are already of limited usefulness, and LG have moved on to their current models. I have no intention of replacing the device any time soon, so have had to accept it is largely acting as a dumb display. More serious is the lack of security updates. For a TV that doesn t require a network connection to function this is not as important, but the IoT is a trickier proposition. For example Matthew Garrett had an awful experience with some intelligent light bulbs, which effectively circumvented any home network security you might have set up. The manufacturer s defence? No longer manufactured or supported. It s cases like these that have slowly led me to a more complete understanding of the freedom that Free Software truly offers to users. It s not just about cost free/low cost software. It s not just about being able to learn from looking at the source to the programs you are running. It s not even about the freedom to be able to modify the programs that we use. It s about giving users true Freedom to use and modify their devices as they see fit. From this viewpoint it is much easier to understand the protections against Tivoization that were introduced with GPLv3, and better appreciate the argument sometimes made that the GPL offers more freedom than BSD style licenses.

25 March 2016

Guido G nther: More sandboxing

More sandboxing When working on untrusted code or data it's impossible to predict what happens when one does a: 
bundle install --path=vendor
or 
npm install
Does this phone out your private SSH and GPG keys? Does a 
evince Downloads/justdownloaded.pdf
try to exploit the PDF viewer? While you can run stuff in separate virtual machines this can get cumbersome. libvirt-sandbox to the rescue! It allows to sandbox applications using libvirt's virtualization drivers. It took us a couple of years (The ITP is from 2012) but we finally have it in Debian's NEW queue. When libvirt-sandbox creates a sandbox it uses your root filesystem mounted read only by default so you have access to all installed programs (this can be changed with the --root option though). It can use either libvirt's QEMU or LXC drivers. We're using the later in the examples below: So in order to make sure the above bundler call has no access to your $HOME you can use: 
sudo virt-sandbox \
   -m ram:/tmp=10M \
   -m ram:$HOME=10M \
   -m ram:/var/run/screen=1M \
   -m host-bind:/path/to/your/ruby-stuff=/path/to/your/ruby-stuff \
   -c lxc:/// \
   -S $USER \
   -n rubydev-sandbox \
   -N dhcp,source=default \
   /bin/bash
This will make your $HOME unaccessible by mounting a tmpfs over it and using separate network, ipc, mount, pid and utc namespaces allowing you to invoke bundler with less worries. /path/to/your/ruby-stuff is bind mounted read-write into the sandbox so you can change files there. Bundler can fetch new gems using libvirt's default network connection. And for the PDF case: 
sudo virt-sandbox \
  -m ram:$HOME=10M \
  -m ram:/dev/shm=10M \
  -m host-bind:$HOME/Downloads=$HOME/Downloads \
  -c lxc:/// \
  -S $USER \
  -n evince-sandbox \
  --env="DISPLAY=:0" \
  --env="XAUTHORITY=$XAUTHORITY" \
  /usr/bin/evince Downloads/justdownloaded.pdf
Note that the above example shares /tmp with the sandbox in order to give it access to the X11 socket. A better isolation can probably be achieved using xpra or xvnc but I haven't looked into this yet. Besides the command line program virt-sandbox there's also the library libvirt-sandbox which makes it simpler to build new sandboxing applications. We're not yet shipping virt-sandbox-service (a tool to provision sandboxed system services) in the Debian packages since it's RPM distro specific. Help on porting this to Debian is greatly appreciated.

23 March 2016

John Goerzen: Free cars, sunsets, and Kansas

Will you have a car I can borrow? I asked. Sure. No charge. There s a sign telling you where to find the key. It is pretty common for small airports to have a car for a pilot to borrow when flying in. This lets a person go into town for lunch, or visit friends. And it s usually free, with a can to donate a few bucks or a polite request to fill up the tank when you re done. Still, when I had called ahead to ask about flying into the airport in a small town in north-central Kansas, I hadn t expected to be told to just waltz into the place and take the key. But they had no staff at the airport most of the time. So, to me another person from a small town it made perfect sense. Somehow, because of that phone call, this town I had visited once, maybe 25 years ago, seemed instantly familiar. My mom grew up in a small town near there. She wanted me to see where she grew up, to meet some people that meant a lot to her. As it s quite a distance from home, I offered to fly her there. So, Laura, mom, and I climbed into a Cessna one morning for the flight northwest. We touched down at the airport, and I pulled the plane up to the little terminal building. Smith Center, KS airport terminal After I took care of parking the plane, I went to find the car. Except the car was missing. Some other pilot had flown in the same day and was using it, according to the logbook on the desk. I called the number on a sign which rang to the sheriff s office and they confirmed it. According to the logbook, this was only the third time that car had been driven since Thanksgiving. Were we stuck at the airport a few miles out of down? Nope. Mom called the people we were going to meet, a wonderful couple in their upper 80s. They drove out to pick us up. I m rather glad the car was gone, because I had such a great time visiting with them. Norris told me about the days when the state highways were gravel how they d have to re-blade them every few days due to all the traffic. I heard about what happened when the people in that community heard of some folks in Africa in need of car equipment they modified a tractor to fit in a shipping container and shipped it to Africa, along with a lot of books, blankets, supplies, and anything else needed to fill up a huge shipping container. Sounds like something people around here would do. We drove around a couple of the small towns. The town my mom grew up in has seen better days. Its schools closed years ago, the old hotel whose owner gave her piano lessons is condemned, and many houses have been lost. But the town lives on. A new community center was built a few years ago. The grain elevator is expanding. Every time a business on Main Street closes, the grocery store expands a little bit: it s now a grocery store with a little hardware store and a little restaurant mixed in. The mall , as the locals jokingly call it. And, of course, two beautiful small churches still meet every Sunday. Here s the one my mom attended as a child. IMG_7085 We drove past the marker at the geographic center of the contiguous United States. Norris saw some other visitors, rolled down his windows, and treated them and us to an unexpected story of the time thousands of people banded together to completely build a house in a single day, just down the road. Smiles all around. So here I was, nearly 200 miles from home, in an unfamiliar town but one where I could just feel the goodness. After spending a few hours with these people, I felt like they were old friends. As I flew us home, I spotted one of my favorite Kansas sights: a beautiful sunset. From the plane, it almost looks like the land at the horizon turns blue like the ocean, and above it the last hint of sun paints the canvas-sky. In this week of controversy, politics, and reports of violence, it reminds me that we all get the privilege of sharing this beautiful Earth. I didn t ask anybody on that trip about their politics, religion, or opinions on any of the divisive issues of the day. Whether they agree with me on those things or not is irrelevant. I got to spend a day with good-hearted and delightful people, so I flew back with a smile. IMG_7099

Petter Reinholdtsen: Full battery stats collector is now available in Debian

Since this morning, the battery-stats package in Debian include an extended collector that will collect the complete battery history for later processing and graphing. The original collector store the battery level as percentage of last full level, while the new collector also record battery vendor, model, serial number, design full level, last full level and current battery level. This make it possible to predict the lifetime of the battery as well as visualise the energy flow when the battery is charging or discharging. The new tools are available in /usr/share/battery-stats/ in the version 0.5.1 package in unstable. Get the new battery level graph and lifetime prediction by running: 
/usr/share/battery-stats/battery-stats-graph /var/log/battery-stats.csv
Or select the 'Battery Level Graph' from your application menu. The flow in/out of the battery can be seen by running (no menu entry yet): 
/usr/share/battery-stats/battery-stats-graph-flow
I'm not quite happy with the way the data is visualised, at least when there are few data points. The graphs look a bit better with a few years of data. A while back one important feature I use in the battery stats collector broke in Debian. The scripts in /usr/lib/pm-utils/power.d/ were no longer executed. I suspect it happened when Jessie started using systemd, but I do not know. The issue is reported as bug #818649 against pm-utils. I managed to work around it by adding an udev rule to call the collector script every time the power connector is connected and disconnected. With this fix in place it was finally time to make a new release of the package, and get it into Debian. If you are interested in how your laptop battery is doing, please check out the battery-stats in Debian unstable, or rebuild it on Jessie to get it working on Debian stable. :) The upstream source is available from github. As always, patches are very welcome.

Ingo Juergensmann: Xen randomly crashing server

It's a long story... an oddessey of almost two years... But to start from the beginning: Back then I rented a server at Hetzner until they decided to bill for every IP address you got from them. I got a /26 in the past and so I would have to pay for every IP address of that subnet in addition to the server rent of 79.- EUR/month. That would have meant nearly doubling the monthly costs. So I moved with my server from Hetzner to rrbone Net, which offered me a /26 on a rented Cisco C200 M2 server for a competitve price. After migrating the VMs from Hetzner to rrbone with the same setup that was running just fine at Hetzner I experienced spontaneous reboots of the server, sometimes several times per day and in short time frame. The hosting provider was very, very helpful in debugging this like exchanging the memory, setting up a remote logging service for the CIMC and such. But in the end we found no root cause for this. The CIMC logs showed that the OS was rebooting the machine. Anyway, I then bought my own server and exchanged the Cisco C200 by my own hardware, but the reboots still happen as before. Sometimes the servers runs for weeks, sometimes the server crashes 4-6 times a day, but usually it's like a pattern: when it crashes and reboots, it will do that again within a few hours and after the second reboot the chances are high that the server will run for several days without a reboot - or even weeks. The strange thing is, that there are absolutely no hints in the logs, neither syslog or in the Xen logs, so I assume that's something quite deep in the kernel that causes the reboot. Another hint is, that the reboots fairly often happened, when I used my Squid proxy on one of the VMs to access the net. I'm connecting for example by SSH with portforwarding to one VM, whereas the proxy runs on another VM, which led to network traffic between the VMs. Sometimes the server crashed on the very firsts proxy requests. So, I exchanged Squid by tinyproxy or other proxies, moved the proxy from one VM to that VM I connect to using SSH, because I thought that the inter-VM traffic may cause the machine to reboot. Moving the proxy to another virtual server I rented at another hosting provider to host my secondary nameserver did help a little bit, but with no real hard proof and statistics, just an impression of mine. I moved from xm toolstack to xl toolstack as well, but didn't help either. The reboots are still happening and in the last few days very frequent. Even with the new server I exchanged the memory, used memory mirroring as well, because I thought that it might be a faulty memory module or something, but still rebooting out of the blue. During the last weekend I configured grub to include "noreboot" command line and then got my first proof that somehow the Xen network stack is causing the reboots: This is a screenshot of the IPMI console, so it's not showing the full information of that kernel oops, but as you can see, there are most likely such parts involved like bridge, netif, xenvif and the physical igb NIC. Here's another screenshot of a crash from this night: Slightly different information, but still somehow network involved as you can see in the first line (net_rx_action). So the big question is: is this a bug Xen or with my setup? I'm using xl toolstack, the xl.conf is basically the default, I think:
## Global XL config file ##

# automatically balloon down dom0 when xen doesn't have enough free
# memory to create a domain
autoballoon=0

# full path of the lockfile used by xl during domain creation
#lockfile="/var/lock/xl"

# default vif script
#vif.default.script="vif-bridge"
With this the default network scripts of the distribution (i.e. Debian stable) should be used. The network setup consists of two brdiges:
auto xenbr0
iface xenbr0 inet static
address 31.172.31.193
netmask 255.255.255.192
gateway 31.172.31.254
bridge_ports eth0
pre-up brctl addbr xenbr0 auto xenbr1
iface xenbr1 inet static
address 192.168.254.254
netmask 255.255.255.0
pre-up brctl addbr xenbr1
There are some more lines to that config like setting up some iptables rules with up commands and such. But as you can see my eth0 NIC is part of the "main" xen bridge with all the IP addresses that are reachable from the outside. The second bridge is used for internal networking like database connections and such. I would rather like to use a netconsole to capture the full debug output in case of a new crash, but unfortunately this only works until the bridge is brought up and in place:
[ 0.000000] Command line: placeholder root=UUID=c3....22 ro debug ignore_loglevel loglevel=7 netconsole=port@31.172.31.193/eth0,514@5.45.x.y/e0:ac:f1:4c:y:x
[ 32.565624] netpoll: netconsole: local port $port
[ 32.565683] netpoll: netconsole: local IPv4 address 31.172.31.193
[ 32.565742] netpoll: netconsole: interface 'eth0'
[ 32.565799] netpoll: netconsole: remote port 514
[ 32.565855] netpoll: netconsole: remote IPv4 address 5.45.x.y
[ 32.565914] netpoll: netconsole: remote ethernet address e0:ac:f1:4c:y:x
[ 32.565982] netpoll: netconsole: device eth0 not up yet, forcing it
[ 36.126294] netconsole: network logging started
[ 49.802600] netconsole: network logging stopped on interface eth0 as it is joining a master device
So, the first question is: how to use netconsole with an interface that is used on a bridge? The second question is: is the setup with two bridges with Xen ok? I've been using this setup for years now and it worked fairly well on the Hetzner server as well, although I used there xm toolstack with a mix of bridge and routed setup, because Hetzner didn't like to see the MAC addresses of the other VMs on the switch and shut the port down if that happens.
Kategorie:
Debian
Tags:
Debian
Xen
Server
Bug

22 March 2016

Michal Čihař: Weekly phpMyAdmin contributions 2016-W11

Last week was a bit less working for me as I had few days off. The 4.6.0 is about to be released in the time as this blog post will be published, so the focus was obviously on fixing bugs. Most of annoying issues I've fixed were caused by our SQL Parser which we need to generate some export statements and which didn't properly understand some more complex syntax. Most of the export issues are now fixed, but there are still situations where it doesn't parse the query right, usually this is true for nested SELECT statements or some more complex constructs (CASE). Anyway it's being improved and works better with every release. Handled issues:

Filed under: English phpMyAdmin 0 comments

17 March 2016

Patrick Matth i: Debian Jessie 8.3: Short howto for Corosync+Pacemaker Active/Passive Cluster with two nodes and DRBD/LVM

Hello, since I had to change my old heartbeat v1 setup to an more modern Corosync+Pacemaker setup, because heartbeat v1 does not support systemd (it first looks like it is working, but it fails on service start/stops), I want to share a simple setup: First you have to activate the jessie-backports repository, because the cluster stack is not available/broken in Debian Jessie. Install the required packages with:
apt-get install -t jessie-backports libqb0 fence-agents pacemaker corosync pacemaker-cli-utils crmsh drbd-utils
After that configure your DRBD and LVM (VG+LV) on it (there are enough tutorials for it). Then deploy this configuration to /etc/corosync/corosync.conf:
totem
version: 2
token: 3000
token_retransmits_before_loss_const: 10
clear_node_high_bit: yes
crypto_cipher: none
crypto_hash: none
transport: udpu
interface
ringnumber: 0
bindnetaddr: 192.168.99.0

logging
to_logfile: yes
logfile: /var/log/corosync/corosync.log
debug: off
timestamp: on
logger_subsys
subsys: QUORUM
debug: off

quorum
provider: corosync_votequorum
two_node: 1
wait_for_all: 1
nodelist
node
ring0_addr: node1-1

node
ring0_addr: node1-2

Both nodes require a passwordless keypair, which is copied to the another node, so that you can ssh from one to each other. Then you can start with crm configure:
property stonith-enabled=no
property no-quorum-policy=ignore
property default-resource-stickiness=100 primitive DRBD_r0 ocf:linbit:drbd params drbd_resource= r0 op start interval= 0 timeout= 240 \
op stop interval= 0 timeout= 100 \
op monitor role=Master interval=59s timeout=30s \
op monitor role=Slave interval=60s timeout=30s
primitive LVM_r0 ocf:heartbeat:LVM params volgrpname= data1 op monitor interval= 30s
primitive SRV_MOUNT_1 ocf:heartbeat:Filesystem params device= /dev/mapper/data1-lv1 directory= /srv/storage fstype= ext4 options= noatime,nodiratime,nobarrier op monitor interval= 40s primitive IP-rsc ocf:heartbeat:IPaddr2 params ip= 123.123.123.123 nic= eth0 cidr_netmask= 24 meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive IPInt-rsc ocf:heartbeat:IPaddr2 params ip= 192.168.99.4 nic= eth1 cidr_netmask= 24 meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart primitive MariaDB-rsc lsb:mysql meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive Redis-rsc lsb:redis-server meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive Memcached-rsc lsb:memcached meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive PHPFPM-rsc lsb:php5-fpm meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive Apache2-rsc lsb:apache2 meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive Nginx-rsc lsb:nginx meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart group APCLUSTER LVM_r0 SRV_MOUNT_1 IP-rsc IPInt-rsc MariaDB-rsc Redis-rsc Memcached-rsc PHPFPM-rsc Apache2-rsc Nginx-rsc
ms ms_DRBD_APCLUSTER DRBD_r0 meta master-max= 1 master-node-max= 1 clone-max= 2 clone-node-max= 1 notify= true colocation APCLUSTER_on_DRBD_r0 inf: APCLUSTER ms_DRBD_APCLUSTER:Master
order APCLUSTER_after_DRBD_r0 inf: ms_DRBD_APCLUSTER:promote APCLUSTER:start commit
The last (bold marked) lines made me some headache. In short they define that the DRBD device on the active node has to be the primary one and that it is required to start the APCLUSTER on the host, since the LVM, filesystem and services require to access its data. Just a short copy paste howto for an simple use case with not so much deep explanaitions..

15 March 2016

Michal Čihař: Weekly phpMyAdmin contributions 2016-W10

Last week was quite equally split between refactoring and bug fixing. As we're getting closer to 4.6.0 release more people give feedback on it and some corner case issues are being discovered. However the biggest challenge was strange bug that was reported long time ago - it affected only Italian translation on Windows server. After lot of debugging, I've realized that trying to encode Cardinalit </td> to json (using json_encode) causes this problem. Obviously there have to be more conditions met as this string only doesn't cause it, however removing it or placing x between and < fixes the problem. In the end I've workarounded it in our code and hopefully I will get to create proper bugreport for php so that it can be fixed upstream as well. The refactoring was mostly focused on encoding conversion part which is now all embedded in the Encoding class. It was quite simplified and should also speed up pages a bit as it does the initialization only when needed not on every phpMyAdmin's page. Handled issues:

Filed under: English phpMyAdmin 0 comments

14 March 2016

Julian Andres Klode: Dropping SHA-1 support in APT

Tomorrow is the anniversary of Caesar s assassination APT will see a new release, turning of support for SHA-1 checksums in Debian unstable and in Ubuntu xenial, the upcoming LTS release. While I have no knowledge of an imminent attack on our use of SHA1, Xenial (Ubuntu 16.04 LTS) will be supported for 5 years, and the landscape may change a lot in the next 5 years. As disabling the SHA1 support requires a bit of patching in our test suite, it s best to do that now rather than later when we re forced to do it. This will mean that starting tomorrow, some third party repositories may stop working, such as the one for the web browser I am writing this with. Debian Derivatives should be mostly safe for that change, if they are registered in the Consensus, as that has checks for that. This is a bit unfortunate, but we have no real choice: Technical restrictions prevent us from just showing a warning in a sensible way. There is one caveat, however: GPG signatures may still use SHA1. While I have prepared the needed code to reject SHA1-based signatures in APT, a lot of third party repositories still ship Release files signed with signatures using SHA-1 as the digest algorithms. Some repositories even still use 1024-bit DSA keys. I plan to enforce SHA2 for GPG signatures some time after the release of xenial, and definitely for Ubuntu 16.10, so around June-August (possibly during DebConf). For xenial, I plan to have a SRU (stable release update) in January to do the same (it s just adding one member to an array). This should give 3rd party providers a reasonable time frame to migrate to a new digest algorithm for their GPG config and possibly a new repository key. Summary
Filed under: Debian, Ubuntu

13 March 2016

Vincent Sanders: I changed my mind, Erase and rewind

My recent rack design turned out to simply not be practical. It did not hold all the SBC I needed it to and most troubling accessing connectors was impractical. I was forced to remove the enclosure from the rack and go back to piles of SBC on a shelf.

View of the acrylic being laser cut through the heavily tinted window
This sent me back to the beginning of the design process. The requirement for easy access to connectors had been compromised on in my first solution because I wanted a compact 1U size. This time I returned to my initial toast rack layout but retaining the SBC inside their clip cases.

By facing the connectors downwards and providing basic cable management the design should be much more practical.

My design process is to use the QCAD package to create layered 2D outlines which are then converted from DXF into toolpaths with Lasercut CAM software. The toolpaths are then uploaded to the laser cutter directly from the PC running Lasercut.

Assembled sub rack enclosureDespite the laser cutters being professional grade systems the Lasercut software is a continuous cause of issues for many users, it is the only closed source piece of software in the production process and it has a pretty poor user interface. On this occasion my main issue with it was my design was quite large at 700mm by 400mm which caused the software to crash repeatedly. I broke the design down into two halves and this allowed me to continue.

Once I defeated the software the design was laser cut from 3mm clear extruded acrylic. The assembled is secured with 72 off M3 nuts and bolts. The resulting construction is very strong and probably contains much more material than necessary.

One interesting thing I discovered is that in going from a 1U enclosure holding 5 units to a 2U design holding 11 units I had increased the final weight from 320g to 980g and when all 11 SBC are installed that goes up to a whopping 2300g. Fortunately this is within the mechanical capabilities of the material but it is the heaviest thing I have ever constructed from 3mm acrylic.

bolted into the rack and operatingOnce installed in the rack with all SBC inserted and connected this finally actually works and provides a practical solution. The self is finally clear of SBC and has enough space for all the other systems I need to accommodate for various projects.

As usual the design files are all freely available though I really cannot see anyone else needing to replicate this.

Antoine Beaupr : State of Mapping on the Debian Desktop

TL;DR: Mapbox Studio classic is not as good as Tilemill, Mapbox studio is very promising, but you still need to signup for access, and kosmtik seems to be working right now. Use kosmtik and help getting it in Debian.

The requirement I have been following the development of fascinating tools from the Development seed people, now seemingly all focused on the Mapbox brand. They created what seemed to me a revolutionary desktop tool called Tilemill. Tilemill allowed you to create custom map stylesheets on your desktop to outline specific areas or patterns or "things". My interest was to create outdoor maps - OSM has pretty good biking maps, but no generic outdoor tiles out there (I need stuff for skiing, canoeing, biking)... Mapbox has Mapbox outdoors but that's a paid plan as well. Oh and there's a place to download Garmin data files especially made for outdoors as well, and that could be loaded in Qmapshack, but they don't have coverage in Canada at all. Besides, I would like to print maps, I know, crazy... So I have been looking forward to seeing Tilemill packaged in Debian given how annoying it is to maintain Node apps (period). Unfortunately, by the time Debian people figured out all the Node dependencies, the Tilemill project stopped and has been stalled since 2012. It seems the Mapbox people have now been working on other products, and in the meantime, the community, scratching their heads, just switched to other projects.

Overview of alternatives I wrote this long post to the Debian bugtrackers to try to untangle all of this, but figured this would be interesting to a wider community than the narrow group of people working on Javascript packages, so I figured I would send this on Debian planet. So Here's a summary of what happened so far, after Tilemill development stopped. Hang on to your tiles boys and girls, there's a lot going on!

Mapbox Studio classic Mapbox people have released a new product in September 2014 named Mapbox studio classic. The code is still freely available and seems to be a fork of tilemill. Mapbox classic still has releases on github, last one is from November 2015. It looks like Mapbox studio classic has some sort of Mapbox.com lock-in, and there are certainly new copyright issues, if only with the bundled fonts, but it could probably be packaged after addressing those issues. There is an ITP for Mapbox Studio classic as well.

Mapbox Studio Then there's Mapbox Studio, which is a full rewrite of Mapbox Studio classic. You need to "signup" somehow to get access, even though parts of the code are free, namely the Mapbox GL studio project. It is an interesting project because it aims to make all this stuff happen in a web browser, which means it "should" work everywhere. Unfortunately for us, it means it doesn't work anywhere without a signup form, so that's out for me at least. There is an [ITP for Mapbox-studio][] yet it is unclear to me what that one means because the source code to Mapbox-studio doesn't seem to be available, as far as i can tell (and the ITP doesn't say either). That is actually the ITP for Mapbox studio classic.

Kosmtik The Openstreetmap-carto developers have mostly switched to kosmtik instead of Mapbox. Kosmtik is another Node desktop app that seems fairly lightweight and mostly based on plugins. Ross has an ITP for kosmtik. The package is waiting on other node dependencies to be uploaded (yes, again).

The future of Tilemill And there's still this RFP for tilemill, which should probably be closed now because the project seems dead and plenty of alternatives exist. I wonder if node some dependencies that were packaged for Tilemill actually now need to be removed from Debian, because they have become useless leaf packages... I am leaving the Tilemill RFP open for someone to clean that up.

CartoCSS Oh, and finally one could mention another Mapbox project, Carto, a command line CSS tools that implements some sort of standard CSS language that all those tools end up using to talk to Mapnik, more or less. There are no RFPs for that.

Next.

Previous.