My work colleagues know me well as a Free/Libre software zealot, constantly pointing out to them how people should behave, how FLOSS software trumps commercial software and how this is the only way forward. This for the last 20 odd years. It's a strain to argue this repeatedly: at various times, I have been asked to set out more clearly why I use FLOSS, what the advantages are, why and how to contribute to FLOSS software.
"We are creating a world that all may enter without privilege or prejudice accorded by race, economic power, military force, or station of birth. We are creating a world where anyone, anywhere may express his or her beliefs, no matter how singular, without fear of being coerced into silence or conformity. Your legal concepts of property, expression, identity, movement, and context do not apply to us. They are all based on matter, and there is no matter here ... In our world, whatever the human mind may create can be reproduced and distributed infinitely at no cost. The global conveyance of thought no longer requires your factories to accomplish." [John Perry Barlow - Declaration of the independence of cyberspace 1996 https://www.eff.org/cyberspace-independence]
That's some of it right there: I was seduced by a modem and the opportunities it gave. I've lived in this world since 1994, come to appreciate it and never really had the occasion to regret it.
I'm involved in the Debian community - which is very much a "do-ocracy" - and I've lived with Debian GNU Linux since 1995 and not had much cause to regret that either, though I do regret that force of circumstance has meant that I can't contribute as much as I'd like. Pretty much every machine I touch ends up running Debian, one way or the other, or should do if I had my way. Digging through my emails since then on the various mailing lists - some of them are deeply technical, though fewer these days: some are Debian political: most are trying to help people with problems / report successes or, occasionally thanks and social chit chat. Most people in the project have never met me - though that's not unusual in an organisation with a thousand developers spread worldwide - and so the occasional chance to talk to people in real life is invaluable.
The crucial thing is that there is common purpose and common intelligence - however crazy mailing list flame wars can get sometimes - and committed, caring people. Some of us may be crazy zealots, some picky and argumentative - Debian is what we have in common, pretty much.
It doesn't depend on physical ability. Espy (Joel Klecker) was one of our best and brightest until his death at age 21: almost nobody knew he was dying until after his death. My own physical limitations are pretty much irrelevant provided I can type.
It does depend on collaboration and the strange, dysfunctional family that is our community and the wider FLOSS community in which we share and in which some of us have multiple identities in working with different projects. This is going to end up too long for Planet Debian - I'll end this post here and then continue with some points on how to contribute and why employers should let their employers work on FLOSS.
I bought some awful WiFi lightbulbs a few months ago. The short version: they introduced terrible vulnerabilities on your network, they violated the GPL and they were also just bad at being lightbulbs. Since then I've bought some other Internet of Things devices, and since people seem to have a bizarre level of fascination with figuring out just what kind of fractal of poor design choices these things frequently embody, I thought I'd oblige.
Today we're going to be talking about the KanKun SP3, a plug that's been around for a while. The idea here is pretty simple - there's lots of devices that you'd like to be able to turn on and off in a programmatic way, and rather than rewiring them the simplest thing to do is just to insert a control device in between the wall and the device andn ow you can turn your foot bath on and off from your phone. Most vendors go further and also allow you to program timers and even provide some sort of remote tunneling protocol so you can turn off your lights from the comfort of somebody else's home.
The KanKun has all of these features and a bunch more, although when I say "features" I kind of mean the opposite. I plugged mine in and followed the install instructions. As is pretty typical, this took the form of the plug bringing up its own Wifi access point, the app on the phone connecting to it and sending configuration data, and the plug then using that data to join your network. Except it didn't work. I connected to the plug's network, gave it my SSID and password and waited. Nothing happened. No useful diagnostic data. Eventually I plugged my phone into my laptop and ran adb logcat, and the Android debug logs told me that the app was trying to modify a network that it hadn't created. Apparently this isn't permitted as of Android 6, but the app was handling this denial by just trying again. I deleted the network from the system settings, restarted the app, and this time the app created the network record and could modify it. It still didn't work, but that's because it let me give it a 5GHz network and it only has a 2.4GHz radio, so one reset later and I finally had it online.
The first thing I normally do to one of these things is run nmap with the -O argument, which gives you an indication of what OS it's running. I didn't really need to in this case, because if I just telnetted to port 22 I got a dropbear ssh banner. Googling turned up the root password ("p9z34c") and I was logged into a lightly hacked (and fairly obsolete) OpenWRT environment.
It turns out that here's a whole community of people playing with these plugs, and it's common for people to install CGI scripts on them so they can turn them on and off via an API. At first this sounds somewhat confusing, because if the phone app can control the plug then there clearly is some kind of API, right? Well ha yeah ok that's a great question and oh good lord do things start getting bad quickly at this point.
I'd grabbed the apk for the app and a copy of jadx, an incredibly useful piece of code that's surprisingly good at turning compiled Android apps into something resembling Java source. I dug through that for a while before figuring out that before packets were being sent, they were being handed off to some sort of encryption code. I couldn't find that in the app, but there was a native ARM library shipped with it. Running strings on that showed functions with names matching the calls in the Java code, so that made sense. There were also references to AES, which explained why when I ran tcpdump I only saw bizarre garbage packets.
But what was surprising was that most of these packets were substantially similar. There were a load that were identical other than a 16-byte chunk in the middle. That plus the fact that every payload length was a multiple of 16 bytes strongly indicated that AES was being used in ECB mode. In ECB mode each plaintext is split up into 16-byte chunks and encrypted with the same key. The same plaintext will always result in the same encrypted output. This implied that the packets were substantially similar and that the encryption key was static.
Some more digging showed that someone had figured out the encryption key last year, and that someone else had written some tools to control the plug without needing to modify it. The protocol is basically ascii and consists mostly of the MAC address of the target device, a password and a command. This is then encrypted and sent to the device's IP address. The device then sends a challenge packet containing a random number. The app has to decrypt this, obtain the random number, create a response, encrypt that and send it before the command takes effect. This avoids the most obvious weakness around using ECB - since the same plaintext always encrypts to the same ciphertext, you could just watch encrypted packets go past and replay them to get the same effect, even if you didn't have the encryption key. Using a random number in a challenge forces you to prove that you actually have the key.
At least, it would do if the numbers were actually random. It turns out that the plug is just calling rand(). Further, it turns out that it never calls srand(). This means that the plug will always generate the same sequence of challenges after a reboot, which means you can still carry out replay attacks if you can reboot the plug. Strong work.
But there was still the question of how the remote control works, since the code on github only worked locally. tcpdumping the traffic from the server and trying to decrypt it in the same way as local packets worked fine, and showed that the only difference was that the packet started "wan" rather than "lan". The server decrypts the packet, looks at the MAC address, re-encrypts it and sends it over the tunnel to the plug that registered with that address.
That's not really a great deal of authentication. The protocol permits a password, but the app doesn't insist on it - some quick playing suggests that about 90% of these devices still use the default password. And the devices are all based on the same wifi module, so the MAC addresses are all in the same range. The process of sending status check packets to the server with every MAC address wouldn't take that long and would tell you how many of these devices are out there. If they're using the default password, that's enough to have full control over them.
There's some other failings. The github repo mentioned earlier includes a script that allows arbitrary command execution - the wifi configuration information is passed to the system() command, so leaving a semicolon in the middle of it will result in your own commands being executed. Thankfully this doesn't seem to be true of the daemon that's listening for the remote control packets, which seems to restrict its use of system() to data entirely under its control. But even if you change the default root password, anyone on your local network can get root on the plug. So that's a thing. It also downloads firmware updates over http and doesn't appear to check signatures on them, so there's the potential for MITM attacks on the plug itself. The remote control server is on AWS unless your timezone is GMT+8, in which case it's in China. Sorry, Western Australia.
It's running Linux and includes Busybox and dnsmasq, so plenty of GPLed code. I emailed the manufacturer asking for a copy and got told that they wouldn't give it to me, which is unsurprising but still disappointing.
The use of AES is still somewhat confusing, given the relatively small amount of security it provides. One thing I've wondered is whether it's not actually intended to provide security at all. The remote servers need to accept connections from anywhere and funnel decent amounts of traffic around from phones to switches. If that weren't restricted in any way, competitors would be able to use existing servers rather than setting up their own. Using AES at least provides a minor obstacle that might encourage them to set up their own server.
Overall: the hardware seems fine, the software is shoddy and the security is terrible. If you have one of these, set a strong password. There's no rate-limiting on the server, so a weak password will be broken pretty quickly. It's also infringing my copyright, so I'd recommend against it on that point alone.
In a prior post,
I gave an overview of the OpenStack CI system and how jobs were
started. In that I said
(It is a gross oversimplification, but for the purposes of OpenStack
CI, Jenkins is pretty much used as a glorified ssh/scp wrapper. Zuul
Version 3, under development, is working to remove the need for
Jenkins to be involved at all).
Well some recent security issues with Jenkins and other changes has
led to a roll-out of what is being called Zuul 2.5, which has indeed
removed Jenkins and makes extensive use of Ansible as the basis for
running CI tests in OpenStack. Since I already had the diagram, it
seems worth updating it for the new reality.
OpenStack CI Overview
While previous post was really focused on the image-building
components of the OpenStack CI system, overview is the same but more
focused on the launchers that run the tests.
The process starts when a developer uploads their code to
gerrit via the git-review tool. There is no further action
required on their behalf and the developer simply waits for
results of their jobs.
Gerrit provides a JSON-encoded "fire-hose" output of everything
happening to it. New reviews, votes, updates and more all get sent
out over this pipe. Zuul is the overall scheduler
that subscribes itself to this information and is responsible for
managing the CI jobs appropriate for each change.
Zuul has a configuration
that tells it what jobs to run for what projects. Zuul can do lots
of interesting things, but for the purposes of this discussion we
just consider that it puts the jobs it wants run into gearman
for a launcher to consume. gearman is a
job-server; as they explain it "[gearman] provides a generic
application framework to farm out work to other machines or
processes that are better suited to do the work". Zuul puts into
gearman basically a tuple (job-name,node-type) for each
job it wants run, specifying the unique job name to run and what
type of node it should be run on.
A group of Zuul launchers
are subscribed to gearman as workers. It is these Zuul
launchers that will consume the job requests from the queue and
actually get the tests running. However, a launcher needs two
things to be able to run a job a job definition (what to actually
do) and a worker node (somewhere to do it).
The first part what to do is provided by job-definitions
stored in external YAML files. The Zuul launcher knows how to
process these files (with some help from Jenkins Job Builder, which
despite the name is not outputting XML files for Jenkins to
consume, but is being used to help parse templates and macros
within the generically defined job definitions). Each Zuul
launcher gets these definitions pushed to it constantly by Puppet,
thus each launcher knows about all the jobs it can run
automatically. Of course Zuul also knows about these same job
definitions; this is the job-name part of the tuple we said it
put into gearman.
The second part somewhere to run the test takes some more
explaining. To the next point...
Several cloud companies donate capacity in their clouds for
OpenStack to run CI tests. Overall, this capacity is managed by a
customized management tool called nodepool
(you can see the details of this capacity at any given time by
checking the nodepool configuration).
Nodepool watches the gearman queue and sees what requests are
coming out of Zuul. It looks at node-type of jobs in the queue
(i.e. what platform the job has requested to run on) and decides
what types of nodes need to start and which cloud providers have
capacity to satisfy demand.
Nodepool will start fresh virtual machines (from images built daily
as described in the prior post), monitor their start-up and, when
they're ready, put a new "assignment job" back into gearman with
the details of the fresh node. One of the active Zuul launchers
will pick up this assignment job and register the new node to
itself.
At this point, the Zuul launcher has what it needs to actually get
jobs started. With an fresh node registered to it and waiting for
something to do, the Zuul launcher can advertise its ability to
consume one of the waiting jobs from the gearman queue. For
example, if a ubuntu-trusty node is provided to the Zuul
launcher, the launcher can now consume from gearman any job it
knows about that is intended to run on an ubuntu-trusty node
type. If you're looking at the launcher code
this is driven by the NodeWorker class you can see this being
created in response to an assignment via
LaunchServer.assignNode.
To actually run the job where the "job hits the metal" as it were
the Zuul launcher will dynamically construct an Ansible playbook to run. This
playbook is a concatenation of common setup and teardown operations
along with the actual test scripts the jobs wants to run. Using
Ansible to run the job means all the flexibility an orchestration
tool provides is now available to the launcher. For example, there
is a custom console streamer
library that allows us to live-stream the console output for the
job over a plain TCP connection, and there is the possibility to
use projects like ARA
for visualisation of CI runs. In the future, Ansible will allow
for better coordination when running multiple-node testing jobs
after all, this is what orchestration tools such as Ansible are
made for! While the Ansible run can be fairly heavyweight
(especially when you're talking about launching thousands of jobs
an hour), the system scales horizontally with more launchers able
to consume more work easily.
When checking your job results on logs.openstack.org you will
see a _zuul_ansible directory now which contains copies of the
inventory, playbooks and other related files that the launcher used
to do the test run.
Eventually, the test will finish. The Zuul launcher will put the
result back into gearman, which Zuul will consume (log copying
is interesting but a topic for another day). The testing node will
be released back to nodepool, which destroys it and starts all over
again nodes are not reused and also have no sensitive details on
them, as they are essentially publicly accessible. Zuul will wait
for the results of all jobs for the change and post the result back
to Gerrit; it either gives a positive vote or the dreaded negative
vote if required jobs failed (it also handles merges to git, but
that is also a topic for another day).
Work will continue within OpenStack Infrastructure to further enhance
Zuul; including better support for multi-node jobs and "in-project"
job definitions (similar to the https://travis-ci.org/ model);
for full details see the spec.
Time passes. I had left several (too many?) pending things to be done un the quiet weeks between the end of the lective semestre and the beginning of muy Summer trip to Winter. But Saturday gets closer every moment... And our long trip to the South begins.
Among many other things, I wanted to avance with some Deb an stuff - both packaging and WRT keyring analysis. I want to contacto some people I left pending interactions with, but honestly, that will only come face to face un Capetown.
As to "real life", I hace too many pending issues at work to even begin with; I hope to get some time at South frica todo do some decent UNAM sysadmining. Also, I want to play the idea of using Git for my students' workflow (handing in projects and assignments, at least)... This can be interesting to talk with the Deb an colleagues about, actually.
As a Masters student, I'm making good advances, and will probably finish muy class work next semester, six months ahead of schedule, but muy thesis work so far has progressed way slower than what I'd like. I have at least a better defined topic and approach, so I'll start the writing phase soon.
And the personal life? Family? I am more complete and happy than ever before. My life su completely different from two years ago. Yes, that was obvious. But it's also the only thing I can come up with. Having twin babies (when will they make the transition from "babies" to "kids"? No idea... We will find out as it comes) is more than beautiful, more than great. Our life has changed in every possible aspect. And yes, I admire my loved Regina for all of the energy and love she puts on the babies... Life is asymetric, I am out for most of the day... Mommy is always there.
As I said, happier than ever before.
etckeeper was a sleeper success for me. I created it, wrote one
blog post about it, installed it on all my computers, and mostly forgot
about it, except when I needed to look something up in the git history
of /etc it helpfully maintains. It's a minor project.
Then I started getting patches porting it to many other version control
systems, and other linux distributions, and fixing problems, and adding
features. Mountains and mountains of patches over time.
And then I started hearing about distributions that install it by default.
(Though Debian for some reason never did so I keep having to install it
everywhere by hand.)
Writing this blog post, I noticed etckeeper had accumulated enough patches
from other people to warrant a new release. That happens pretty regularly.
So it's still a minor project as far as I'm concerned, but quite a few
people seem to be finding it useful. So it goes with free software.
Next: twenty years of free software -- part 3 myrepos
Thanks to our lovely translators, Debian developers reference has now added Russian and Italian version. Hope people who use those languages can look into Debian development more deeply.
And also added fancy CSS for pages, hope you like it...
We have looong queue in BTS, so please help us to make it better with patch (or just article), not just post your opinion. I cannot squash those since I'm not good at English... ;-), so please give your hands.
And maybe it'd be better to use date for versioning, IMHO (not 3.4.18 but 20160621).
Jenny Lawson, who blogs as The Bloggess,
is best known for a combination of being extremely funny and being
extremely open about her struggles with mental illness (a combination of
anxiety and depression, alongside a few other problems). Her first book
primarily told the story of her family, childhood, and husband.
Furiously Happy is a more random and disconnected book, but insofar
as it has a theme, it's about surviving depression, anxiety, and other
manifestations of your brain being an asshole.
I described Lawson's previous book, Let's
Pretend This Never Happened, as the closest thing I've found to a
stand-up comedy routine in book form. Furiously Happy is very
similar, but it lacks the cohesiveness of a routine. Instead, it feels
like a blog collection: a pile of essays with only some vague themes and
not a lot of continuity from essay to essay.
This doesn't surprise me. Second books are very different than first
books, particularly second books by someone whose writing focus is not
writing books, and particularly for non-fiction. I feel like Let's
Pretend This Never Happened benefited from drawing on Lawson's full life
experience to form the best book she could write. When that became wildly
popular, everyone of course wanted a second book, including me. But when
the writing is this personal, the second book is, out of necessity, partly
leftovers. Lawson's recent experiences don't generate as much material as
her whole life up to the point of the first book.
That said, there is a bit of a theme, and the title fits it. Early in the
book, Lawson describes how, after the death of a friend and a bout of
depression, she decided to be furiously, vehemently happy to get back at
the universe, to spite its attempts to destroy her mood. It's one of the
best bits in this book. The surrounding philosophy is about embracing the
moment, enjoying the hell out of everything that one can enjoy, and taking
a lot of chances.
A lot of the stories in this book come after the beginning of Lawson's
fame and popularity. She has book tours, a vacation tour of Australia,
and a community of people from her blog. That, of course, doesn't make
the depression and anxiety any better; indeed, it provides a lot of
material for her anxiety to work with. Lawson talks a lot about
surviving, about how important that community is to her, about not
believing your brain when it lies to you. This isn't as uniformly funny
as her first book, and sometimes it feels a bit too much like an earnest
pep talk. But there are also moments of insightful life philosophy mixed
into the madcap adventures and stream-of-consciousness wild associations.
Lawson also does for anxiety what Allie Brosh
does for depression: make the severe form of it relatable to people who
have not suffered from it. I was particularly struck by her description
of flying: the people around her are getting nervous and anxious as the
plane starts to take off, and she's finally able to relax because her
anxiety focused on all the things she had to do in order to get onto the
right plane at the right time. Once she didn't have to make any decisions
or do anything other than sit in one place, her anxiety let go. I don't
have any type of clinical anxiety, but I was able to identify with that
moment of relief and its contrast with anxiety in a deeper way than with
other descriptions.
Furiously Happy is a bit more serious and earnest, and I'm not sure
it worked as well. I liked Lawson's first book better; this felt more
like a blog archive. But she's still funny, entertaining, and delightful,
and I'm happy to support her with a book purchase. Start with either
her blog or Let's Pretend This Never
Happened if you're new to Lawson, but if you're already a fan, here's more
of her writing.
Rating: 7 out of 10
I'm forty years old. I've been developing free software for twenty years.
A decade ago, I wrote a series of posts about my first
ten years of free software,
looking back over projects I'd developed. These retrospectives seem even more
valuable in retrospect; there are things in the old posts that jog my memory,
and other details I've forgotten by now.
So, I'm doing it again. Over the next two weeks (with a week off in the middle
for summer vacation), I'll be writing one post each day about a free software
project I've developed in the past decade.
We begin with Ikiwiki. I started it 10 years ago, and still use it
to this day; it's the engine that builds this website, and nearly all
my other websites, as well as wikis and websites belonging to tons and
tons of other projects, like NetBSD, X.org, Freedesktop.org, FreedomBox and
many other users.
Indeed I'm often reading a website and find myself wondering "hey.. is this
using Ikiwiki?", and glance at the html and find that yes, it is. So,
Ikiwiki is a reasonably successful and widely used peice of software,
at least in its niche.
More important to me, it was a static site generator before we knew what
those were. It wasn't the first, but it broke plenty of new ground. I'm
particularly proud of the way it combines a wiki with blogging support
seamlessly, and the incremental updating of the static pages including
updating wikilinks and backlinks. Some of these and other features are
still pretty unique to Ikiwiki despite the glut of other static site
generators available now.
Ikiwiki is written in Perl, which was great for getting lots of other
contributions (including many of its 113 plugins), but has also held it
back some lately. There are less Perl programmers these days. And over the
past decade, concurrency has become very important, but Ikiwiki's
implementation is stubbornly single threaded, and multithreading such a
Perl program is a losing propoisition. I occasionally threaten to rewrite
it in Haskell, but I doubt I will.
Ikiwiki has several developers now, and I'm the least active of them. I
stepped back because I can't write Perl very well anymore, and am mostly
very happy with how Ikiwiki works, so only pop up now and then when
something annoys me.
Next: twenty years of free software -- part 2 etckeeper
I'm back from the GTK hackfest in Toronto, Canada and mostly recovered from
jetlag, so it's time to write up my notes on what we discussed there.
Despite the hackfest's title, I was mainly there to talk about non-GUI
parts of the stack, and technologies that fit more closely in what could
be seen as the freedesktop.org platform than they do in GNOME. In particular,
I'm interested in Flatpak as a way to deploy self-contained "apps" in a
freedesktop-based, sandboxed runtime environment layered over
the Universal Operating System and
its many derivatives,
with both binary and source compatibility with other GNU/Linux distributions.
I'm mainly only writing about discussions I was directly involved in:
lots of what sounded like good discussion about the actual graphics toolkit
went over my head completely :-)
More notes, mostly from Matthias
Clasen, are available on the GNOME wiki.
In no particular order:
Thinking with portals
We spent some time discussing Flatpak's portals, mostly on Tuesday.
These are the components that expose a subset of desktop functionality
as D-Bus services that can be used by contained applications: they are
part of the security boundary between a contained app and the rest of the
desktop session. Android's intents are a similar concept seen elsewhere.
While the portals are primarily designed for Flatpak, there's no real
reason why they couldn't be used by other app-containment solutions
such as Canonical's Snap.
One major topic of discussion was their overall design and layout. Most
portals will consist of a UX-independent part in Flatpak itself, together
with a UX-specific implementation of any user interaction the portal
needs. For example, the portal for file selection has a D-Bus service
in Flatpak, which interacts with some UX-specific service that will
pop up a standard UX-specific "Open" dialog for GNOME and probably
other GTK environments, that dialog is in (a branch of) GTK.
A design principle that was reiterated in this discussion is that
the UX-independent part should do as much as possible, with the
UX-specific part only carrying out the user interactions that need
to comply with a particular UX design (in the GTK case, GNOME's design).
This minimizes the amount of work that needs to be redone for other
desktop or embedded environments, while still ensuring that the other
environments can have their chosen UX design. In particular, it's
important that, as much as possible, the security- and performance-sensitive
work (such as data transport and authentication) is shared between all
environments.
The aim is for portals to get the user's permission to carry out actions,
while keeping it as implicit as possible, avoiding an "are you sure?" step
where feasible. For example, if an application asks to open a file,
the user's permission is implicitly given by them selecting the file
in the file-chooser dialog and pressing OK: if they do not want this
application to open a file at all, they can deny permission by cancelling.
Similarly, if an application asks to stream webcam data, the UX we expect
is for GNOME's Cheese app (or a similar non-GNOME app) to appear, open
the webcam to provide a preview window so they can see what they are about
to send, but not actually start sending the
stream to the requesting app until the user has pressed a "Start" button.
When defining the API "contracts" to be provided by applications in that
situation, we will need to be clear about whether the provider is expected
to obtain confirmation like this: in most cases I would anticipate that
it is.
One security trade-off here is that we have to have a small amount of
trust in the providing app. For example, continuing the example of Cheese
as a webcam provider, Cheese could (and perhaps should) be a contained
app itself, whether via something like Flatpak, an LSM like AppArmor
or both. If Cheese is compromised somehow, then whenever it is running, it
would be technically possible for it to open the webcam, stream video and
send it to a hostile third-party application. We concluded that this is an
acceptable trade-off: each application needs to be trusted with the privileges
that it needs to do its job, and we should not put up barriers that are easy
to circumvent or otherwise serve no purpose.
The main (only?) portal so far is the file chooser, in which the contained
application asks the wider system to show an "Open..." dialog, and if the user
selects a file, it is returned to the contained application through a FUSE
filesystem, the document portal. The reference implementation of the UX for
this is in GTK, and is basically a GtkFileChooserDialog. The intention is
that other environments such as KDE will substitute their own equivalent.
Other planned portals include:
image capture (scanner/camera)
opening a specified URI
this needs design feedback on how it should work for non-http(s)
sharing content, for example on social networks (like Android's Sharing menu)
proxying joystick/gamepad input (perhaps via Wayland or FUSE, or perhaps
by modifying libraries like SDL with a new input source)
network proxies (GProxyResolver) and availability (GNetworkMonitor)
contacts/address book, probably vCard-based
notifications, probably based on freedesktop.org Notifications
video streaming (perhaps using Pinot, analogous to PulseAudio but for video)
Environment variables
GNOME on Wayland currently has a problem with environment variables:
there are some traditional ways to set environment variables for X11
sessions or login shells using shell script fragments (/etc/X11/Xsession.d,
/etc/X11/xinit/xinitrc.d, /etc/profile.d), but these do not apply to
Wayland, or to noninteractive login environments like cron and
systemd --user. We are also keen to avoid requiring a Turing-complete shell
language during session startup, because it's difficult to reason about
and potentially rather inefficient.
Some uses of environment variables can be dismissed as unnecessary or even
unwanted, similar to the statement in Debian Policy 9.9: "A program must not
depend on environment variables to get reasonable defaults." However,
there are two common situations where environment variables can be necessary
for proper OS integration: search-paths like $PATH, $XDG_DATA_DIRS and
$PYTHONPATH (particularly necessary for things like Flatpak), and
optionally-loaded modules like $GTK_MODULES and $QT_ACCESSIBILITY
where a package influences the configuration of another package.
There is a stopgap solution in GNOME's gdm display manager,
/usr/share/gdm/env.d, but this is gdm-specific and
insufficiently expressive to provide the functionality needed by
Flatpak: "set XDG_DATA_DIRS to its specified default value if unset,
then add a couple of extra paths".
pam_env comes closer PAM is run at every transition from "no user logged
in" to "user can execute arbitrary code as themselves" but it doesn't
support .d fragments, which are required if we want distribution packages
to be able to extend search paths. pam_env also turns off per-user
configuration by default, citing security concerns.
I'll write more about this when I have a concrete proposal for how to solve it.
I think the best solution is probably a PAM module similar to pam_env but
supporting .d directories, either by modifying pam_env directly or
out-of-tree, combined with clarifying what the security concerns for
per-user configuration are and how they can be avoided.
Relocatable binary packages
On Windows and OS X, various GLib APIs automatically discover where the
application binary is located and use search paths relative to that;
for example, if C:\myprefix\bin\app.exe is running, GLib might put
C:\myprefix\share into the result of g_get_system_data_dirs(),
so that the application can ask to load app/data.xml from the data
directories and get C:\myprefix\share\app\data.xml. We would like
to be able to do the same on Linux, for example so that the apps in a
Flatpak or Snap package can be constructed from RPM or dpkg packages without
needing to be recompiled for a different --prefix, and so that other
third-party software packages like the games on Steam and gog.com can
easily locate their own resources.
Relatedly, there are currently no well-defined semantics for what happens
when a .desktop file or a D-Bus .service file has Exec=./bin/foo.
The meaning of Exec=foo is well-defined (it searches $PATH) and the
meaning of Exec=/opt/whatever/bin/foo is obvious. When this came up in
D-Bus previously, my assertion was that the meaning should be the same as
in .desktop files, whatever that is.
We agreed to propose that the meaning of a non-absolute path in a .desktop
or .service file should be interpreted relative to the directory
where the .desktop or .service file was found: for example, if
/opt/whatever/share/applications/foo.desktop says Exec=../../bin/foo,
then /opt/whatever/bin/foo would be the right thing to execute.
While preparing a mail to the freedesktop and D-Bus mailing lists proposing
this, I found that I had
proposed the same thingalmost 2 years ago...
this time I hope I can actually make it happen!
Flatpak and OSTree bug fixing
On the way to the hackfest, and while the discussion moved to topics that I
didn't have useful input on, I spent some time fixing up the Debian packaging
for Flatpak and its dependencies. In particular, I did my first upload
as a co-maintainer of bubblewrap,
uploaded ostree to unstable (with the known limitation that the grub, dracut
and systemd integration is missing for now
since I haven't been able to test it yet), got
most of the way through packaging Flatpak 0.6.5 (which I'll upload soon),
cherry-picked the right patches to make ostree compile on Debian 8 in an effort
to make backports trivial, and spent some time disentangling
a flatpak test failure
which was breaking the Debian package's installed-tests.
I'm still looking into
ostree test failures on little-endian MIPS,
which I was able to reproduce on a Debian porterbox just before the end of the
hackfest.
OSTree + Debian
I also had some useful conversations with developers from
Endless, who recently opened up a version of their
OSTree build scripts
for public access. Hopefully that information brings me a bit closer to
being able to publish a walkthrough for
how to deploy a simple Debian derivative using OSTree
(help with that is very welcome of course!).
GTK life-cycle and versioning
The life-cycle of GTK releases has already been
mentioned here and elsewhere, and there are some interesting responses
in the comments on my earlier blog post.
It's important to note that what we
discussed at the hackfest is only a proposal: a hackfest discussion between
a subset of the GTK maintainers and a small number of other GTK users
(I am in the latter category)
doesn't, and shouldn't, set policy for all of GTK or for all of GNOME. I
believe the intention is that the GTK maintainers will discuss the proposals
further at GUADEC, and make a decision after that.
As I said before, I hope that being more realistic about API and ABI
guarantees can avoid GTK going too far towards either of the possible
extremes: either becoming unable to advance because it's too constrained
by compatibility, or breaking applications because it isn't constrained
enough. The current situation, where it is meant to be compatible within
the GTK 3 branch but in practice applications still sometimes break,
doesn't seem ideal for anyone, and I hope we can do better in future.
Acknowledgements
Thanks to everyone involved, particularly:
Matthias Clasen, who organised the hackfest and took a lot of notes
Allison Lortie, who provided on-site cat-herding and led us to some
excellent restaurants
Red Hat Inc., who provided the venue (a conference room
in their Toronto office), snacks, a lot of coffee, and several participants
my employers Collabora Ltd., who sponsored my travel and accomodation
This year a significant number of students are working on RTC-related projects as part of Google Summer of Code, under the umbrella of the Debian Project. You may have already encountered some of them blogging on Planet or participating in mailing lists and IRC.
WebRTC plugins for popular CMS and web frameworks
There are already a range of pseudo-WebRTC plugins available for CMS and blogging platforms like WordPress, unfortunately, many of them are either not releasing all their source code, locking users into their own servers or requiring the users to download potentially untrustworthy browser plugins (also without any source code) to use them.
Mesut is making plugins for genuinely free WebRTC with open standards like SIP. He has recently created the WPCall plugin for WordPress, based on the highly successful DruCall plugin for WebRTC in Drupal.
Keerthana has started creating a similar plugin for MediaWiki.
What is great about these plugins is that they don't require any browser plugins and they work with any server-side SIP infrastructure that you choose. Whether you are routing calls into a call center or simply using them on a personal blog, they are quick and convenient to install. Hopefully they will be made available as packages, like the DruCall packages for Debian and Ubuntu, enabling even faster installation with all dependencies.
Would you like to try running these plugins yourself and provide feedback to the students? Would you like to help deploy them for online communities using Drupal, WordPress or MediaWiki to power their web sites? Please come and discuss them with us in the Free-RTC mailing list.
You can read more about how to run your own SIP proxy for WebRTC in the RTC Quick Start Guide.
Finding all the phone numbers and ham radio callsigns in old emails
Do you have phone numbers and other contact details such as ham radio callsigns in old emails? Would you like a quick way to data-mine your inbox to find them and help migrate them to your address book?
Jaminy is working on Python scripts to do just that. Her project takes some inspiration from the Telify plugin for Firefox, which detects phone numbers in web pages and converts them to hyperlinks for click-to-dial. The popular libphonenumber from Google, used to format numbers on Android phones, is being used to help normalize any numbers found. If you would like to test the code against your own mailbox and address book, please make contact in the #debian-data channel on IRC.
A truly peer-to-peer alternative to SIP, XMPP and WebRTC
The team at Savoir Faire Linux has been busy building the Ring softphone, a truly peer-to-peer solution based on the OpenDHT distribution hash table technology.
Several students (Simon, Olivier, Nicolas and Alok) are actively collaborating on this project, some of them have been fortunate enough to participate at SFL's offices in Montreal, Canada. These GSoC projects have also provided a great opportunity to raise Debian's profile in Montreal ahead of DebConf17 next year.
Linux Desktop Telepathy framework and reSIProcate
Another group of students, Mateus, Udit and Balram have been busy working on C++ projects involving the Telepathy framework and the reSIProcate SIP stack. Telepathy is the framework behind popular softphones such as GNOME Empathy that are installed by default on the GNU/Linux desktop.
I previously wrote about starting a new SIP-based connection manager for Telepathy based on reSIProcate. Using reSIProcate means more comprehensive support for all the features of SIP, better NAT traversal, IPv6 support, NAPTR support and TLS support. The combined impact of all these features is much greater connectivity and much greater convenience.
The students are extending that work, completing the buddy list functionality, improving error handling and looking at interaction with XMPP.
Streamlining provisioning of SIP accounts
Currently there is some manual effort for each user to take the SIP account settings from their Internet Telephony Service Provider (ITSP) and transpose these into the account settings required by their softphone.
Pranav has been working to close that gap, creating a JAR that can be embedded in Java softphones such as Jitsi, Lumicall and CSipSimple to automate as much of the provisioning process as possible. ITSPs are encouraged to test this client against their services and will be able to add details specific to their service through Github pull requests.
The project also hopes to provide streamlined provisioning mechanisms for privately operated SIP PBXes, such as the Asterisk and FreeSWITCH servers used in small businesses.
Improving SIP support in Apache Camel and the Jitsi softphone
Apache Camel's SIP component and the widely known Jitsi softphone both use the JAIN SIP library for Java.
Nik has been looking at issues faced by SIP users in both projects, adding support for the MESSAGE method in camel-sip and looking at why users sometimes see multiple password prompts for SIP accounts in Jitsi.
If you are trying either of these projects, you are very welcome to come and discuss them on the mailing lists, Camel users and Jitsi users.
GSoC students at DebConf16 and DebConf17 and other events
Many of us have been lucky to meet GSoC students attending DebConf, FOSDEM and other events in the past. From this year, Google now expects the students to complete GSoC before they become eligible for any travel assistance. Some of the students will still be at DebConf16 next month, assisted by the regular travel budget and the diversity funding initiative. Nik and Mesut were already able to travel to Vienna for the recent MiniDebConf / LinuxWochen.at
As mentioned earlier, several of the students and the mentors at Savoir Faire Linux are based in Montreal, Canada, the destination for DebConf17 next year and it is great to see the momentum already building for an event that promises to be very big.
Explore the world of Free Real-Time Communications (RTC)
If you are interesting in knowing more about the Free RTC topic, you may find the following resources helpful:
Try Lumicall and Ring apps for Android with your freinds
RTC mentoring team 2016
We have been very fortunate to build a large team of mentors around the RTC-themed projects for 2016. Many of them are first time GSoC mentors and/or new to the Debian community. Some have successfully completed GSoC as students in the past. Each of them brings unique experience and leadership in their domain.
Helping GSoC projects in 2016 and beyond
Not everybody wants to commit to being a dedicated mentor for a GSoC student. In fact, there are many ways to help without being a mentor and many benefits of doing so.
Simply looking out for potential applicants for future rounds of GSoC and referring them to the debian-outreach mailing list or an existing mentor helps ensure we can identify talented students early and design projects around their capabilities and interests.
Testing the projects on an ad-hoc basis, greeting the students at DebConf and reading over the student wikis to find out where they are and introduce them to other developers in their area are all possible ways to help the projects succeed and foster long term engagement.
Google gives Debian a USD $500 grant for each student who completes a project successfully this year. If all 2016 students pass, that is over $10,000 to support Debian's mission.
As some of the world knows full well by now, I've been noodling with Go
for a few years, working through its pros, its cons, and thinking a lot
about how humans use code to express thoughts and ideas. Go's got a lot of
neat use cases, suited to particular problems, and used in the right place,
you can see some clear massive wins.
I've started writing Debian tooling in Go, because it's a pretty natural fit.
Go's fairly tight, and overhead shouldn't be taken up by your operating system.
After a while, I wound up hitting the usual blockers, and started to build up
abstractions. They became pretty darn useful, so, this blog post is announcing
(a still incomplete, year old and perhaps API changing) Debian package for Go.
The Go importable name is pault.ag/go/debian. This contains a lot of utilities
for dealing with Debian packages, and will become an edited down "toolbelt"
for working with or on Debian packages.
Module Overview
Currently, the package contains 4 major sub packages. They're a changelog
parser, a control file parser, deb file format parser, dependency parser
and a version parser. Together, these are a set of powerful building blocks
which can be used together to create higher order systems with reliable
understandings of the world.
changelog
The first (and perhaps most incomplete and least tested) is a changelog file
parser.. This provides the
programmer with the ability to pull out the suite being targeted in the
changelog, when each upload was, and the version for each. For example, let's
look at how we can pull when all the uploads of Docker to sid took place:
funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/changelogs/main/d/docker.io/unstable_changelog")iferr!=nilpanic(err)allEntries,err:=changelog.Parse(resp.Body)iferr!=nilpanic(err)for_,entry:=rangeallEntriesfmt.Printf("Version %s was uploaded on %s\n",entry.Version,entry.When)
The output of which looks like:
Version 1.8.3~ds1-2 was uploaded on 2015-11-04 00:09:02 -0800 -0800
Version 1.8.3~ds1-1 was uploaded on 2015-10-29 19:40:51 -0700 -0700
Version 1.8.2~ds1-2 was uploaded on 2015-10-29 07:23:10 -0700 -0700
Version 1.8.2~ds1-1 was uploaded on 2015-10-28 14:21:00 -0700 -0700
Version 1.7.1~dfsg1-1 was uploaded on 2015-08-26 10:13:48 -0700 -0700
Version 1.6.2~dfsg1-2 was uploaded on 2015-07-01 07:45:19 -0600 -0600
Version 1.6.2~dfsg1-1 was uploaded on 2015-05-21 00:47:43 -0600 -0600
Version 1.6.1+dfsg1-2 was uploaded on 2015-05-10 13:02:54 -0400 EDT
Version 1.6.1+dfsg1-1 was uploaded on 2015-05-08 17:57:10 -0600 -0600
Version 1.6.0+dfsg1-1 was uploaded on 2015-05-05 15:10:49 -0600 -0600
Version 1.6.0+dfsg1-1~exp1 was uploaded on 2015-04-16 18:00:21 -0600 -0600
Version 1.6.0~rc7~dfsg1-1~exp1 was uploaded on 2015-04-15 19:35:46 -0600 -0600
Version 1.6.0~rc4~dfsg1-1 was uploaded on 2015-04-06 17:11:33 -0600 -0600
Version 1.5.0~dfsg1-1 was uploaded on 2015-03-10 22:58:49 -0600 -0600
Version 1.3.3~dfsg1-2 was uploaded on 2015-01-03 00:11:47 -0700 -0700
Version 1.3.3~dfsg1-1 was uploaded on 2014-12-18 21:54:12 -0700 -0700
Version 1.3.2~dfsg1-1 was uploaded on 2014-11-24 19:14:28 -0500 EST
Version 1.3.1~dfsg1-2 was uploaded on 2014-11-07 13:11:34 -0700 -0700
Version 1.3.1~dfsg1-1 was uploaded on 2014-11-03 08:26:29 -0700 -0700
Version 1.3.0~dfsg1-1 was uploaded on 2014-10-17 00:56:07 -0600 -0600
Version 1.2.0~dfsg1-2 was uploaded on 2014-10-09 00:08:11 +0000 +0000
Version 1.2.0~dfsg1-1 was uploaded on 2014-09-13 11:43:17 -0600 -0600
Version 1.0.0~dfsg1-1 was uploaded on 2014-06-13 21:04:53 -0400 EDT
Version 0.11.1~dfsg1-1 was uploaded on 2014-05-09 17:30:45 -0400 EDT
Version 0.9.1~dfsg1-2 was uploaded on 2014-04-08 23:19:08 -0400 EDT
Version 0.9.1~dfsg1-1 was uploaded on 2014-04-03 21:38:30 -0400 EDT
Version 0.9.0+dfsg1-1 was uploaded on 2014-03-11 22:24:31 -0400 EDT
Version 0.8.1+dfsg1-1 was uploaded on 2014-02-25 20:56:31 -0500 EST
Version 0.8.0+dfsg1-2 was uploaded on 2014-02-15 17:51:58 -0500 EST
Version 0.8.0+dfsg1-1 was uploaded on 2014-02-10 20:41:10 -0500 EST
Version 0.7.6+dfsg1-1 was uploaded on 2014-01-22 22:50:47 -0500 EST
Version 0.7.1+dfsg1-1 was uploaded on 2014-01-15 20:22:34 -0500 EST
Version 0.6.7+dfsg1-3 was uploaded on 2014-01-09 20:10:20 -0500 EST
Version 0.6.7+dfsg1-2 was uploaded on 2014-01-08 19:14:02 -0500 EST
Version 0.6.7+dfsg1-1 was uploaded on 2014-01-07 21:06:10 -0500 EST
control
Next is one of the most complex, and one of the oldest parts of go-debian,
which is the control file parser
(otherwise sometimes known as deb822). This module was inspired by the way
that the json module works in Go, allowing for files to be defined in code
with a struct. This tends to be a bit more declarative, but also winds up
putting logic into struct tags, which can be a nasty anti-pattern if used too
much.
The first primitive in this module is the concept of a Paragraph, a struct
containing two values, the order of keys seen, and a map of string to string.
All higher order functions dealing with control files will go through this
type, which is a helpful interchange format to be aware of. All parsing of
meaning from the Control file happens when the Paragraph is unpacked into
a struct using reflection.
The idea behind this strategy that you define your struct, and let the Control
parser handle unpacking the data from the IO into your container, letting you
maintain type safety, since you never have to read and cast, the conversion
will handle this, and return an Unmarshaling error in the event of failure.
Additionally, Structs that define an anonymous member of control.Paragraph
will have the raw Paragraph struct of the underlying file, allowing the
programmer to handle dynamic tags (such as X-Foo), or at least, letting
them survive the round-trip through go.
The default decoder
contains an argument, the ability to verify the input control file using an
OpenPGP keyring, which is exposed to the programmer through the
(*Decoder).Signer() function. If the passed argument is nil, it will not
check the input file signature (at all!), and if it has been passed, any
signed data must be found or an error will fall out of the NewDecoder call.
On the way out, the opposite happens, where the struct is introspected,
turned into a control.Paragraph, and then written out to the io.Writer.
Here's a quick (and VERY dirty) example showing the basics of reading and
writing Debian Control files with go-debian.
packagemainimport("fmt""io""net/http""strings""pault.ag/go/debian/control")typeAllowedPackagestructPackagestringFingerprintstringfunc(a*AllowedPackage)UnmarshalControl(instring)errorin=strings.TrimSpace(in)chunks:=strings.SplitN(in," ",2)iflen(chunks)!=2returnfmt.Errorf("Syntax sucks: '%s'",in)a.Package=chunks[0]a.Fingerprint=chunks[1][1:len(chunks[1])-1]returnniltypeDMUAstructFingerprintstringUidstringAllowedPackages[]AllowedPackage control:"Allow" delim:"," funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/dm.txt")iferr!=nilpanic(err)decoder,err:=control.NewDecoder(resp.Body,nil)iferr!=nilpanic(err)fordmua:=DMUAiferr:=decoder.Decode(&dmua);err!=niliferr==io.EOFbreakpanic(err)fmt.Printf("The DM %s is allowed to upload:\n",dmua.Uid)for_,allowedPackage:=rangedmua.AllowedPackagesfmt.Printf(" %s [granted by %s]\n",allowedPackage.Package,allowedPackage.Fingerprint)
Output (truncated!) looks a bit like:
...
The DM Allison Randal <allison@lohutok.net> is allowed to upload:
parrot [granted by A4F455C3414B10563FCC9244AFA51BD6CDE573CB]
...
The DM Benjamin Barenblat <bbaren@mit.edu> is allowed to upload:
boogie [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
dafny [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
transmission-remote-gtk [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
urweb [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
...
The DM <aelmahmoudy@sabily.org> is allowed to upload:
covered [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
dico [granted by 6ADD5093AC6D1072C9129000B1CCD97290267086]
drawtiming [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
fonts-hosny-amiri [granted by BD838A2BAAF9E3408BD9646833BE1A0A8C2ED8FF]
...
...
deb
Next up, we've got the deb module. This contains code to handle reading
Debian 2.0 .deb files. It contains a wrapper that will parse the control
member, and provide the data member through the
archive/tar interface.
Here's an example of how to read a .deb file, access some metadata, and
iterate over the tar archive, and print the filenames of each of the
entries.
dependency
The dependency package provides an interface to parse and compute
dependencies. This package is a bit odd in that, well, there's no other
library that does this. The issue is that there are actually two different
parsers that compute our Dependency lines, one in Perl (as part of dpkg-dev)
and another in C (in dpkg).
To date, this has resulted in me filing
threedifferentbugs.
I also found a broken package in the
archive,
which actually resulted in another bug being (totally accidentally)
already fixed.
I hope to continue to run the archive through my parser in hopes of finding
more bugs! This package is a bit complex, but it basically just returns what
amounts to be an AST
for our Dependency lines. I'm positive there are bugs, so file them!
funcmain()dep,err:=dependency.Parse("foo bar, baz, foobar [amd64] bazfoo [!sparc], fnord:armhf [gnu-linux-sparc]")iferr!=nilpanic(err)anySparc,err:=dependency.ParseArch("sparc")iferr!=nilpanic(err)for_,possi:=rangedep.GetPossibilities(*anySparc)fmt.Printf("%s (%s)\n",possi.Name,possi.Arch)
Gives the output:
foo (<nil>)
baz (<nil>)
fnord (armhf)
version
Right off the bat, I'd like to thank
Michael Stapelberg for letting me graft this
out of dcs and into the go-debian package.
This was nearly entirely his work (with a one or two line function I added
later), and was amazingly helpful to have. Thank you!
This module implements Debian version comparisons and parsing, allowing for
sorting in lists, checking to see if it's native or not, and letting the
programmer to implement smart(er!) logic based on upstream (or Debian)
version numbers.
This module is extremely easy to use and very straightforward, and not worth
writing an example for.
Final thoughts
This is more of a "Yeah, OK, this has been useful enough to me at this point
that I'm going to support this" rather than a "It's stable!" or even
"It's alive!" post. Hopefully folks can report bugs and help iterate on
this module until we have some really clean building blocks to build
solid higher level systems on top of. Being able to have multiple libraries
interoperate by relying on go-debian will be a massive ease.
I'm in need of more documentation, and to finalize some parts of the older
sub package APIs, but I'm hoping to be at a "1.0" real soon now.
* find somebody in Europe who prints good stickers and doesn't require illustrator (or other proprietary software) to submit files for non-rectangular shapes * find out which Debian team I should contact to submit the files so that they can be used by everybody interested.
But neither will happen today, nor probably tomorrow, because lazy O:-)
Edit: now that I'm awake I realized I forgot to thank @Enrico Zini Zini and MadameZou for their help in combining my two proposals in a better design.
Wheezy's LTS period started a few weeks ago and the LTS team had to make an
early support decision concerning the Java eco-system since Wheezy ships two Java
runtime environments OpenJDK 6 and
OpenJDK 7. (To be fair, there are
actually three but gcj has
been superseded by OpenJDK a long time ago and the latter should be preferred
whenever possible.)
OpenJDK 6 is currently maintained by Red Hat and we mostly rely on their
upstream work as well as on package updates from Debian's maintainer Matthias Klose and
Tiago St rmer Daitx from Ubuntu. We already knew that both intend
to support OpenJDK 6 until April 2017 when Ubuntu 12.04 will reach its
end-of-life. Thus we had basically two options, supporting OpenJDK 6 for another
twelve months or dropping support right from the start. One of my first steps
was to ask for feedback and advice on
debian-java since
supporting only one JDK seemed to be the more reasonable solution. We agreed on warning users
via various channels about the intended change, especially about possible
incompatibilities
with OpenJDK 7. Even Andrew Haley, OpenJDK 6 project lead, participated in the discussion and confirmed that, while still supported, OpenJDK 6 security
releases are "always the last in the queue when there is urgent work to be
done".
I informed debian-lts about
my findings and issued a call for tests later.
Eventually we decided to concentrate our efforts on OpenJDK
7 because we are confident that for the
majority of our users one Java implementation is sufficient during a stable
release cycle. An immediate positive effect in making OpenJDK 7 the default is that
resources can be relocated to more pressing issues. On the other hand we were
also forced to make compromises. The switch to a newer default implementation
usually triggers a major transition with dozens of FTBFS bugs and the OpenJDK 7
transition
was no exception. I pondered about the usefulness of fixing all these bugs for
Wheezy LTS again and focussing on runtime issues instead and finally decided
that the latter was both more reasonable and more economic.
Different from regular default Java changes, users will still be able to use OpenJDK 6 to
compile their packages and the security impact for development systems is in general neglectable.
More important was to avoid runtime installations of OpenJDK 6. I identified eighteen
packages that strictly depended on the now obsolete JRE and fixed those
issues
on 4 May 2016 together with an update of
java-common and announced the switch to
OpenJDK 7 with a Debian NEWS file.
If you are not a regular reader of Debian
news and also not subscribed to
debian-lts,
debian-lts-announce or
debian-java, remember 26 June
2016 is the day when OpenJDK 7 will be made the default Java implementation in
Wheezy LTS. Of course there is no need to wait. You can switch right now:
The last few months have been a bit of a crazy period of ups and downs, with a
tempest of events beneath the apparent and deceivingly calm surface waters of
being unemployed (still at it).
The daily grind
Chief activities are, of course, those related to the daily grind of
job-hunting, sending applications, and preparing and attending interviews.
It is demoralising when one searches for many days or weeks without seeing
anything suitable for one's skills or interests, or other more general life
expectations. And it takes a lot of time and effort to put one's best in the
applications for positions that one is really, really, interested in. And
even for the ones which are meh, for a variety of reasons (e.g. one is not
very suitable for what the offer demands).
After that, not being invited to interviews (or doing very badly at them) is
bad, of course, but quick and not very painful. A swift, merciful end to the
process.
But it's all the more draining when waiting for many weeks when not a few
months with the uncertainty of not knowing if one is going to be lucky enough
to be summoned for an interview; harbouring some hope one has to appear
enthusiastic in the interviews, after all , while trying to keep it contained
lest it grows too much ; then in the interview hearing good words and some
praises, and feeling the impression that one will fit in, that one did nicely
and that chances are good letting the hope grow again ; start to think about
life changes that the job will require to make a quick decision should the
offer finally arrives ; perhaps make some choices and compromises based on the
uncertain result; then wait for a week or two after the interview to know the
result...
... only to end up being unsuccessful.
All the effort and hopes finally get squashed with a cold, short email or
automatic response, or more often than not, complete radio silence from
prospective employers, as an end to a multi-month-long process. An emotional
roller coaster [1], which
happened to me several times in the last few months.
All in a day's work
The months of preparing and waiting for a new job often imply an impasse that
puts many other things that one cares about on hold, and one makes plans that
will never come to pass.
All in a day's (half-year's?) work of an unemployed poor soul.
But not all is bad.
This period was also a busy time doing some plans about life, mid- and
long-term; the usual and some really unusual! family events; visits to and
from friends, old and new; attending nice little local Debian gatherings or the
bigger gathering of Debian
SunCamp2016, and other
work for side projects or for other events that will happen soon...
And amidst all that, I managed to get some work done on aptitude.
Two pictures worth (less than) a thousand bugs
To be precise, worth 709 bugs 488 bugs in the first graph, plus 221 in the
second.
In 2015-11-15 (link to the post Work on aptitude):
In 2016-06-18:
Numbers
The BTS numbers for aptitude right now are:
221 (259 if counting all merged bugs independently)
1 Release Critical (but it is an artificial bug to keep it from migrating to
testing)
43 (55 unmerged) with severity Important or Normal
160 (182 unmerged) with severity Minor or Wishlist
17 (21 unmerged) marked as Forwarded or Pending
Highlights
Beyond graphs and stats, I am specially happy about two achievements in the last
year:
To have aptitude working today, first and foremost
Apart from the abandon that suffered in previous years, I mean
specifically the critical step of getting it through the troubles of the
last summer, with the GCC-5/C++11 transition in parallel with a
transition of the Boost library (explained in more detail in Work on
aptitude).
Without that, possibly aptitude would not have survived until today.
Improvements to the suggestions of the resolver
In the version 0.8, there were a lot of changes related with improving
the order of the suggestions from the resolver, when it finds conflicts
or other problems with the planned actions.
Historically, but specially in the last few years, there have been many
complaints about the nonsensical or dangerous suggestions from the
resolver. The first solution offered by the resolver was very often
regarded as highly undesirable (for example, removal of many packages),
and preferable solutions like upgrades of one or only a handful of
packages being offered only after many removals; and keeps only
offered as last resort.
Perhaps these changes don't get a lot of attention, given that in the first case
it's just to keep working (with few people realising that it could have
collapsed on the spot, if left unattended), and the second can probably go
unnoticed because it just works or it started to work more smoothly
doesn't get as much immediate attention as it suddenly broke! .
Still, I wanted to mention them, because I am quite proud of those.
Thanks
Even if I put a lot of work on aptitude in the last year, the results of the
graph and numbers have not been solely achieved by me.
Special thanks go to Axel Beckert (abe /
XTaran) and the apt team, David
Kalnischkies and Julian Andres Klode who, despite the claim in that page, does
not mostly work python-aptanymore... but also in the main tools.
They help with fixing some of the issues directly, or changing things in apt
that benefit aptitude, testing changes, triaging bugs or commenting on them,
patiently explaining to me why something in libapt doesn't do what I think it
does, and good company in general.
Not the least, for holding impromptu BTS group therapy / support meetings, for
those cases when prolonged exposure to BTS activity starts to induce very bad
feelings.
Thanks also to people who sent their translation updates, notified about
corrections, sent or tested patches, submitted bugs, or tried to help in other
ways. Change logs for details.
Notes
[1] ^ It's even an example in
the Cambridge Dictionaries Online website, for the entry of roller
coaster:
He was on an emotional roller coaster for a while when he lost his job.
I wanted to write this blogpost since April, and even announced it in two previous posts, but never got to actually write it until now. And with the recent events in Snappy and Flatpak land, I can not defer this post any longer (unless I want to answer the same questions over and over on IRC ^^).
As you know, I develop the Limba 3rd-party software installer since 2014 (see this LWN article explaining the project better then I could do ) which is a spiritual successor to the Listaller project which was in development since roughly 2008. Limba got some competition by Flatpak and Snappy, so it s natural to ask what the projects next steps will be.
Meeting with the competition
At last FOSDEM and at the GNOME Software sprint this year in April, I met with Alexander Larsson and we discussed the rather unfortunate situation we got into, with Flatpak and Limba being in competition.
Both Alex and I have been experimenting with 3rd-party app distribution for quite some time, with me working on Listaller and him working on Glick and Glick2. All these projects never went anywhere. Around the time when I started Limba, fixing design mistakes done with Listaller, Alex started a new attempt at software distribution, this time with sandboxing added to the mix and a new OSTree-based design of the software-distribution mechanism. It wasn t at all clear that XdgApp, later to be renamed to Flatpak, would get huge backing by GNOME and later Red Hat, becoming a very promising candidate for a truly cross-distro software distribution system.
The main difference between Limba and Flatpak is that Limba allows modular runtimes, with things like the toolkit, helper libraries and programs being separate modules, which can be updated independently. Flatpak on the other hand, allows just one static runtime and enforces everything that is not in the runtime already to be bundled with the actual application. So, while a Limba bundle might depend on multiple individual other bundles, Flatpak bundles only have one fixed dependency on a runtime. Getting a compromise between those two concepts is not possible, and since the modular vs. static approach in Limba and Flatpak where fundamental, conscious design decisions, merging the projects was also not possible.
Alex and I had very productive discussions, and except for the modularity issue, we were pretty much on the same page in every other aspect regarding the sandboxing and app-distribution matters.
Sometimes stepping out of the way is the best way to achieve progress
So, what to do now? Obviously, I can continue to push Limba forward, but given all the other projects I maintain, this seems to be a waste of resources (Limba eats a lot of my spare time). Now with Flatpak and Snappy being available, I am basically competing with Canonical and Red Hat, who can make much more progress faster then I can do as a single developer. Also, Flatpaks bigger base of contributors compared to Limba is a clear sign which project the community favors more.
Furthermore, I started the Listaller and Limba projects to scratch an itch. When being new to Linux, it was very annoying to me to see some applications only being made available in compiled form for one distribution, and sometimes one that I didn t use. Getting software was incredibly hard for me as a newbie, and using the package-manager was also unusual back then (no software center apps existed, only package lists). If you wanted to update one app, you usually needed to update your whole distribution, sometimes even to a development version or rolling-release channel, sacrificing stability.
So, if now this issue gets solved by someone else in a good way, there is no point in pushing my solution hard. I developed a tool to solve a problem, and it looks like another tool will fix that issue now before mine does, which is fine, because this longstanding problem will finally be solved. And that s the thing I actually care most about.
I still think Limba is the superior solution for app distribution, but it is also the one that is most complex and requires additional work by upstream projects to use it properly. Which is something most projects don t want, and that s completely fine.
And that being said: I think Flatpak is a great project. Alex has much experience in this matter, and the design of Flatpak is sound. It solves many issues 3rd-party app development faces in a pretty elegant way, and I like it very much for that. Also the focus on sandboxing is great, although that part will need more time to become really useful. (Aside from that, working with Alexander is a pleasure, and he really cares about making Flatpak a truly cross-distributional, vendor independent project.)
Moving forward
So, what I will do now is not to stop Limba development completely, but keep it going as a research project. Maybe one can use Limba bundles to create Flatpak packages more easily. We also discussed having Flatpak launch applications installed by Limba, which would allow both systems to coexist and benefit from each other. Since Limba (unlike Listaller) was also explicitly designed for web-applications, and therefore has a slightly wider scope than Flatpak, this could make sense.
In any case though, I will invest much less time in the Limba project. This is good news for all the people out there using the Tanglu Linux distribution, AppStream-metadata-consuming services, PackageKit on Debian, etc. those will receive more attention
An integral part of Limba is a web service called LimbaHub to accept new bundles, do QA on them and publish them in a public repository. I will likely rewrite it to be a service using Flatpak bundles, maybe even supporting Flatpak bundles and Limba bundles side-by-side (and if useful, maybe also support AppImageKit and Snappy). But this project is still on the drawing board.
Let s see
P.S: If you come to Debconf in Cape Town, make sure to not miss my talks about AppStream and bundling
Hi to all the souls on planet.debian.org . I hope to meet many of you in Debconf16 which is being held at UCT (University of Cape Town), Rhondenbosch, South Africa and am excited to be part of it. I pushed a blog post about my journey to debconf till date.
I express my sympathy and condolences to all the people who died in the cowardly shooting spree done by a madman in Orlando . I have been upset about this development but as what s done is done, it s best to just keep moving.
Closer to my own reality, I was shocked to discover during my whole visa experience that nowhere there is any knowledge about vaccinations that people should have when they are traveling internationally. I was kinda rudely awakened by this mail which prompted me to take Hep A shot and also start a thread on the mailing list. The more comprehensive info. I got was at the CDC site . It seems to be a go-to site to find about what is recommended. While I will not be taking any shots now as it s nearing to the date of travel, if I had known before, it would have been a valuable resource in itself. Definitely something to be bookmarked if you are going overseas and are worried or have health concerns.
On another note, About few days back, there was a discussion as recently, travel advisories have been issued about possible terror action in South Africa by various embassies due to the holy month of Ramadan.
And the recent attack just proves that it takes just one twisted personality with a perverse sense of justice or whatever s/he/ thinks as just to do what s/he did and guns and more security and not the answer.
My take on it some would describe as simplistic, there are things which you can control, there are things beyond your control. No security agency, no country can guarantee it. By being either home or away, you can t wish you will not get bitten. Got images of Final Destination invoked when I was sharing that, for as an Indian, do believe a bit on fate and karma. Also, me being single also plays a part, perhaps I would have been more cautious or have different motivations if I had a family so I do understand some of the concerns which have been raised by people in that thread. At the end of it, it really is a non-choice in my book.
If you don t take part due to fear,uncertainty of a possible attack, you have already given in to fear and uncertainty and I believe this goes against the very philosophy of what Debian stands for, being bold, taking chances and having trust in your fellow men. If we haven t allowed proprietary, commercial software to win over us, how can we allow less than 0.1% radically motivated people to scare us ? And the recent attack just proves that it takes just one twisted personality with a perverse sense of justice or whatever s/he/ thinks as just to do what s/he did. More guns and more security are certainly not the answer here.
A more troubling part is not terrorism but caste-ism and racism which have been also making news (not in a good way) in India. Now while I cannot claim to have any knowledge about Africans apart from 2-3 conversations which didn t go anywhere, two-three preconceived notions about them can easily be countered. As far as drugs are concerned, IF some Africans are doing drugs smuggling, it would be wrong to pin all of it onto them. India has been fighting drugs smuggling from the 70 s itself. From what we know and have learnt over the years from consuming media, India shares porous borders with almost all our neighbors. Of those, Pakistan is supposed to be the largest grower and supplier and then Nepal where young boys are used as Traffickers. The recent film Rocky Handsome and the recent upcoming controversial movie Udta Punjab are trying to explore these issues. As far as drinking in open is concerned, I have seen and been part of Punjabi parties where both men and women drink without abandon in farm parties. Russians, Germans and Israelis can also out-drink a person on their day/mood. As far as sexuality is concerned, we are the second most populous country in the world, so the less said the better .
I believe though that underneath this racism is money, greed, phobia, language barriers and just maybe some lifestyle choices as well. I have seen North-Eastern, Chinese, Buddhist people being colored with the same brush. The more interesting case is with the Buddhist as can be seen in Himachal Pradesh, where locals feel they are deprived as Europeans come and indulge Buddhists for their monasteries and their way of life while locals don t get much money from them. This is partly true, but also due to our own short-comings in dealing with westerners. I have never seen my own countrymen going out of their way to make westerners or any tourists for that matter feel welcome in a genuine way. More often, the behavior is between hostility, jealousy and a perverted sense of hero-worship due to the color of skin. As far as being racist and bigoted are concerned, it seems we are not alone, just day before came across this news from Malaysia which is interesting in the sense that how people from different communities frame their own history and people around them, forget that is false and misleading to what we in India know to be the truth, it probably is/would be interesting for somebody who does comparative history, myth and folklore analysis. I was actually planning to talk about some of the talks I am looking forward to hearing and seeing on Debconf but guess that will have to wait for few more days. Hope to publish one more before actually flying to Debconf. Till later. Filed under: Miscellenous Tagged: #Debconf16, #racism, #terrorism, #vaccinations
What happened in the Reproducible
Builds effort between June 5th and June 11th 2016:
Media coverage
Ed Maste gave a talk at BSDCan 2016 on
reproducible builds
(slides,
video).
GSoC and Outreachy updates
Weekly reports by our participants:
Scarlett Clark
worked on making some packages reproducible, focusing on KDE backend and
utility programs.
Ceridwen
published an initial design for the interface for reprotest, including a
discussion on different types of build variations and the difficulties of
specifying certain types of variations.
Valerie Young improved documentation for
building our tests website, began migrating Debian-specific pages into a new
namespace, and planned future work around its navigation.
Documentation update
- Ximin Luo proposed a modification
to our SOURCE_DATE_EPOCH spec explaining FORCE_SOURCE_DATE.
Some upstream build tools (e.g. TeX, see below) have expressed a desire to
control which cases of embedded timestamps should obey SOURCE_DATE_EPOCH.
They were not convinced by our arguments on why this is a bad idea, so we
agreed on an environment variable FORCE_SOURCE_DATE for them to implement
their desired behaviour - named generically, so that at least we can set it
centrally. For more details, see the text just linked. However, we strongly
urge most build tools not to use this, and instead obey SOURCE_DATE_EPOCHunconditionally in all cases.
Toolchain fixes
TeX Live 2016 released
with SOURCE_DATE_EPOCH support for all engines except LuaTeX and original TeX.
Continued discussion
(alternative archive)
with TeX upstream, about SOURCE_DATE_EPOCH corner cases, eventually
resulting in the FORCE_SOURCE_DATE proposal from above.
gcc-5/5.4.0-4 by Matthias Klose now avoids storing
-fdebug-prefix-map in DW_AT_producer, thanks to original patch by
Daniel Kahn Gillmor.
sphinx/1.4.3-1 by Dmitry Shachnev now drops Debian-specific patches
relating to SOURCE_DATE_EPOCH applied upstream, original patch by Alexis
Bienven e.
asciidoctor/1.5.4-2 by C dric Boutillier now supports
SOURCE_DATE_EPOCH, thanks to original patch by Alexis Bienven e.
dh-python/1.5.4-2 by Piotr O arowski now behaves better in some
cases, thanks to original patch by Chris Lamb.
Andrew Ayer uploaded strip-nondeterminism/0.018-2 fixing #826700, a packaging improvement for Multi-Arch to ease cross-build
situations.
2 days later Andrew released strip-nondeterminism/0.019; now
strip-nondeterminism is able to:
recursively normalize JAR files embedded within JAR files (#823917)
clamp the timestamp, the same way tar >=1.28-2.2 can (for now available only for gzip archives)
disorderfs development
Andrew Ayer released disorderfs/0.4.3, fixing a issue with umask handling (#826891)
tests.reproducible-builds.org
Valerie Young namespaced the Debian-specific pages to /debian/
namespace, with redirects to
for the previous URLs.
Holger Levsen improved the reliability of build jobs: the availability of
both build nodes (for a given build) is now being tested when a build job is
started, to better cope when one of the 25 build nodes go down for some reason.
Ximin Luo improved the index of identified
issues to
include the total popcon scores of each issue, which is now also used for
sorting that page.
Misc.
Steven Chamberlain submitted a patch to FreeBSD's
makefs
to allow reproducible builds of the kfreebsd installer.
Ed Maste committed a patch to FreeBSD's
binutils to
enable determinstic archives by default in GNU ar.
Helmut Grohne
experimentedwith
cross+native reproductions of dash with some success, using
rebootstrap.
This week's edition was written by Ximin Luo, Chris Lamb, Holger Levsen, Mattia
Rizzolo and reviewed by a bunch of Reproducible builds folks on IRC.
Suppose you have a gpg keyid like 9F6C6333 that corresponds to both key
1AE0322EB8F74717BDEABF1D44BB1BA79F6C6333 and
88BB08F633073D7129383EE71EA37A0C9F6C6333, and you don't know which of the two
to use.
You go to http://pgp.cs.uu.nl/ and find out that the site uses short key IDs,
so the two keys are indistinguishable.
Building on Clint's hopenpgp-tools,
I made a script that
screenscrapes http://pgp.cs.uu.nl/ for trust paths, downloads all the
potentially connecting keys in a temporary keyring, and runs hkt findpaths on
it:
$ ./verify-trust-paths 1793D6AB75663E6BF104953A634F4BD1E7AD5568 1AE0322EB8F74717BDEABF1D44BB1BA79F6C6333
hkt (hopenpgp-tools) 0.18
Copyright (C) 2012-2016 Clint Adams
hkt comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions.
(4,[1,4,3,6])(1,1793D6AB75663E6BF104953A634F4BD1E7AD5568)(3,F8921D3A7404C86E11352215C7197699B29B232A)(4,C331BA3F75FB723B5873785B06EAA066E397832F)(6,1AE0322EB8F74717BDEABF1D44BB1BA79F6C6333)
$ ./verify-trust-paths 1793D6AB75663E6BF104953A634F4BD1E7AD5568 88BB08F633073D7129383EE71EA37A0C9F6C6333
hkt (hopenpgp-tools) 0.18
Copyright (C) 2012-2016 Clint Adams
hkt comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions.
(0,[])
This is a start: it could look in the local keyring for all ultimately trusted
key finegrprints and use those as starting points. It could just take as an
argument a short keyid and automatically check all matching fingerprints.
I'm currently quite busy with https://nm.debian.org and at the moment
verify-trust-paths scratches
enough of my itch that I can move on with my other things.
Please send patches, or take it over: I'd like to see this grow.
Yesterday I was fixing an issue with one of the servers behind kallithea-scm.org: the hook intended to propagage pushes from Our Own Kallithea to Bitbucket stopped working. Until yesterday, that server was using Debian s flavour of System V init and djb s d montools to keep things running. To make the hook asynchronous, I wrote a service to be managed to d montools, so that concurrency issued would be solved by it. However, I didn t implement any timeouts, so when last week wget froze while pulling Weblate s hook, there was nothing to interrupt it, so the hook stopped working since d montools thought it s already running and wouldn t re-trigger it. Killing wget helped, but I decided I need to do something with it to prevent the situation from happening in the future.
I ve been using systemd at work for the last year, so I am now confident I m happier with systemd than with d montools, so I decided to switch the server to systemd. Not surprisingly, I prepared unit files in about 5 minutes without having to look into the manuals again, while with d montools I had to check things every time I needed to change something. The tricky thing was the switch itself. It is a virtual server, presumably running in Xen, and I don t have access to the console, so if I bork break something, I need to summon Bradley Kuhn or someone from Conservancy, who s kindly donated the server to the project. In any case, I decided to attempt to upgrade without a reboot, so that I have more options to roll back my changes in the case things go wrong.
After studying the manpages of both systemd s init and sysvinit s init, I realised I can install systemd as /sbin/init and ask already running System V init to re-exec. However, systemd s init can t talk to System V init, so before installing systemd I made a backup on it. It s also important to stop all running services (except probably ssh) to make sure systemd doesn t start second instances of each. And then: /tmp/init u and we re running systemd! A couple of additional checks, and it s safe to reboot.
Only when I did all that I realised that in the case of systemd not working I d probably not be able to undo my changes if my connection interrupted. So, even though at the end it worked, probably it s not a good idea to perform such manipulations when you don t have an alternative way to connect to the server :)
When you get involved in a new community, such as Debian, find out early
where, if that happens, you can find support, understanding, and help to make
it stop.
Last night I asked a group of friends what do they do if they start feeling
uncomfortable when they are alone in a group of people they don't know. Here
are the replies:
Wait outside the group until I figure the group out.
Find someone to talk for a while until you get comfortable.
If a person is making things uncomfortable for you, let them know, and leave
if nobody cares.
Sit there in silence.
Work around unwelcome people by bearing them for a bit while trying to
integrate with others.
Some people are easy to bribe into friendship, just bring cake.
While you don't know what is going on, you try to replicate what others are
doing.
Spend time trying to get a feeling of what are the consequences of taking
actions.
Purposefully disagree with people in a new environment to figure out if
having a different opinion is accepted.
Once I was new and I was asked to be the person that invites everyone for
lunch, that forced me to talk to everyone, and integrate.
When you are the first one to point something out, you'll probably soon find
out you're not alone.
The reaction on the first time something is exposed, influences how often
similar cases will be reported.
I think a lot of these point are good food for thought about barriers of entry,
and about safety nets that a group has or might want to have.