SOURCE_DATE_EPOCH that contains a known timestamp in Unix epoch format. With this variable exported, tools that would embed the localtime in their generated or processed files, can use this externally supplied date. This would happen only if SOURCE_DATE_EPOCH is exported, so the behaviour of the tool wouldn t change if the variable is not set.
The first package I patched to implement this behaviour was gcc. The reason behind this is that there are about 420 unreproducible packages due to using the __DATE__, __TIME__ and __TIMESTAMP__ C macros. My patch changes the behavior of the macros __DATE__ and __TIME__ if SOURCE_DATE_EPOCH is exported. I submitted this patch to the gcc-patches list. Even though there was some interesting discussions in the list, the patch has not been accepted yet. Seeing how the reproducible builds idea is gaining momentum and becoming widespread, I m positive that at some point the gcc people will be more receptive for such patch.
The second work with SOURCE_DATE_EPOCH was in debhelper; I patched this building tool to export the variable with the latest debian/changelog entry timestamp. With this patch, all the tools that run under dh will be able to use it to embed deterministic timestamps. Unfortunately some parts of the build process of some packages don t happen under debhelper, so the variable needs to be exported in a different way.
Having submitted the debhelper patch allowed many packages to become reproducible after the tools that embedded timestamps were patched to honour SOURCE_DATE_EPOCH. As of today, the toolchain packages I have patched to do that are: gcc, libxslt, gettext, ghostscript and qt4-x11 (qhelpgenerator).
I have also continued working on fixing individual packages affected by timestamps, random orderings (such as the ones from listing hash keys) and locale depending orderings; I have tagged packages in our infrastructure to note what kind of issue makes them unreproducible; I have updated some parts of the Reproducible Builds Wiki.
I've been using qsub for a while now on the cluster here at the
IGB at UofI. qsub is a command line
program which is used to submit jobs to a scheduler to eventually be
run on one (or more) nodes of a cluster.
Unfortunately, qsub's interface is horrible. It requires that you
write a shell script for every single little thing you run, and
doesn't do simple things like providing defaults or running multiple
jobs at once with slightly different arguments. I've dealt with this
for a while using some rudimentary shell scripting, but I finally had
enough.
So instead, I wrote a wrapper around qsub called dqsub.
What used to require a complicated invocation like:
echo -e '#!/bin/bash\nmake foo' \
qsub -q default -S /bin/bash -d $(pwd) \
-l mem=8G,nodes=1:ppn=4 -;
can now be run with
dqsub --mem 8G --ppn 4 make foo;
Want to run some command in every single directory which starts with
SRX? That's easy:
ls -1 SRX* dqsub --mem 8G --ppn 4 --array chdir make bar;
Want instead to behave like xargs but do the same thing?
ls -1 SRX* dqsub --mem 8G --ppn 4 --array xargs make bar -C;
Now, this wrapper isn't complete yet, but it's already more than
enough to do what I require, and has saved me quite a bit of time
already.
You can steal dqsub for yourself
Feel free to request specific features, too.
The TODO group is an industry body that appears to be trying to define community best practices or something. I don't really know what their backstory is and whether they're trying to do meaningful work or just provide a fig leaf of respectability to organisations that dislike being criticised for doing nothing to improve the state of online communities but don't want to have to actually do anything, and their initial work on codes of conduct was, perhaps, suboptimal. But they do appear to be trying to improve things - this commit added a set of inappropriate behaviours, and also clarified that reverseisms were not actionable behaviour.
It s Seafair weekend in Seattle. As always, the centerpiece is the H1 Unlimited hydroplane races on Lake Washington.
In my social circle, I m nearly the only person I know who grew up in area. None of the newcomers I know had heard of hydroplane racing before moving to Seattle. Even after I explain it to them i.e., boats with 3,000+ horse power airplane engines that fly just above the water at more than 320kph (200mph) leaving 10m+ (30ft) wakes behind them! most people seem more puzzled than interested.
I grew up near the shore of Lake Washington and could see (and hear!) the races from my house. I don t follow hydroplane racing throughout the year but I do enjoy watching the races at Seafair. Here s my attempt to explain and make the case for the races to new Seattleites.
Before Microsoft, Amazon, Starbucks, etc., there were basically three major Seattle industries: (1) logging and lumber based industries like paper manufacturing; (2) maritime industries like fishing, shipbuilding, shipping, and the navy; (3) aerospace (i.e., Boeing). Vintage hydroplane racing represented the Seattle trifecta: Wooden boats with airplane engines!
The wooden U-60 Miss Thriftway circa 1955 (Thriftway is a Washinton-based supermarket that nobody outside has heard of) below is a picture of old-Seattle awesomeness. Modern hydroplanes are now made of fiberglass but two out of three isn t bad.
Although the boats are racing this year in events in Indiana, San Diego, and Detroit in addition to the two races in Washington, hydroplane racing retains deep ties to the region. Most of the drivers are from the Seattle area. Many or most of the teams and boats are based in Washington throughout the year. Many of the sponsors are unknown outside of the state. This parochialness itself cultivates a certain kind of appeal among locals.
In addition to old-Seattle/new-Seattle cultural divide, there s a class divide that I think is also worth challenging. Although the demographics of hydro-racing fans is surprisingly broad, it can seem like Formula One or NASCAR on the water. It seems safe to suggest that many of the demographic groups moving to Seattle for jobs in the tech industry are not big into motorsports. Although I m no follower of motorsports in general, I ve written before cultivated disinterest in professional sports, and it remains something that I believe is worth taking on.
It s not all great. In particular, the close relationship between Seafair and the military makes me very uneasy. That said, even with the military-heavy airshow, I enjoy the way that Seafair weekend provides a little pocket of old-Seattle that remains effectively unchanged from when I was a kid. I d encourage others to enjoy it as well!
It s Seafair weekend in Seattle. As always, the centerpiece is the H1 Unlimited hydroplane races on Lake Washington.
In my social circle, I m nearly the only person I know who grew up in area. None of the newcomers I know had heard of hydroplane racing before moving to Seattle. Even after I explain it to them i.e., boats with 3,000+ horse power airplane engines that fly just above the water at more than 320kph (200mph) leaving 10m+ (30ft) wakes behind them! most people seem more puzzled than interested.
I grew up near the shore of Lake Washington and could see (and hear!) the races from my house. I don t follow hydroplane racing throughout the year but I do enjoy watching the races at Seafair. Here s my attempt to explain and make the case for the races to new Seattleites.
Before Microsoft, Amazon, Starbucks, etc., there were basically three major Seattle industries: (1) logging and lumber based industries like paper manufacturing; (2) maritime industries like fishing, shipbuilding, shipping, and the navy; (3) aerospace (i.e., Boeing). Vintage hydroplane racing represented the Seattle trifecta: Wooden boats with airplane engines!
The wooden U-60 Miss Thriftway circa 1955 (Thriftway is a Washinton-based supermarket that nobody outside has heard of) below is a picture of old-Seattle awesomeness. Modern hydroplanes are now made of fiberglass but two out of three isn t bad.
Although the boats are racing this year in events in Indiana, San Diego, and Detroit in addition to the two races in Washington, hydroplane racing retains deep ties to the region. Most of the drivers are from the Seattle area. Many or most of the teams and boats are based in Washington throughout the year. Many of the sponsors are unknown outside of the state. This parochialness itself cultivates a certain kind of appeal among locals.
In addition to old-Seattle/new-Seattle cultural divide, there s a class divide that I think is also worth challenging. Although the demographics of hydro-racing fans is surprisingly broad, it can seem like Formula One or NASCAR on the water. It seems safe to suggest that many of the demographic groups moving to Seattle for jobs in the tech industry are not big into motorsports. Although I m no follower of motorsports in general, I ve written before cultivated disinterest in professional sports, and it remains something that I believe is worth taking on.
It s not all great. In particular, the close relationship between Seafair and the military makes me very uneasy. That said, even with the military-heavy airshow, I enjoy the way that Seafair weekend provides a little pocket of old-Seattle that remains effectively unchanged from when I was a kid. I d encourage others to enjoy it as well!
Dear Debian/KDE users,
Hadley is a popular figure, and rightly so as he successfully introduced many newcomers to the wonders offered by R. His approach strikes some of us old greybeards as wrong---I particularly take exception with some of his writing which frequently portrays a particular approach as both the best and only one. Real programming, I think, is often a little more nuanced and aware of tradeoffs which need to be balanced. As a book on another language once popularized: "There is more than one way to do things." But let us leave this discussion for another time.
As the reach of the Hadleyverse keeps spreading, we sometimes find ourselves at the receiving end of a cost/benefit tradeoff. That is what this post is about, and it uses a very concrete case I encountered yesterday.
As blogged earlier, the RcppZiggurat package was updated. I had not touched it in a year, but Brian Ripley had sent a brief and detailed note concerning something flagged by the Solaris compiler (correctly suggesting I replace fabs() with abs() on integer types). (Allow me to stray from the main story line here for a second to stress just how insane a work load he is carrying, essentially for all of us. R and the R community are so just so indebted to him for all his work---which makes the usual social media banter about him so unfortunate. But that too shall be left for another time.) Upon making the simple fix, and submitting to GitHub the usual Travis CI was triggered. And here is what I saw:

- ./travis-tool.sh install_r RcppGSL rbenchmark microbenchmark highlight
which implies source builds of the four listed packages and all their dependencies with the following line implying binary installations of already built packages:
- ./travis-tool.sh install_aptget libgsl0-dev r-cran-rcppgsl r-cran-rbenchmark r-cran-microbenchmark r-cran-highlight
In this particular case I also needed to build a binary package of my RcppGSL package as this one is not (yet) handled by Michael. I happen to have (re-)discovered the beauty of PPAs for Travis earlier this year and revitalized an older and largely dormant launchpad account I had for this PPA of mine. How to build a simple .deb package will also have to left for a future post to keep this more concise.
This can be used with the existing r-travis setup---but one needs to use the older, initial variant in order to have the ability to install .deb packages. So in the .travis.yml of RcppZiggurat I just use
before_install:
## PPA for Rcpp and some other packages
- sudo add-apt-repository -y ppa:edd/misc
## r-travis by Craig Citro et al
- curl -OL http://raw.github.com/craigcitro/r-travis/master/scripts/travis-tool.sh
- chmod 755 ./travis-tool.sh
- ./travis-tool.sh bootstrap
to add my own PPA and all is good. If you do not have a PPA, or do not want to create your own packages you can still benefit from the PPAs by Michael and "mix and match" by installing from binary what is available, and from source what is not.
Here we were able to use an all-binary approach, so let's see the resulting performance:

Follow-up: The post got some pretty immediate feedback shortly after I posted it. Craig Citro pointed out (quite correctly) that I could use r_binary_install which would also install the Ubuntu binaries based on their R packages names. Having built R/CRAN packages for Debian for so long, I am simply more used to the r-cran-* notations, and I think I was also the one contributing install_aptget to r-travis ... Yihui Xie spoke up for the "new" Travis approach deploying containers, caching of packages and explicit whitelists. It was in that very (GH-based) discussion that I started to really lose faith in the new Travis approach as they want use to whitelist each and every package. With 6900 and counting at CRAN I fear this simply does not scale. But different approaches are certainly welcome. I posted my 1:03 to 1:15 minutes result. If the "New School" can do it faster, I'd be all ears.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The full changelog can be found in official announcement mail by Alberts
and in changelog.gz files in each package, but I want to list the most
imporant improvements in this release (compared to 3.14):
metacity-common package).
In a comment to my previous
post,
Daniele asked the entirely reasonable question,
Would you like to comment on why you think that DNSSEC+DANE are not a possible and much better alternative?Where DANE fails to be a feasible alternative to the current system is that it is not widely acknowledged to be superior in every possible way . A weak demonstration of this is that no browser has implemented DANE support, and very few other TLS-using applications have, either. The only thing I use which has DANE support that I m aware of is Postfix and SMTP is an application in which the limitations of DANE have far less impact. My understanding of the limitations of DANE, for large-scale deployment, are enumerated below.
But many (~4% in past tests) of users can t resolve a TXT record when they can resolve an A record for the same name. In practice, consumer DNS is hijacked by many devices that do a poor job of implementing DNS.Consider that
TXT records are far, far older than TLSA records. It seems
likely that TLSA records would fail to be retrieved greater than 4% of the
time. Extrapolate to the likely failure rate for lookup of TLSA records would
be, and imagine what that would do to the reliability of DANE verification.
It would either be completely unworkable, or else would cause a whole new
round of just click through the security error training. Ugh.
This also impacts DNSSEC itself. Lots of recursive resolvers don t validate
DNSSEC, and some providers mangle DNS responses in some way, which breaks
DNSSEC. Since OSes don t support DNSSEC validation by default (for
example, by having the name resolution APIs indicate DNSSEC validation
status), browsers would essentially have to ship their own validating
resolver code.
Some people have concerns around the single point of control for DNS
records, too. While the weakest link nature of the CA model is terribad,
there is a significant body of opinion that replacing it with a single,
minimally-accountable organisation like ICANN isn t a great trade.
Finally, performance is also a concern. Having to go out-of-band to
retrieve TLSA records delays page generation, and nobody likes slow page
loads.
TLSA records can be manipulated in that DNS control
panel. Those records would then automatically be DNSSEC signed by the DNS
provider and served up to anyone who comes along. Ouch.
In theory, of course, you should choose a suitably secure DNS provider, to
prevent this problem. Given that there are regular hijackings of
high-profile domains (which, presumably, the owners of those domains would
also want to prevent), there is something in the DNS service provider
market which prevents optimal consumer behaviour. Market for
lemons,
perchance?
This was my seventh month working on Debian LTS. I was assigned
14.75 hours of work
by Freexian's
Debian LTS initiative.
p7zip
I did not receive any feedback from upstream for my proposed fix for
CVE-2015-1038
mentioned last month, so I went ahead and uploaded it based on my
own testing. (I also uploaded the fix to wheezy-security,
jessie-security and sid.)
Afterwards, I received a request from upstream for a patch
against their latest release (even the version in sid is quite
a long way behind that), so I ported the fix forward to that.
linux-2.6
I backported further security fixes, but had to give up on one
(CVE-2014-8172,
AIO soft lockup) as the fix depends on wide-ranging changes.
For CVE-2015-1805
(pipe iovec overrun leading to memory corruption), the upstream fix
was also not applicable, but this looked so serious that we needed
to fix it anyway. Red Hat had already fixed this in their
2.6.32-based kernel and they didn't have overlapping changes to the
pipe implementation, so I was able to extract this fix from their
source tarball. I uploaded and
issued DLA-246-1.
Unfortunately, I failed to notice that Linux 2.6.32.66 had
introduced two regressions that were fixed in 2.6.32.67. While
these didn't appear in my testing, one of them did affect several
users that were quick to upgrade. I applied the upstream fixes,
made a second upload and issued DLA-246-2.
I also triaged the issues that are still unfixed, and I spent some
time working on a fix for
CVE-2015-1350
(unprivileged chown removes setcap attribute), but I haven't yet
completed the backport to 2.6.32 or tested it.
openssl
I looked at OpenSSL, which is still marked as affected by
CVE-2015-4000
(encryption downgrade aka Logjam). After discussion with the LTS
team I made a note of the current situation, which is that a full
fix (rejecting Diffie-Hellman keys shorter than 1024 bits) must wait
until more servers have been upgraded.
Update on 2015-07-02 22:15 UTC: as Petter Reinholdtsen noted in the comments:
Try adding /etc/apt/apt.conf.d/90squid with content like this: Acquire::http::Pipeline-Depth 0; It turn off the feature in apt confusing proxies.this indeed avoids those Hash Sum mismatch failures with HAProxy as well. Thanks, Petter! Many of you might know apt s Hash Sum mismatch issue and there are plenty of bug reports about it (like #517874, #624122, #743298 + #762079). Recently I saw the Hash Sum mismatch usually only when using random mirrors with e.g. httpredir.debian.org in apt s sources.list, but with a static mirror such issues usually don t exist anymore. A customer of mine has a Debian mirror and this issue wasn t a problem there neither, until recently: Since the mirror also includes packages provided to customers and the mirror needs to be available 24/7 we decided to provide another instance of the mirror and put those systems behind HAProxy (version 1.5.8-3 as present in Debian/jessie). The HAProxy setup worked fine and we didn t notice any issues in our tests, until the daily Q/A builds randomly started to report failures:
Failed to fetch http://example.org/foobar_amd64.deb Hash Sum mismatchWhen repeating the download there was no problem though. This problem only appeared about once every 15-20 minutes with random package files and it affected only Debian/squeeze clients (wheezy and jessie aren t affected at all). The problem also didn t appear when directly accessing the mirrors behind HAproxy. We tried plenty of different options for apt (Acquire::http::No-Cache=true, Acquire::http::No-Partial=true, ) and also played with some HAProxy configurations, nothing really helped. With apt s Debug::Acquire::http=True we saw that there really was a checksum failure and HTTP status code 102 ( Processing , or in terms of apt: Waiting for headers ) seems to be involved. The actual problem between apt on Debian/squeeze and HAProxy is still unknown to us though. While digging deeper into this issue is on my todo list yet, I found a way to avoid those Hash Sum mismatch failures: switch from http to https in sources.list. As soon as https is used the problem doesn t appear anymore. I m documenting it here just in case anyone else should run into it.
Jo and I spent a few days in Sweden and had an awesome time! The
main reason for being there was Leif and Maria's wedding way up north
in Skellefte . They cunningly organised their ceremony for the
Midsummer weekend, which was an excellent plan - we had a full weekend
of partying while we were there. :-)
We had some time to ourselves while we were there, so we wandered
about a little and got to see some of the beautiful coastal
countryside.
Then on the way home we stopped off in Ume to visit Mattias
Wadenstein (maswan) and his wife Melanie, and he showed me around some
of the machines that he's been admining on behalf of Debian. Maybe I'm
a sad geek, but I feel quite a bond with one of the machines there,
pettersson.debian.org. It's the official CD build
machine for Debian, and I've been responsible for thrashing it really
hard for the last 5 years or so... :-)
Massive thanks to the
University of Ume and
their Academic Computer Club for
hosting Debian machines and serving all the CD images for us!
The only downsides from the trip were the massive tiredness
(midnight sun is pretty, but notconducive to sleep!)
the mosquito bites and the nasty plague^Wcold that we picked up while
we were there... Ah well. :-)
Last weekend Hermann Zapf, one of the greatest font designers and typographers of our times, has passed away at the age of 96. Much has been written about his life, his immense sphere of influence, his excellent typefaces, and some of the links to other obituaries can be found here at the end. There is nothing of value I can contribute, but let Hermann Zapf speak for himself with a few images from two books by or dedicated to him.
The books I have taken these photos from are:
In his long life, Hermann Zapf has created a wide range of typefaces, some if which have become the main staple in the printing business, most notably Palatino, Janson, Optima, and Zapf Dingbats. But there are many more:
While probably not as popular anymore as some years (decades?) ago, Palatino still is one of my favorite typefaces. I often wished to have the freedom to choose my own fonts for publications, but normally that doesn t happen. Palatino has found many offspring (or imitations?), and there are excellent renderings for use with (La)TeX, too.
Hermann Zapf also had connections with Don Knuth and the AMS (American Mathematical Society), and he designed for the AMS the Euler typeface, using Knuth s MetaFont program.
Last weekend, when I heard of Hermann Zapf s death, I was at a conference at Tsukuba University. What a coincidence that all the signs in the building (floor numbers, indications, etc) were set
in another of his great typefaces, the Optima. I had a strange feeling, and reset my presentation in Optima/Euler in honor of Hermann Zapf s great life.
Although I never met Hermann Zapf personally I missed the TeX conferences where he participated it was to a great extent his influence that years ago I got started in typography. Living a life like he did, dedicating himself to the beauty and expressiveness of letters, design, calligraphy, seems like a dream for me. But beauty can be found even in the strangest equation in mathematics, the queerest theorem in logic. While not all of us are as gifted as Hermann Zapf in creating beauty, we should strive to increase our ability to appreciate the beauty.
The beauty of letters is subtle, evasive, and needs training, that is practice, to be appreciated. Hermann Zapf taught us how even self-training can lead to excellence. His letters, which I meet every day, are a permanent reminder to strive for beauty and clarity, and never relent in our exercise.
Thanks Hermann Zapf.
Links to tributes around the world
English
Linotype (Type foundry where Zapf started working)
What happened about the reproducible
builds effort for this week:
Presentations
On May 26th,Holger Levsen presented reproducible builds in Debian at CCC Berlin for the Datengarten 52. The presentation was in German and the slides in English. Audio and video recordings are available.
Toolchain fixes
pyuic output imports in stable order. Original patch by Reiner Herrmann.zip or unzip.control.tar. The patch has also been submitted based on the main branch.
Daniel Kahn Gillmor proposed a patch to add support for externally-supplying build date to help2man. This sparkled a discussion about agreeing on a common name for an environment variable to hold the date that should be used. It seems opinions are converging on using SOURCE_DATE_UTC which would hold a ISO-8601 formatted date in UTC) (e.g. 2015-06-05T01:08:20Z). Kudos to Daniel, Brendan O'Dea, Ximin Luo for pushing this forward.
Lunar proposed a patch to Tar upstream adding a --clamp-mtime option as a generic solution for timestamp variations in tarballs which might also be useful for dpkg. The option changes the behavior of --mtime to only use the time specified if the file mtime is newer than the given time. So far, upstream is not convinced that it would make a worthwhile addition to Tar, though.
Daniel Kahn Gillmor reached out to the libburnia project to ask for help on how to make ISO created with xorriso reproducible. We should reward Thomas Schmitt with a model upstream trophy as he went through a thorough analysis of possible sources of variations and ways to improve the situation. Most of what is missing with the current version in Debian is available in the latest upstream version, but libisoburn in Debian needs help. Daniel backported the missing option for version 1.3.2-1.1.
akira submitted a new issue to Doxygen upstream regarding the timestamps added to the generated manpages.
Packages fixed
The following 49 packages became reproducible due to changes in their
build dependencies:
activemq-protobuf,
bnfc,
bridge-method-injector,
commons-exec,
console-data,
djinn,
github-backup,
haskell-authenticate-oauth,
haskell-authenticate,
haskell-blaze-builder,
haskell-blaze-textual,
haskell-bloomfilter,
haskell-brainfuck,
haskell-hspec-discover,
haskell-pretty-show,
haskell-unlambda,
haskell-x509-util,
haskelldb-hdbc-odbc,
haskelldb-hdbc-postgresql,
haskelldb-hdbc-sqlite3,
hasktags,
hedgewars,
hscolour,
https-everywhere,
java-comment-preprocessor,
jffi,
jgit,
jnr-ffi,
jnr-netdb,
jsoup,
lhs2tex,
libcolor-calc-perl,
libfile-changenotify-perl,
libpdl-io-hdf5-perl,
libsvn-notify-mirror-perl,
localizer,
maven-enforcer,
pyotherside,
python-xlrd,
python-xstatic-angular-bootstrap,
rt-extension-calendar,
ruby-builder,
ruby-em-hiredis,
ruby-redcloth,
shellcheck,
sisu-plexus,
tomcat-maven-plugin,
v4l2loopback,
vim-latexsuite.
The following packages became reproducible after getting fixed:
modified-by option.__DATE__ and __TIME__ macros from source.debian/changelog entry in manpage.debian/changelog entry.HTML_TIMESTAMP to NO in Doxygen configuration.HTML_TIMESTAMP to NO in Doxygen configuration.HTML_TIMESTAMP set to NO.$datetime from the file api_footer.html.$datetime from the file footer.html.$PATH. The second build will be done with a /i/capture/the/path added. (Holger Levsen)
Holger Levsen with the help of Alexander Couzens wrote extra job to test the reproducibility of coreboot. Thanks James McCoy for helping with certificate issues.
Mattia Rizollo made some more internal improvements.
strip-nondeterminism development
Andrew Ayer released strip-nondeterminism/0.008-1. This new version fixes the gzip handler so that it now skip adding a predetermined timestamp when there was none.
Holger Levsen sponsored the upload.
Documentation update
The pages about timestamps in manpages generated by Doxygen, GHC .hi files, and Jar files have been updated to reflect their status in upstream.
Markus Koschany documented an easy way to prevent Doxygen to write timestamps in HTML output.
Package reviews
83 obsolete
reviews have
been removed, 71 added and 48 updated this week.
Meetings
A meeting was held on 2015-06-03. Minutes and full logs are available.
It was agreed to hold such a meeting every two weeks for the time being. The time of the next meeting should be announced soon.
$ /usr/bin/ssl-cert-check -s enc.com.au -p 443 Host Status Expires Days ----------------------------------------------- ------------ ------------ ---- unable to load certificate 140364897941136:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE unable to load certificate 139905089558160:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE unable to load certificate 140017829234320:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE unable to load certificate 140567473276560:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE enc.com.au:443 Expired -2457182
$ echo "" /usr/bin/openssl s_client -connect enc.com.au:443 CONNECTED(00000003) 140115756086928:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:769: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 7 bytes and written 345 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated ---
$ echo "" /usr/bin/openssl s_client -connect enc.com.au:443 -servername enc.com.au CONNECTED(00000003) depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO ECC Certification Authority verify error:num=20:unable to get local issuer certificate --- (lots of good SSL type messages)
TLSSERVERNAME="FALSE"
$ ssl-cert-check -s enc.com.au -p https Host Status Expires Days ----------------------------------------------- ------------ ------------ ---- enc.com.au:https Valid Sep 30 2015 114
I've recently found myself explaining polkit (formerly PolicyKit) to one of
Collabora's clients, and thought that
blogging about the same topic might be useful for
other people who are confused by it;
so, here is why udisks2 and polkit are the way they are.
As always, opinions in this blog are my own, not Collabora's.
Privileged actions
Broadly, there are two ways a process can do something: it can do it
directly (i.e. ask the kernel directly), or it can use inter-process
communication to ask a service to do that operation on its behalf.
If it does it directly, the components that say whether it can succeed
are the Linux kernel's normal permissions checks
(DAC),
and if configured, AppArmor, SELinux or a similar
MAC layer.
All very simple so far.
Unfortunately, the kernel's relatively coarse-grained checks are not sufficient
to express the sorts of policies that exist on a desktop/laptop/mobile
system. My favourite example for this sort of thing is mounting filesystems.
If I plug in a USB stick with a FAT filesystem, it's reasonable to expect
my chosen user interface to either mount it automatically, or
let me press a button to mount it. Similarly, to avoid data loss, I
should be able to unmount it when I'm finished with it.
However, mounting and unmounting a USB stick is fundamentally the same
system call as mounting and unmounting any other filesystem - and if
ordinary users can do arbitrary mount system calls, they can cause all
sorts of chaos, for instance by mounting a filesystem that contains setuid
executables (privilege escalation), or umounting a critical OS filesystem
like /usr (denial of service). Something needs to arbitrate: you can mount
filesystems, but only under certain conditions .
The kernel developer motto for this sort of thing is mechanism, not policy :
they are very keen to avoid encoding particular environments' policies
(the sort of thing you could think of as business rules ) in the kernel,
because that makes it non-generic and hard to maintain. As a result,
direct mount/unmount actions are only allowed by privileged processes,
and it's up to user-space processes to arrange for a privileged process
to make the desired mount syscall.
Here are some other privileged actions which laptop/desktop users can
reasonably expect to just work , with or without requiring a sysadmin-like
(root-equivalent) user:
% ls -l /bin/mount
-rwsr-xr-x 1 root root 40000 May 22 11:37 /bin/mount
If you look at it from an odd angle, this is inter-process communication
across a privilege boundary: I run the setuid executable, creating a
process. Because the executable has the setuid bit set, the kernel makes
the process highly privileged: its effective uid is root, and it has
all the necessary capabilities to mount filesystems. I submit the request
by passing it in the command-line arguments. mount does some checks -
specifically, it looks in /etc/fstab to see whether the filesystem I'm
trying to mount has the user or users flag - then carries out the
mount system call.
There are a few obvious problems with this:
udisks2 is one such privileged daemon, which uses D-Bus as its IPC channel.
D-Bus is a commonly-used inter-process system; one of its
intended/designed uses is to let user processes and system services
communicate, especially this sort of communication between a privileged
daemon and its less-privileged clients.
People sometimes criticize D-Bus as not doing anything you couldn't do yourself
with some AF_UNIX sockets. Well, no, of course it doesn't - the important
bit of the reference implementation and the various interoperable
reimplementations consists of a daemon and some AF_UNIX sockets, and the
rest is
a simple matter of programming.
However, it's sufficient for most uses in its problem space, and is usually better than inventing your
own.
The advantage of D-Bus over doing your own thing is precisely that you
are not doing your own thing: good IPC design is hard, and D-Bus makes
some structural decisions so that fewer application
authors have to think about them. For instance, it has a central hub daemon
(the dbus-daemon, or message bus ) so that n communicating applications
don't need O(n ) sockets; it uses the dbus-daemon to provide a
total message ordering
so you don't have to think about message reordering; it has a distributed
naming model (which can also be used as a distributed mutex) so you don't
have to design that; it has a serialization format and a type system so you
don't have to design one of those; it has a framework for activating"
run-on-demand daemons so they don't have to use resources initially,
implemented using a setuid helper and/or systemd; and so on.
If you have religious objections to D-Bus, you can mentally replace
D-Bus with AF_UNIX or something and most of this article will
still be true.
Is this OK?
In either case - exec'ing a privileged helper, or submitting a request to a
privileged daemon via IPC - the privileged process has two questions that
it needs to answer before it does its work:
/bin/mount only executable by a particular group,
or only executable by certain AppArmor profiles, or similar. That works
up to a point, but cannot distinguish between physically-present and
not-physically-present users, or other facts that might be interesting
to your local security policy.
Similarly, in the IPC model, you can make certain communication channels
impossible, for instance by using dbus-daemon's ability to decide which
messages to deliver, or AF_UNIX sockets' filesystem permissions,
or a MAC framework like AppArmor.
Both of these are quite coarse-grained checks which don't really
understand the finer details of what is going on. If the answer to
"is this safe? is something of the form maybe, it depends on... ,
then they can't do the right thing: they must either let it through
and let the domain-specific privileged process do the check, or deny it
and lose potentially useful functionality.
For instance, in an AppArmor environment, some applications have absolutely no
legitimate reason to talk to udisks2, so the AppArmor policy can just
block it altogether. However, once again, this is a coarse-grained
check: the kernel has mechanism, not policy, and it doesn't know what
the service does or why. If the application does need to be able to talk
to the service at all, then finer-grained access control
(obeying some, but not all, requests) has to be the service's job.
dbus-daemon does have the ability to match messages in a relatively
fine-grained way, based on the object path, interface and member in
the message, as well as the routing information that it uses itself
(i.e. the source and destination). However, it is not clear that this
makes a great deal of sense conceptually: these are facts about the
mechanics of the IPC, not facts about the domain-specific request
(because the mechanics of the IPC are all that dbus-daemon understands).
For instance, taking the udisks2 example again, dbus-daemon can't
distinguish between an attempt to adjust mount options for a USB stick
(probably fine) and an attempt to adjust mount options for /usr (not good).
To have a domain-specific security policy, we need a domain-specific
component, for instance udisks2, to get involved. Unlike dbus-daemon,
udisks2 knows that not all disks are equal, knows which
categories make sense to distinguish, and can identify which categories
a particular disk is in. So udisks2 can make a more informed decision.
So, a naive approach might be to write a function in udisks2 that looks
something like this pseudocode:
may_i_mount_this_disk (user, disk, mount options) boolean
if (user is root user is root-equivalent)
return true;
if (disk is not removable)
return false;
if (mount options are scary)
return false;
if (user is in manipulate non-local disks group)
return true;
if (user is not logged-in locally)
return false;
# https://en.wikipedia.org/wiki/Multiseat_configuration
if (user is not logged-in on the same seat where the disk is
plugged in)
return false;
return true;
Delegating the security policy to something central
The pseudocode security policy outlined above is reasonably
complicated already, and doesn't necessarily cover everything that you
might want to consider.
Meanwhile, not every system is the same. A general-purpose Linux
distribution like Debian might run on server/mainframe systems with only
remote users, personal laptops/desktops with one root-equivalent user,
locked-down corporate laptops/desktops, mobile devices and so on; these
systems should not necessarily all have the same security policy.
Another interesting factor is that for some privileged operations, you
might want to carry out interactive authorization: ask the requesting user
to confirm that the action (which might have come from a background
process) should take place (like Windows'
UAC),
or to prove that the person currently at the keyboard is the same as
the person who logged in by giving their password
(like sudo).
We could in principle write code for all of this in udisks2, and in
NetworkManager, and in systemd, ... - but that clearly doesn't scale,
particularly if you want the security policy to be configurable.
Enter polkit
(formerly PolicyKit), a system service for applying security policies
to actions.
The way polkit works is that the application does its domain-specific
analysis of the request - in the case of udisks2, whether the device to
be mounted is removable, whether the mount options are reasonable, etc. -
and converts it into an action. The action gives polkit a way to
distinguish between things that are conceptually different, without
needing to know the specifics. For instance, udisks2 currently divides
up filesystem-mounting into org.freedesktop.udisks2.filesystem-mount,
org.freedesktop.udisks2.filesystem-mount-fstab,
org.freedesktop.udisks2.filesystem-mount-system and
org.freedesktop.udisks2.filesystem-mount-other-seat.
The application also finds the identity of the user making the request.
Next, the application sends the action, the identity of the requesting
user, and any other interesting facts to polkit. As currently implemented,
polkit is a D-Bus service, so this is an IPC request via D-Bus.
polkit consults its database of policies in order to choose one of several
results:
polkit decide this? The first thing is that it reads the
machine-readable description of the actions, in /usr/share/polkit-1/actions,
which specifies a default policy. Next, it evaluates a local security policy
to see what that says. In the current version of polkit, the local security
policy is configured by writing JavaScript in /etc/polkit-1/rules.d
(local policy) and /usr/share/polkit-1/rules.d (OS-vendor defaults).
In older versions such as the one currently shipped in Debian unstable,
there was a plugin architecture; but in practice
nobody wrote plugins for it, and instead everyone used the
example local authority shipped with polkit, which was configured
via files in /etc/polkit-1/localauthority and
/etc/polkit-1/localauthority.d.
These policies can take into account useful facts like:
polkit.addRule(function(action, subject)
if ((action.id == "org.freedesktop.locale1.set-locale"
action.id == "org.freedesktop.locale1.set-keyboard"
action.id == "org.freedesktop.hostname1.set-static-hostname"
action.id == "org.freedesktop.hostname1.set-hostname"
action.id == "org.gnome.controlcenter.datetime.configure") &&
subject.local &&
subject.active &&
subject.isInGroup ("sudo"))
return polkit.Result.YES;
);
which is reasonably close to being pseudocode for active local users in the
sudo group may set the system locale, keyboard layout, hostname and time,
without needing to authenticate . A system administrator could of course
override that by dropping a higher-priority policy for some or all of these
actions into /etc/polkit-1/rules.d.
Summary
dbus-daemon)polkit evaluates a configurable policy to determine whether privileged
services should carry out requested actionsNext.