Search Results: "xam"

12 October 2015

Michael Stapelberg: ratt: Rebuild All The Things!

When uploading a new library package which changes its API/behavior in a subtle way, typically you will only hear about the downstream breakage after you ve uploaded the new library package (via bug reports telling you that your package FTBFS, fails to build from source). I prefer quality-assurance to happen proactively rather than reactively whenever possible, so I set out to write a tool which automates the rebuild of reverse-build-dependencies, i.e. all packages whose build process could be affected by the package one is about to upload. The result is a tool called ratt, which is short for Rebuild All The Things! . It injects the newly-built Debian package you provide it, figures out all the reverse-build-dependencies, invokes sbuild for all of them, and finally presents you with a list of packages that failed to build. To demonstrate how the tool works, let s assume we want to upload a new version of the Go library github.com/jacobsa/gcloud. To keep this example brief, we don t actually do anything to the package but bump its version number:
midna /tmp $ debcheckout golang-github-jacobsa-gcloud-dev
declared git repository at git://anonscm.debian.org/pkg-go/packages/golang-github-jacobsa-gcloud.git
git clone git://anonscm.debian.org/pkg-go/packages/golang-github-jacobsa-gcloud.git golang-github-jacobsa-gcloud-dev ...
Cloning into 'golang-github-jacobsa-gcloud-dev'...
remote: Counting objects: 84, done.
remote: Compressing objects: 100% (73/73), done.
remote: Total 84 (delta 19), reused 0 (delta 0)
Receiving objects: 100% (84/84), 87.70 KiB   0 bytes/s, done.
Resolving deltas: 100% (19/19), done.
Checking connectivity... done.
midna /tmp $ cd golang-github-jacobsa-gcloud-dev
midna /tmp/golang-github-jacobsa-gcloud-dev $ dch -i -m 'dummy new version'
midna /tmp/golang-github-jacobsa-gcloud-dev $ git commit -a -m 'dummy new version'
midna /tmp/golang-github-jacobsa-gcloud-dev $ gbp buildpackage --git-pbuilder
[ ]
So far, so good. Now let s invoke ratt to rebuild all reverse-build-dependencies:
midna /tmp/golang-github-jacobsa-gcloud-dev $ ratt golang-github-jacobsa-gcloud_0.0\~git20150709-2_amd64.changes         
2015/08/16 11:48:41 Loading changes file "golang-github-jacobsa-gcloud_0.0~git20150709-2_amd64.changes"
2015/08/16 11:48:41  - 1 binary packages: golang-github-jacobsa-gcloud-dev
2015/08/16 11:48:41  - corresponding .debs (will be injected when building):
2015/08/16 11:48:41     golang-github-jacobsa-gcloud-dev_0.0~git20150709-2_all.deb
2015/08/16 11:48:41 Loading sources index "/var/lib/apt/lists/ftp.ch.debian.org_debian_dists_sid_contrib_source_Sources"
2015/08/16 11:48:41 Loading sources index "/var/lib/apt/lists/ftp.ch.debian.org_debian_dists_sid_main_source_Sources"
2015/08/16 11:48:43 Loading sources index "/var/lib/apt/lists/ftp.ch.debian.org_debian_dists_sid_non-free_source_Sources"
2015/08/16 11:48:43 Building golang-github-jacobsa-ratelimit_0.0~git20150723.0.2ca5e0c-1 (commandline: [sbuild --arch-all --dist=sid --nolog golang-github-jacobsa-ratelimit_0.0~git20150723.0.2ca5e0c-1 --extra-package=golang-github-jacobsa-gcloud-dev_0.0~git20150709-2_all.deb])
2015/08/16 11:49:19 Build results:
2015/08/16 11:49:19 PASSED: golang-github-jacobsa-ratelimit_0.0~git20150723.0.2ca5e0c-1
Thanks for reading this far, and I hope ratt makes your life a tiny bit easier. As ratt just entered Debian unstable, you can install it using apt-get install ratt. If you have any feedback, I m eager to hear it.

11 October 2015

Ben Armstrong: Colours of Autumn 2015, Bluff Trail

My friend Ross Mayhew and I enjoyed a perfect Autumn afternoon yesterday, full of colours on the Bluff Trail. Not all of these photos do justice to the splendour and intricate detail I had hoped to capture, but I hope you enjoy them all the same. Click the photo to start the slideshow.
Late afternoon at the top of Pot Lake loopLate afternoon at the top of Pot Lake loop
On my way to hike, the Canada holly hints of things to come Canada holly berries bright red and close to the branch (vs. dusky red false holly berries on long stems) Unsure which fern this is. Ross says Christmas, but the leaf margins aren t serrated, but smooth. Sheltered by this mossy stump, a pretty mushroom Moss found climbing up a rotted stump, peculiar in that it has flat, fern-like fronds Ross and I spent a while examining this peculiar flat-fronded moss The reds of the huckleberries and maples were striking A tiny fern by a trickle of water across the path A familiar view overlooking Cranberry Lake, now in its fall splendour Brilliant Canada holly berries along the bog at our hike s end
facebooktwittergoogle_plusredditpinterestlinkedinmail

9 October 2015

Antoine Beaupr : Finding Debian release history with etckeeper

I praise etckeeper enough. That you use or not formal configuration management tools, you should have etckeeper installed, even if to do ad-hoc changes in an emergency. For example, I just thought I could use it to figure out which Debian release I originally installed on this machine, and when I did the upgrades. Turns out it's amazingly easy:
cd /etc
sudo git log --oneline -p --no-prefix  /etc/issue
Here is the result here:
root@marcos:/etc# git log --oneline -p --no-prefix --pretty=format:'%s (%ai)' /etc/issue   cat
Initial commit (2011-02-23 00:14:08 -0500)
diff --git issue issue
new file mode 100644
index 0000000..9d52ed2
--- /dev/null
+++ issue
@@ -0,0 +1,2 @@
+Debian GNU/Linux 6.0 \n \l
+
saving uncommitted changes in /etc prior to apt run (2011-05-22 19:08:20 -0400)
diff --git issue issue
index 9d52ed2..647d490 100644
--- issue
+++ issue
@@ -1,2 +1,2 @@
-Debian GNU/Linux 6.0 \n \l
+Debian GNU/Linux wheezy/sid \n \l
committing changes in /etc after apt run (2013-01-23 09:40:27 -0500)
diff --git issue issue
index 647d490..d363ace 100644
--- issue
+++ issue
@@ -1,2 +1,2 @@
-Debian GNU/Linux wheezy/sid \n \l
+Debian GNU/Linux 7.0 \n \l
committing changes in /etc after apt run (2013-06-15 13:02:04 -0400)
diff --git issue issue
index d363ace..efc8255 100644
--- issue
+++ issue
@@ -1,2 +1,2 @@
-Debian GNU/Linux 7.0 \n \l
+Debian GNU/Linux 7 \n \l
committing changes in /etc after apt run (2014-02-02 23:28:12 -0500)
diff --git issue issue
index efc8255..e65d112 100644
--- issue
+++ issue
@@ -1,2 +1,2 @@
-Debian GNU/Linux 7 \n \l
+Debian GNU/Linux jessie/sid \n \l
committing changes in /etc after apt run (2014-12-18 11:32:43 -0500)
diff --git issue issue
index e65d112..6478eed 100644
--- issue
+++ issue
@@ -1,2 +1,2 @@
-Debian GNU/Linux jessie/sid \n \l
+Debian GNU/Linux 8 \n \l
In other words: Unfortunately, there isn't much more details about the exact upgrade points, especially since /etc/os-release is a symlink starting with Jessie. Besides things are much more in flux than we would like to believe, especially when you run a rolling distribution like "testing", but it still gives a good idea of my upgrade history: There is of course more information available directly in git log, namely the exact package version changes. With some more commandline filtering, we can see exactly when each upgrade was done, including minor releases:
# git log --date iso --grep base-files --reverse   egrep "^commit ^Date base-files"
commit 34ad962ff10c6e4e201378698e0fe0d4b03c8c39
Date:   2011-04-02 22:21:11 -0400
    -base-files 6.0
    +base-files 6.0squeeze1
commit 1cba7d2e097091e86eba1a2d8e4f5a9771e746a1
Date:   2011-07-09 19:40:05 -0400
    -base-files 6.3
    +base-files 6.4
commit cb409a0fbe2f95c3cd6a7c1ff0af263b55c7d597
Date:   2011-09-28 17:54:57 -0400
    -base-files 6.4
    +base-files 6.5
commit cf6d9dab6f79f4b50e5bd80dcba1769b0aa6c84b
Date:   2012-03-25 19:18:44 -0400
    -base-files 6.5
    +base-files 6.7
commit bb1b6aab4406276388542cefd7b4eff92d960533
Date:   2012-06-29 00:42:32 -0400
    -base-files 6.7
    +base-files 6.9
commit c6d9218ba75b3276ea44d949ef3410c35713d487
Date:   2012-09-29 14:25:01 -0400
    -base-files 6.9
    +base-files 6.11
commit bd730398e572c8403b2b9c0421df64a407669877
Date:   2013-01-23 09:40:27 -0500
    -base-files 6.11
    +base-files 7.1
commit af42616d72c4fe5c4a0e43ee8948031732735ec5
Date:   2013-06-15 13:02:04 -0400
    -base-files 7.1
    +base-files 7.1wheezy1
commit d2ef5df9c689073b62b4898a12e42bb8488c8cdc
Date:   2013-10-14 18:15:11 -0400
    -base-files 7.1wheezy1
    +base-files 7.1wheezy2
commit 104ea49559eb2b6b2aff21817b3980b274882a28
Date:   2013-12-14 11:07:09 -0500
    -base-files 7.1wheezy2
    +base-files 7.1wheezy3
commit bfa19b7ad737cb22c495af9429e922e6ec46202d
Date:   2014-02-02 23:28:12 -0500
    -base-files 7.1wheezy3
    +base-files 7.2
commit 830570ca4083af0c4b87c6c7c746c9513e260cf7
Date:   2014-05-15 11:20:13 -0400
    -base-files 7.2
    +base-files 7.3
commit 4c9e6ca21bb8189810199e6d518250432a62391d
Date:   2014-07-27 19:50:14 -0400
    -base-files 7.3
    +base-files 7.5
commit 4bc6abe2ba193cb6fac4eae00fa855eb32b86400
Date:   2014-10-20 16:14:09 -0400
    -base-files 7.5
    +base-files 7.6
commit a5e2ce476982201338220a314b1f9ccb03c99517
Date:   2014-11-28 19:45:56 -0500
    -base-files 7.6
    +base-files 7.10
commit eaa69b112fee60ef71938d9a725c07f9f29b2011
Date:   2014-12-18 11:32:43 -0500
    -base-files 7.10
    +base-files 8
commit f95a199df77030dc9ee6ab55bf4fa246fa88c959
Date:   2015-07-24 12:34:04 -0400
    -base-files 8
    +base-files 8+deb8u1
commit db4ab55b7327cecea54c5fe6a65560ba0e385978
Date:   2015-09-07 19:09:47 -0400
    -base-files 8+deb8u1
    +base-files 8+deb8u2

8 October 2015

Norbert Preining: Looking at the facts: Sarah Sharp s crusade

Much has been written around the internet about this geeky kernel maintainer Sarah Sharp who left kernel development. I have now spent two hours reading through lkml posts, and want to summarize a few mails from the long thread, since most of the usual news sites just rewrap the original blog of hers without adding any background. darth-cookie The whole thread evolved out call for stable kernel review by Greg Kroah-Hartman where he complained about too many patches that are not actually in rc1 before going into stable:
<rant>
  I'm sitting on top of over 170 more patches that have been marked for
  the stable releases right now that are not included in this set of
  releases.  The fact that there are this many patches for stable stuff
  that are waiting to be merged through the main -rc1 merge window cycle
  is worrying to me.
  [...]
from where it developed into a typical Linus rant on people flagging crap for stable, followed by some jokes:
On Fri, Jul 12, 2013 at 8:47 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
> I tend to hold things off after -rc4 because you scare me more than Greg
> does ;-)
 
Have you guys *seen* Greg? The guy is a freakish giant. He *should*
scare you. He might squish you without ever even noticing.
 
              Linus
and Ingo Molnar giving advice to Greg KH:
So Greg, if you want it all to change, create some _real_ threat: be frank 
with contributors and sometimes swear a bit. That will cut your mailqueue 
in half, promise!
with Greg KH taking a funny position in answering:
Ok, I'll channel my "inner Linus" and take a cue from my kids and start
swearing more.
Up to now a pretty decent and normal thread with some jokes and poking, nobody minded, and reading through it I had a good time. The thread continues with a discussion on requirements what to submit to stable, and some side threads on particular commits. And then, out of the blue, Social Justice Warrior (SJW) Sarah Sharp pops in with a very important contribution:
Seriously, guys?  Is this what we need in order to get improve -stable?
Linus Torvalds is advocating for physical intimidation and violence.
Ingo Molnar and Linus are advocating for verbal abuse.
 
Not *fucking* cool.  Violence, whether it be physical intimidation,
verbal threats or verbal abuse is not acceptable.  Keep it professional
on the mailing lists.
 
Let's discuss this at Kernel Summit where we can at least yell at each
other in person.  Yeah, just try yelling at me about this.  I'll roar
right back, louder, for all the people who lose their voice when they
get yelled at by top maintainers.  I won't be the nice girl anymore.
Onto which Linus answers in a great way:
That's the spirit.
 
Greg has taught you well. You have controlled your fear. Now, release
your anger. Only your hatred can destroy me.
 
Come to the dark side, Sarah. We have cookies.
On goes Sarah, gearing up in her SJW mode and starting to rant:
However, I am serious about this.  Linus, you're one of the worst
offenders when it comes to verbally abusing people and publicly tearing
their emotions apart.
 
http://marc.info/?l=linux-kernel&m=135628421403144&w=2
http://marc.info/?l=linux-acpi&m=136157944603147&w=2
 
I'm not going to put up with that shit any more.
Linus himself made clear what he thinks of her:
Trust me, there's a really easy way for me to curse at people: if you
are a maintainer, and you make excuses for your bugs rather than
trying to fix them, I will curse at *YOU*.
 
Because then the problem really is you.
[...]
It is easy to verify what Linus said, by reading the above two links and the answers of the maintainers, both agreed that it was their failure and were sorry. (Mauro s answer, Rafael s answer) It is just the geeky SJW that was not even attacked (who would dare to attack a woman nowadays?). The overall reaction to her by the maintainers can be exemplified by Thomas Gleixner s post:
Just for the record. I got grilled by Linus several times over the
last years and I can't remember a single instance where it was
unjustified.
What follows is a nearly endless discussion with Sarah meandering around, changing permanently her opinion what is acceptable. Linus tried to explain to her in simple words, without success, she continues to rant around. Here arguments are so weak I had nothing but good laugh:
> Sarah, that's a pretty potent argument by Linus, that "acting 
> professionally" risks replacing a raw but honest culture with a
> polished but dishonest culture - which is harmful to developing
> good technology.
> 
> That's a valid concern. What's your reply to that argument?
 
I don't feel the need to comment, because I feel it's a straw man
argument.  I feel that way because I disagree with the definition of
professionalism that people have been pushing.
 
To me, being "professional" means treating each other with respect.  I
can show emotion, express displeasure, be direct, and still show respect
for my fellow developers.
 
For example, I find the following statement to be both direct and
respectful, because it's criticizing code, not the person:
 
"This code is SHIT!  It adds new warnings and it's marked for stable
when it's clearly *crap code* that's not a bug fix.  I'm going to revert
this merge, and I expect a fix from you IMMEDIATELY."
 
The following statement is not respectful, because it targets the
person:
 
"Seriously, Maintainer.  Why are you pushing this kind of *crap* code to
me again?  Why the hell did you mark it for stable when it's clearly
not a bug fix?  Did you even try to f*cking compile this?"
Fortunately, she was immediately corrected and Ingo Molnar wrote an excellent refutation (starting another funny thread) of all her emails, statements, accusations (all of the email is a good read):
_That_ is why it might look to you as if the person was
attacked, because indeed the actions of the top level maintainer were
wrong and are criticised.
 
... and now you want to 'shut down' the discussion. With all due respect,
you started it, you have put out various heavy accusations here and elsewhere,
so you might as well take responsibility for it and let the discussion be
brought to a conclusion, wherever that may take us, compared to your initial view?
(He retracted that last statement, though I don t see a reason for it) Last but not least, let us return to her blog post, where she states herself that:
FYI, comments will be moderated by someone other than me. As this is my blog, not a
government entity, I have the right to replace any comment I feel like with 
 fart fart fart fart .
and she made lots of use of it, I counted at least 10 instances. She seems to remove or fart fart fart any comment that is not in line with her opinion. Further evidence is provided by this post on lkml. Everyone is free to have his own opinion (sorry, his/her), and I am free to form my own opinion on Sarah Sharp by just simply reading the facts. I am more than happy that one more SJW has left Linux development, as the proliferation of cleaning of speech from any personality has taken too far a grip. Coming to my home-base in Debian, unfortunately there is no one in the position and the state of mind of Linus, so we are suffering the same stupidities imposed by social justice worriers and some brainless feminists (no, don t get me wrong, these are two independent attributes. I do NOT state that feminism is brainless) that Linus and the maintainer crew was able to fend of this time. I finish with my favorite post from that thread, by Steven Rosted (from whom I also stole the above image!):
On Tue, 2013-07-16 at 18:37 -0700, Linus Torvalds wrote:
> Emotions aren't bad. Quite the reverse. 
 
Spock and Dr. Sheldon Cooper strongly disagree.
Post Scriptum (after a bike ride) The last point by Linus is what I criticize most on Debian nowdays, it has become a sterilized over-governed entity, where most fun is gone. Making fun is practically forbidden, since there is the slight chance that some minority somewhere on this planet might feel hurt, and by this we are breaking the CoC. Emotions are restricted to the Happy happy how nice we are and how good we are level of US and also Japanese self-reenforcement society. Post Post Scriptum I just read Sarah Sharp s post on What makes a good community? , and without giving a full account or review, I am just pi**ed by the usage of the word microaggressions I can only recommend everyone to read this article and this article to get a feeling how bs the idea of microaggressions has taken over academia and obviously not only academia. Post3 Scriptum I am happy to see Lars Wirzenius, Gunnar Wolf, and Mart n Ferrari opposing my view. I agree with them that my comments concerning Debian are not mainstream in Debian something that is not very surprising, though, and I think it is great that they have fun in Debian, like many other contributors. Post4 Scriptum Although nobody will read this, here is a great response from a female developer:
[...] To Linus: You're a hero to many of us. Don't change. Please. You DO
NOT need to take time away from doing code to grow a pair of breasts
and judge people's emotional states: [...]
Nothing to add here!

6 October 2015

Gergely Nagy: What dh-exec is, and what it isn't for

Strange as it may be, it turns out I never wrote about dh-exec yet, even though it is close to being four years old. Gosh, time flies so fast when you're having fun! Since its first introduction, there's been a reasonable uptake in dh-exec use: as of this writing, 129 packages build-depend on dh-exec. One might think this would be a cause for celebration, that the package is put to great use. But it's not.You see, a significant number of those 129 packages are doing it wrong, and need not build-depend on dh-exec at all.The purpose of dh-exec is to allow one to do things stock debhelper can't do, such as renaming files during the dh_install phase, or applying architecture or build profile based filtering, or doing environment variable substitution. There are many legit uses for all of these features, but there are some which can be easily solved without using dh-exec. So first, I'll talk a bit about the don'ts.What not to use dh-exec forOne of the most abused part of dh-exec is its variable substitution feature, and it is often used without need, to install multiarch-related files. While that is one intended use-case, there are few situations currently in the archive that stock debhelper can't handle. Let me explain the situation!Lets assume we have an upstream package, where the build system is something as common as autotools or CMake, for which debhelper can automatically set the appropriate flags and paths. Furthermore, lets assume that the upstream build system would install the following files:
/usr/lib/x86_64-linux-gnu/libalala.so.1.2.3
/usr/lib/x86_64-linux-gnu/libalala.so.1
/usr/lib/x86_64-linux-gnu/libalala.so
/usr/lib/x86_64-linux-gnu/libalala.a
/usr/lib/x86_64-linux-gnu/pkgconfig/libalala.pc
/usr/include/lalala/lalala.h
We want to include the first two in the libalala1 package, the rest in libalala-dev, so what do we do? We use stock debhelper, of course! That is all you need. In this case, there is absolutely no need for dh-exec. While using dh-exec without need is not much of an issue, because it only increases the space required for the build and build times by a tiny bit, I would still strongly recommend not introducing dh-exec needlessly. Why? Because of simplicity and aesthetics.So, if you find yourself doing any of these, or similar, that's a sign you are doing things wrong:
usr/lib/$ DEB_HOST_MULTIARCH 
usr/lib/$ DEB_HOST_MULTIARCH /*.so.* /usr/lib/$ DEB_HOST_MULTIARCH /
usr/lib/$ DEB_HOST_MULTIARCH /pkgconfig/*.pc /usr/lib/$ DEB_HOST_MULTIARCH /pkgconfig
usr/lib/$ DEB_HOST_MULTIARCH /package/*.so /usr/lib/$ DEB_HOST_MULTIARCH /package
Unless there are other directories under usr/lib that are not the multiarch triplet, using stock debhelper and wildcards is not only more succinct, simpler and more elegant, but also lighter on resources required.When dh-exec becomes usefulChanging installation pathsOnce you want to change where things get installed, then dh-exec becomes useful:
usr/lib/*.so.* /usr/lib/$ DEB_HOST_MULTIARCH 
usr/lib/$ DEB_HOST_MULTIARCH /package/* /usr/lib/$ DEB_HOST_MULTIARCH /package-plugins/
some/dir/*.so.* /usr/lib/$ DEB_HOST_MULTIARCH 
This usually happens when upstream's build system can't easily be taught about multiarch paths. For most autotools and CMake-based packages, this is not the case.Variable substitutionConsider this case:
#!/usr/bin/dh-exec
/usr/share/octave/packages/mpi-$ DEB_VERSION_UPSTREAM /hello2dimmat.m /usr/share/doc/octave-mpi/examples/hello2dimmat.m
/usr/share/octave/packages/mpi-$ DEB_VERSION_UPSTREAM /hellocell.m /usr/share/doc/octave-mpi/examples/hellocell.m
Here, the build supplies $ DEB_VERSION_UPSTREAM , and using dh-exec allows one to have a generic debian/links file, that does not need updating whenever the upstream version changes. We can't use wildcards here, because dh_link does not expand them.Renaming filesIn case one needs to rename files during the dh_install phase, dh-exec can be put to use for great results:
ssh-agent-filter.bash-completion => usr/share/bash-completion/completions/ssh-agent-filter
FilteringSometimes one would wish to conditionally install something based on the architecture, or the build profile. In this case, dh-exec is the tool to turn to:
<!stage1 !stage2> ../../libdde-linux26/Makeconf* usr/share/libdde_linux26
usr/lib/gvfs/gvfsd-afc                                          [!hurd-any]
usr/lib/gvfs/gvfsd-gphoto2                                      [linux-any]

5 October 2015

B lint R czey: Debian success stories: Automated signature verification

Debian was not generally seen as a bleeding-edge distribution, but it offered a perfect combination of stability and up-to-date software in our field when we chose the platform for our signature verification project. Having an active Debian Developer in the team also helped ensuring that packages which we use were in good shape when the freeze, then the release came and we can still rely on Jessie images with only a few extra packages to run our software stack. Not having to worry about the platform, we could concentrate on the core project and I m proud to announce that our start-up s algorithm won this year s Signature Verification Competition for Online Skilled Forgeries (SigWIComp2015) . The more detailed story can be read already in the English business news and is also on index.hu, a leading Hungarian news site. We are also working on a solution for categorizing users based on cursor/finger movements for targeting content, offers and ads better. This is also covered in the articles.

L szl a signature comparable in quality to the reference signatures

The verification task was not easy. The reference signatures were recorded at very low resolution and frequency and the forgers did a very good job in forging them creating a true challenge for everyone competing. At first glance it is hard to imagine that there is usable information in such small amount of recorded data, but our software is already better than me, for example in telling the difference between genuine and forged signatures. It feels like when the chess program beats the programmer again and again. :-) I would like to thank you all, who helped making Debian an awesome universal operating system and hope we can keep making every release better and better!

4 October 2015

Lunar: Reproducible builds: week 23 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Andreas Metzler uploaded autogen/1:5.18.6-1 in experimental with several patches for reproducibility issues written by Valentin Lorentz. Groovy upstream has merged a change proposed by Emmanuel Bourg to remove timestamps generated by groovydoc. Ben Hutchings submitted a patch to add support for SOURCE_DATE_EPOCH in linux-kbuild as an alternate way to specify the build timestamp. Reiner Herrman has sent a patch adding support for SOURCE_DATE_EPOCH in docbook-utils. Packages fixed The following packages became reproducible due to changes in their build dependencies: commons-csv. fest-reflect, sunxi-tools, xfce4-terminal, The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: Tomasz Rybak uploaded pycuda/2015.1.3-1 which should fix reproducibility issues. The package has not been tested as it is in contrib. akira found an embedded code copy of texi2html in fftw. reproducible.debian.net Email notifications are now only sent once a day per package, instead of on each status change. (h01ger) disorderfs has been temporarily disabled to see if it had any impact on the disk space issues. (h01ger) When running out of disk space, build nodes will now automatically detect the problem. This means test results will not be recorded as FTBFS and the problem will be reported to Jenkins maintainers. (h01ger) The navigation menu of package pages has been improved. (h01ger) The two amd64 builders now use two different kernel versions: 3.16 from stable and 4.1 from backports on the other. (h01ger) We now graph the number of packages which needs to be fixed. (h01ger) Munin now creates graphs on how many builds were performed by build nodes (example). (h01ger) A migration plan has been agreed with DSA on how to turn Jenkins into an official Debian service. A backport of jenkins-job-builder for Jessie is currently missing. (h01ger) Package reviews 119 reviews have been removed, 103 added and 45 updated this week. 16 fail to build from source issues were reported by Chris Lamb and Mattia Rizzolo. New issue this week: timestamps_in_manpages_generated_by_docbook_utils. Misc. Allan McRae has submitted a patch to make ArchLinux pacman record a .BUILDINFO file.

2 October 2015

Daniel Pocock: Want to be selected for Google Summer of Code 2016?

I've mentored a number of students in 2013, 2014 and 2015 for Debian and Ganglia and most of the companies I've worked with have run internships and graduate programs from time to time. GSoC 2015 has just finished and with all the excitement, many students are already asking what they can do to prepare and be selected for Outreachy or GSoC in 2016. My own observation is that the more time the organization has to get to know the student, the more confident they can be selecting that student. Furthermore, the more time that the student has spent getting to know the free software community, the more easily they can complete GSoC. Here I present a list of things that students can do to maximize their chance of selection and career opportunities at the same time. These tips are useful for people applying for GSoC itself and related programs such as GNOME's Outreachy or graduate placements in companies. Disclaimers There is no guarantee that Google will run the program again in 2016 or any future year until the Google announcement. There is no guarantee that any organization or mentor (including myself) will be involved until the official list of organizations is published by Google. Do not follow the advice of web sites that invite you to send pizza or anything else of value to prospective mentors. Following the steps in this page doesn't guarantee selection. That said, people who do follow these steps are much more likely to be considered and interviewed than somebody who hasn't done any of the things in this list. Understand what free software really is You may hear terms like free software and open source software used interchangeably. They don't mean exactly the same thing and many people use the term free software for the wrong things. Not all projects declaring themselves to be "free" or "open source" meet the definition of free software. Those that don't, usually as a result of deficiencies in their licenses, are fundamentally incompatible with the majority of software that does use genuinely free licenses. Google Summer of Code is about both writing and publishing your code and it is also about community. It is fundamental that you know the basics of licensing and how to choose a free license that empowers the community to collaborate on your code well after GSoC has finished. Please review the definition of free software early on and come back and review it from time to time. The The GNU Project / Free Software Foundation have excellent resources to help you understand what a free software license is and how it works to maximize community collaboration. Don't look for shortcuts There is no shortcut to GSoC selection and there is no shortcut to GSoC completion. The student stipend (USD $5,500 in 2014) is not paid to students unless they complete a minimum amount of valid code. This means that even if a student did find some shortcut to selection, it is unlikely they would be paid without completing meaningful work. If you are the right candidate for GSoC, you will not need a shortcut anyway. Are you the sort of person who can't leave a coding problem until you really feel it is fixed, even if you keep going all night? Have you ever woken up in the night with a dream about writing code still in your head? Do you become irritated by tedious or repetitive tasks and often think of ways to write code to eliminate such tasks? Does your family get cross with you because you take your laptop to Christmas dinner or some other significant occasion and start coding? If some of these statements summarize the way you think or feel you are probably a natural fit for GSoC. An opportunity money can't buy The GSoC stipend will not make you rich. It is intended to make sure you have enough money to survive through the summer and focus on your project. Professional developers make this much money in a week in leading business centers like New York, London and Singapore. When you get to that stage in 3-5 years, you will not even be thinking about exactly how much you made during internships. GSoC gives you an edge over other internships because it involves publicly promoting your work. Many companies still try to hide the potential of their best recruits for fear they will be poached or that they will be able to demand higher salaries. Everything you complete in GSoC is intended to be published and you get full credit for it. Imagine a young musician getting the opportunity to perform on the main stage at a rock festival. This is how the free software community works. It is a meritocracy and there is nobody to hold you back. Having a portfolio of free software that you have created or collaborated on and a wide network of professional contacts that you develop before, during and after GSoC will continue to pay you back for years to come. While other graduates are being screened through group interviews and testing days run by employers, people with a track record in a free software project often find they go straight to the final interview round. Register your domain name and make a permanent email address Free software is all about community and collaboration. Register your own domain name as this will become a focal point for your work and for people to get to know you as you become part of the community. This is sound advice for anybody working in IT, not just programmers. It gives the impression that you are confident and have a long term interest in a technology career. Choosing the provider: as a minimum, you want a provider that offers DNS management, static web site hosting, email forwarding and XMPP services all linked to your domain. You do not need to choose the provider that is linked to your internet connection at home and that is often not the best choice anyway. The XMPP foundation maintains a list of providers known to support XMPP. Create an email address within your domain name. The most basic domain hosting providers will let you forward the email address to a webmail or university email account of your choice. Configure your webmail to send replies using your personalized email address in the From header. Update your ~/.gitconfig file to use your personalized email address in your Git commits. Create a web site and blog Start writing a blog. Host it using your domain name. Some people blog every day, other people just blog once every two or three months. Create links from your web site to your other profiles, such as a Github profile page. This helps reinforce the pages/profiles that are genuinely related to you and avoid confusion with the pages of other developers. Many mentors are keen to see their students writing a weekly report on a blog during GSoC so starting a blog now gives you a head start. Mentors look at blogs during the selection process to try and gain insight into which topics a student is most suitable for. Create a profile on Github Github is one of the most widely used software development web sites. Github makes it quick and easy for you to publish your work and collaborate on the work of other people. Create an account today and get in the habbit of forking other projects, improving them, committing your changes and pushing the work back into your Github account. Github will quickly build a profile of your commits and this allows mentors to see and understand your interests and your strengths. In your Github profile, add a link to your web site/blog and make sure the email address you are using for Git commits (in the ~/.gitconfig file) is based on your personal domain. Start using PGP Pretty Good Privacy (PGP) is the industry standard in protecting your identity online. All serious free software projects use PGP to sign tags in Git, to sign official emails and to sign official release files. The most common way to start using PGP is with the GnuPG (GNU Privacy Guard) utility. It is installed by the package manager on most Linux systems. When you create your own PGP key, use the email address involving your domain name. This is the most permanent and stable solution. Print your key fingerprint using the gpg-key2ps command, it is in the signing-party package on most Linux systems. Keep copies of the fingerprint slips with you. This is what my own PGP fingerprint slip looks like. You can also print the key fingerprint on a business card for a more professional look. Using PGP, it is recommend that you sign any important messages you send but you do not have to encrypt the messages you send, especially if some of the people you send messages to (like family and friends) do not yet have the PGP software to decrypt them. If using the Thunderbird (Icedove) email client from Mozilla, you can easily send signed messages and validate the messages you receive using the Enigmail plugin. Get your PGP key signed Once you have a PGP key, you will need to find other developers to sign it. For people I mentor personally in GSoC, I'm keen to see that you try and find another Debian Developer in your area to sign your key as early as possible. Free software events Try and find all the free software events in your area in the months between now and the end of the next Google Summer of Code season. Aim to attend at least two of them before GSoC. Look closely at the schedules and find out about the individual speakers, the companies and the free software projects that are participating. For events that span more than one day, find out about the dinners, pub nights and other social parts of the event. Try and identify people who will attend the event who have been GSoC mentors or who intend to be. Contact them before the event, if you are keen to work on something in their domain they may be able to make time to discuss it with you in person. Take your PGP fingerprint slips. Even if you don't participate in a formal key-signing party at the event, you will still find some developers to sign your PGP key individually. You must take a photo ID document (such as your passport) for the other developer to check the name on your fingerprint but you do not give them a copy of the ID document. Events come in all shapes and sizes. FOSDEM is an example of one of the bigger events in Europe, linux.conf.au is a similarly large event in Australia. There are many, many more local events such as the Debian UK mini-DebConf in Cambridge, November 2015. Many events are either free or free for students but please check carefully if there is a requirement to register before attending. On your blog, discuss which events you are attending and which sessions interest you. Write a blog during or after the event too, including photos. Quantcast generously hosted the Ganglia community meeting in San Francisco, October 2013. We had a wild time in their offices with mini-scooters, burgers, beers and the Ganglia book. That's me on the pink mini-scooter and Bernard Li, one of the other Ganglia GSoC 2014 admins is on the right. Install Linux GSoC is fundamentally about free software. Linux is to free software what a tree is to the forest. Using Linux every day on your personal computer dramatically increases your ability to interact with the free software community and increases the number of potential GSoC projects that you can participate in. This is not to say that people using Mac OS or Windows are unwelcome. I have worked with some great developers who were not Linux users. Linux gives you an edge though and the best time to gain that edge is now, while you are a student and well before you apply for GSoC. If you must run Windows for some applications used in your course, it will run just fine in a virtual machine using Virtual Box, a free software solution for desktop virtualization. Use Linux as the primary operating system. Here are links to download ISO DVD (and CD) images for some of the main Linux distributions: If you are nervous about getting started with Linux, install it on a spare PC or in a virtual machine before you install it on your main PC or laptop. Linux is much less demanding on the hardware than Windows so you can easily run it on a machine that is 5-10 years old. Having just 4GB of RAM and 20GB of hard disk is usually more than enough for a basic graphical desktop environment although having better hardware makes it faster. Your experiences installing and running Linux, especially if it requires some special effort to make it work with some of your hardware, make interesting topics for your blog. Decide which technologies you know best Personally, I have mentored students working with C, C++, Java, Python and JavaScript/HTML5. In a GSoC program, you will typically do most of your work in just one of these languages. From the outset, decide which language you will focus on and do everything you can to improve your competence with that language. For example, if you have already used Java in most of your course, plan on using Java in GSoC and make sure you read Effective Java (2nd Edition) by Joshua Bloch. Decide which themes appeal to you Find a topic that has long-term appeal for you. Maybe the topic relates to your course or maybe you already know what type of company you would like to work in. Here is a list of some topics and some of the relevant software projects:
  • System administration, servers and networking: consider projects involving monitoring, automation, packaging. Ganglia is a great community to get involved with and you will encounter the Ganglia software in many large companies and academic/research networks. Contributing to a Linux distribution like Debian or Fedora packaging is another great way to get into system administration.
  • Desktop and user interface: consider projects involving window managers and desktop tools or adding to the user interface of just about any other software.
  • Big data and data science: this can apply to just about any other theme. For example, data science techniques are frequently used now to improve system administration.
  • Business and accounting: consider accounting, CRM and ERP software.
  • Finance and trading: consider projects like R, market data software like OpenMAMA and connectivity software (Apache Camel)
  • Real-time communication (RTC), VoIP, webcam and chat: look at the JSCommunicator or the Jitsi project
  • Web (JavaScript, HTML5): look at the JSCommunicator
Before the GSoC application process begins, you should aim to learn as much as possible about the theme you prefer and also gain practical experience using the software relating to that theme. For example, if you are attracted to the business and accounting theme, install the PostBooks suite and get to know it. Maybe you know somebody who runs a small business: help them to upgrade to PostBooks and use it to prepare some reports. Make something Make some small project, less than two week's work, to demonstrate your skills. It is important to make something that somebody will use for a practical purpose, this will help you gain experience communicating with other users through Github. For an example, see the servlet Juliana Louback created for fixing phone numbers in December 2013. It has since been used as part of the Lumicall web site and Juliana was selected for a GSoC 2014 project with Debian. There is no better way to demonstrate to a prospective mentor that you are ready for GSoC than by completing and publishing some small project like this yourself. If you don't have any immediate project ideas, many developers will also be able to give you tips on small projects like this that you can attempt, just come and ask us on one of the mailing lists. Ideally, the project will be something that you would use anyway even if you do not end up participating in GSoC. Such projects are the most motivating and rewarding and usually end up becoming an example of your best work. To continue the example of somebody with a preference for business and accounting software, a small project you might create is a plugin or extension for PostBooks. Getting to know prospective mentors Many web sites provide useful information about the developers who contribute to free software projects. Some of these developers may be willing to be a GSoC mentor. For example, look through some of the following: Getting on the mentor's shortlist Once you have identified projects that are interesting to you and developers who work on those projects, it is important to get yourself on the developer's shortlist. Basically, the shortlist is a list of all students who the developer believes can complete the project. If I feel that a student is unlikely to complete a project or if I don't have enough information to judge a student's probability of success, that student will not be on my shortlist. If I don't have any student on my shortlist, then a project will not go ahead at all. If there are multiple students on the shortlist, then I will be looking more closely at each of them to try and work out who is the best match. One way to get a developer's attention is to look at bug reports they have created. Github makes it easy to see complaints or bug reports they have made about their own projects or other projects they depend on. Another way to do this is to search through their code for strings like FIXME and TODO. Projects with standalone bug trackers like the Debian bug tracker also provide an easy way to search for bug reports that a specific person has created or commented on. Once you find some relevant bug reports, email the developer. Ask if anybody else is working on those issues. Try and start with an issue that is particularly easy and where the solution is interesting for you. This will help you learn to compile and test the program before you try to fix any more complicated bugs. It may even be something you can work on as part of your academic program. Find successful projects from the previous year Contact organizations and ask them which GSoC projects were most successful. In many organizations, you can find the past students' project plans and their final reports published on the web. Read through the plans submitted by the students who were chosen. Then read through the final reports by the same students and see how they compare to the original plans. Start building your project proposal now Don't wait for the application period to begin. Start writing a project proposal now. When writing a proposal, it is important to include several things:
  • Think big: what is the goal at the end of the project? Does your work help the greater good in some way, such as increasing the market share of Linux on the desktop?
  • Details: what are specific challenges? What tools will you use?
  • Time management: what will you do each week? Are there weeks where you will not work on GSoC due to vacation or other events? These things are permitted but they must be in your plan if you know them in advance. If an accident or death in the family cut a week out of your GSoC project, which work would you skip and would your project still be useful without that? Having two weeks of flexible time in your plan makes it more resilient against interruptions.
  • Communication: are you on mailing lists, IRC and XMPP chat? Will you make a weekly report on your blog?
  • Users: who will benefit from your work?
  • Testing: who will test and validate your work throughout the project? Ideally, this should involve more than just the mentor.
If your project plan is good enough, could you put it on Kickstarter or another crowdfunding site? This is a good test of whether or not a project is going to be supported by a GSoC mentor. Learn about packaging and distributing software Packaging is a vital part of the free software lifecycle. It is very easy to upload a project to Github but it takes more effort to have it become an official package in systems like Debian, Fedora and Ubuntu. Packaging and the communities around Linux distributions help you reach out to users of your software and get valuable feedback and new contributors. This boosts the impact of your work. To start with, you may want to help the maintainer of an existing package. Debian packaging teams are existing communities that work in a team and welcome new contributors. The Debian Mentors initiative is another great starting place. In the Fedora world, the place to start may be in one of the Special Interest Groups (SIGs). Think from the mentor's perspective After the application deadline, mentors have just 2 or 3 weeks to choose the students. This is actually not a lot of time to be certain if a particular student is capable of completing a project. If the student has a published history of free software activity, the mentor feels a lot more confident about choosing the student. Some mentors have more than one good student while other mentors receive no applications from capable students. In this situation, it is very common for mentors to send each other details of students who may be suitable. Once again, if a student has a good Github profile and a blog, it is much easier for mentors to try and match that student with another project. GSoC logo generic Conclusion Getting into the world of software engineering is much like joining any other profession or even joining a new hobby or sporting activity. If you run, you probably have various types of shoe and a running watch and you may even spend a couple of nights at the track each week. If you enjoy playing a musical instrument, you probably have a collection of sheet music, accessories for your instrument and you may even aspire to build a recording studio in your garage (or you probably know somebody else who already did that). The things listed on this page will not just help you walk the walk and talk the talk of a software developer, they will put you on a track to being one of the leaders. If you look over the profiles of other software developers on the Internet, you will find they are doing most of the things on this page already. Even if you are not selected for GSoC at all or decide not to apply, working through the steps on this page will help you clarify your own ideas about your career and help you make new friends in the software engineering community.

30 September 2015

Chris Lamb: Free software activities in September 2015

Inspired by Rapha l Hertzog, here is a monthly update covering a large part of what I have been doing in the free software world:
Debian The Reproducible Builds project was also covered in depth on LWN as well as in Lunar's weekly reports (#18, #19, #20, #21, #22).
Uploads
  • redis A new upstream release, as well as overhauling the systemd configuration, maintaining feature parity with sysvinit and adding various security hardening features.
  • python-redis Attempting to get its Debian Continuous Integration tests to pass successfully.
  • libfiu Ensuring we do not FTBFS under exotic locales.
  • gunicorn Dropping a dependency on python-tox now that tests are disabled.



RC bugs


I also filed FTBFS bugs against actdiag, actdiag, bangarang, bmon, bppphyview, cervisia, choqok, cinnamon-control-center, clasp, composer, cpl-plugin-naco, dirspec, django-countries, dmapi, dolphin-plugins, dulwich, elki, eqonomize, eztrace, fontmatrix, freedink, galera-3, golang-git2go, golang-github-golang-leveldb, gopher, gst-plugins-bad0.10, jbofihe, k3b, kalgebra, kbibtex, kde-baseapps, kde-dev-utils, kdesdk-kioslaves, kdesvn, kdevelop-php-docs, kdewebdev, kftpgrabber, kile, kmess, kmix, kmldonkey, knights, konsole4, kpartsplugin, kplayer, kraft, krecipes, krusader, ktp-auth-handler, ktp-common-internals, ktp-text-ui, libdevice-cdio-perl, libdr-tarantool-perl, libevent-rpc-perl, libmime-util-java, libmoosex-app-cmd-perl, libmoosex-app-cmd-perl, librdkafka, libxml-easyobj-perl, maven-dependency-plugin, mmtk, murano-dashboard, node-expat, node-iconv, node-raw-body, node-srs, node-websocket, ocaml-estring, ocaml-estring, oce, odb, oslo-config, oslo.messaging, ovirt-guest-agent, packagesearch, php-svn, php5-midgard2, phpunit-story, pike8.0, plasma-widget-adjustableclock, plowshare4, procps, pygpgme, pylibmc, pyroma, python-admesh, python-bleach, python-dmidecode, python-libdiscid, python-mne, python-mne, python-nmap, python-nmap, python-oslo.middleware, python-riemann-client, python-traceback2, qdjango, qsapecng, ruby-em-synchrony, ruby-ffi-rzmq, ruby-nokogiri, ruby-opengraph-parser, ruby-thread-safe, shortuuid, skrooge, smb4k, snp-sites, soprano, stopmotion, subtitlecomposer, svgpart, thin-provisioning-tools, umbrello, validator.js, vdr-plugin-prefermenu, vdr-plugin-vnsiserver, vdr-plugin-weather, webkitkde, xbmc-pvr-addons, xfsdump & zanshin.

Dominique Dumont: Using custom cache object with AngularJS $http

Hello At work, I ve been bitten by the way AngularJS handles cache by default when using $https service. This post will show a simple way to improve cache handling with $http service. The service I m working on must perform the followings tasks: At first, I ve naively used $http.get cache parameter to enable or disable caching using a sequence like:
  1. $http.get(url, cache: true )
  2. $http.post(url)
  3. $http.get(url, cache: false )
  4. $http.get(url, cache: true )
Let s say the calls above use the following data:
  1. $http.get(url, cache: true ) returns foo
  2. $http.post(url) stores bar
  3. $http.get(url, cache: false ) returns bar
I expected the next call $http.get(url, cache: false ) to return bar . But no, I got foo , i.e. the obsolete data. Turns out that cache object is completely left alone when cache: false is passed to $http.get. ok. Fair enough. But this means that the value of the cache parameter should not change for a given URL. The default cache provided by $https cannot be cleared. (Well, actually, you can clear the cache under AngularJS s hood, but that will probably not improve the readability of your code). The naive approach does not work. Let s try another solution by using a custom cache object as suggested by AngularJS doc. This cache object should be created by $cacheFactory service. This cache object can then be passed to $http.get to be used as cache. When needed, the cache can be cleared. In the example above, the cache must be cleared after saving some data to the remote service. There s 2 possibilities to clear a cache: So, we have to use the first solution and create a cache object for each API entry point:
angular.module('app').factory('myService', function ($http, $cacheFactory)  
  var myFooUrl = '/foo-rest-service';
  
  // create cache object. The cache id must be unique
  var fooCache = $cacheFactory('myService.foo'); 
  function getFooData ()  
    return $http.get( myFooUrl,   cache: fooCache  );
   ;
 
  function saveFooData(data)  
    return $http.post( myFooUrl,   cache: fooCache  ).then(function()  
      myCache.removeAll() ;
     );
   
 );
The code above ensures that: This simple approach has the following limitations: If you need more a more advance cache mechanism, you may want to check jmdobry s angular cache project All the best

29 September 2015

Norbert Preining: Multi-boot stick update: TAILS 1.6, SysresCD 4.6.0, GParted 0.23, Debian 8.2

Updates for my multi-boot/multi-purpose USB stick: All components have been updated to the latest versions and I have confirmed that all of them still boot properly although changes in the grub.cfg file are necessary. So going through these explanations one will end up with a usable USB stick that can boot you into TAILS, System Rescue CD, GNU Parted Live CD, GRML, and also can boot into an installation of Debian 8.2 Jessie installation. All this while still being able to use the USB stick as normal media. multi-usb-disk Since there have been a lot of updates, and also changes in the setup and grub config file, I include the full procedure here, that is, merging and updating these previous posts: USB stick with Tails and SystemRescueCD, Tails 1.2.1, Debian jessie installer, System Rescue CD on USB, USB stick update: TAILS 1.4, GParted 0.22, SysResCD 4.5.2, Debian Jessie, and USB stick update: Debian is back, plus GRML. Let us repeat some things from the original post concerning the wishlist and the main players: I have a long wishlist of items a boot stick should fulfill Requirements A USB stick, the iso images of TAILS 1.6, SystemRescueCD 4.6.0, GParted Lice CD 0.23.0, GRML 2014.11, and some tool to access iso images, for example ISOmaster (often available from your friendly Linux distribution). I assume that you have already an USB stick prepared as described previously. If this is not the case, please go there and follow the section on preparing your usb stick. Three types of boot options We will employ three different approaches to boot special systems: the one is directly from an iso image (easiest, simple to update), the other via extraction of the necessary kernels and images (bit painful, needs some handwork), and the last one is a mixture necessary to get Debian booting (most painful, needs additional downloads and handwork). At the moment we have the following status with respect to boot methods: Booting from ISO image Grub has gained quite some time ago the ability to boot directly from an ISO image. In this case the iso image is mounted via loopback, and the kernel and initrd are specified relatively to the iso image root. This system makes it extremely easy to update the respective boot option: just drop the new iso image onto the USB stick, and update the isofile setting. One could even use some -latest method, but I prefer to keep the exact name. For both SystemRescueCD, GNOME Partition Live CD, and GRML, just drop the iso files into /boot/iso/, in my case /boot/iso/systemrescuecd-x86-4.6.0.iso and /boot/iso/gparted-live-0.23.0-1-i586.iso. After that, entries like the following have to be added to grub.cfg. For the full list see grub.cfg:
submenu "System Rescue CD 4.6.0 (via ISO) ---> "  
  set isofile="/boot/iso/systemrescuecd-x86-4.6.0.iso"
  menuentry "SystemRescueCd (64bit, default boot options)"  
        set gfxpayload=keep
        loopback loop (hd0,1)$isofile
        linux   (loop)/isolinux/rescue64 isoloop=$isofile
        initrd  (loop)/isolinux/initram.igz
   
  ...
 
 
submenu "GNU/Gnome Parted Live CD 0.23.0 (via ISO) ---> "  
  set isofile="/boot/iso/gparted-live-0.23.0-1-i586.iso"
  menuentry "GParted Live (Default settings)" 
    loopback loop (hd0,1)$isofile
    linux (loop)/live/vmlinuz boot=live union=overlay username=user config components quiet noswap noeject  ip= net.ifnames=0 nosplash findiso=$isofile
    initrd (loop)/live/initrd.img
   
  ...
 
 
submenu "GRML 2014.11 ---> "  
  menuentry "Grml Rescue System 64bit"  
        iso_path="/boot/iso/grml64-full_2014.11.iso"
        export iso_path
        loopback loop (hd0,1)$iso_path
        set root=(loop)
        kernelopts=" ssh=foobarbaz toram  "
        export kernelopts
        configfile /boot/grub/loopback.cfg
    
 
Note the added isoloop=$isofile and findiso=$isofile that helps the installer find the iso images. Booting via extraction of kernels and images This is a bit more tedious, but still not too bad. Installation of TAILS files Assuming you have access to the files on the TAILS CD via the directory ~/tails, execute the following commands:
mkdir -p /usbstick/boot/tails
cp -a ~/tails/live/* /usbstick/boot/tails/
The grub.cfg entries look now similar to the following:
submenu "TAILS Environment 1.6 ---> "  
  menuentry "Tails64 Live System"  
        linux   /boot/tails/vmlinuz2 boot=live live-media-path=/boot/tails config live-media=removable nopersistent noprompt timezone=Etc/UTC block.events_dfl_poll_msecs=1000 splash noautologin module=Tails
        initrd  /boot/tails/initrd2.img
 
   
  ...
 
The important part here is the live-media-path=/boot/tails, otherwise TAILS will not find the correct files for booting. The rest of the information was extracted from the boot setup of TAILS itself. Mixture of iso image and extraction Debian jessie As mentioned in the previous post, booting Debian/Jessie installation images via any method laid out above didn t work, since the iso images is never found. It turned out that the current installer iso images do not contain the iso-scan package, which is responsible for searching and loading of iso images. But with a small trick one can overcome this: One needs to replace the initrd that is on the ISO image with one that contains the iso-scan package. And we do not need to create these initrd by ourselves, but simply use the ones from hd-media type installer. I downloaded the following four gzipped initrds from one of the Debian mirrors: i386/initrd text mode, i386/initrd gui mode, amd64/initrd text mode, amd64/initrd gui mode, and put them into the USB stick s boot/debian/install.386, boot/debian/install.386/gtk, boot/debian/install.amd, boot/debian/install.amd/gtk, respectively. Finally, I added entries similar to this one (rest see the grub.cfg file):
submenu "Debian 8.2 Jessie NetInstall ---> "  
    set isofile="/boot/iso/firmware-8.2.0-amd64-i386-netinst.iso"
    menuentry '64 bit Install'  
        set background_color=black
        loopback loop (hd0,1)$isofile
        linux    (loop)/install.amd/vmlinuz iso-scan/ask_second_pass=true iso-scan/filename=$isofile vga=788 -- quiet 
        initrd   /boot/debian/install.amd/initrd.gz
     
    ...
 
Again an important point, don t forget the two kernel command line options: iso-scan/ask_second_pass=true iso-scan/filename=$isofile, otherwise you probably will have to make the installer scan all disks and drives completely, which might take ages. Current status of USB stick Just to make sure, the usb stick should contain at the current stage the following files:
/boot/
    iso/
        firmware-8.2.0-amd64-i386-netinst.iso
        gparted-live-0.23.0-1-i586.iso
        grml64-full_2014.11.iso
        systemrescuecd-x86-4.6.0.iso
    tails/
        vmlinuz Tails.module initrd.img ....
    grub/
        fonts/
            lots of files
        locale/
            lots of files
        x86_64-efi/
            lots of files
        font.pf2
        grubenv
        grub.cfg            *this file we create in the next step!!*
/EFI
    BOOT/
        BOOTX64.EFI
The Grub config file grub.cfg The final step is to provide a grub config file in /usbstick/boot/grub/grub.cfg. I created one by looking at the isoboot.cfg files both in the SystemRescueCD, TAILS iso images, GParted iso image, and the Debian/Jessie image, and converting them to grub syntax. Excerpts have been shown above in the various sections. I spare you all the details, grab a copy here: grub.cfg Conclusion That s it. Now you can anonymously provide data about your evil government, rescue your friends computer, fix a forgotten Windows password, and above all, install a proper free operating system. If you have any comments, improvements or suggestions, please drop me a comment. I hope this helps a few people getting a decent USB boot stick running. Enjoy.

27 September 2015

Lunar: Reproducible builds: week 22 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Packages fixed The following 22 packages became reproducible due to changes in their build dependencies: breathe, cdi-api, geronimo-jpa-2.0-spec, geronimo-validation-1.0-spec, gradle-propdeps-plugin, jansi, javaparser, libjsr311-api-java, mac-widgets, mockito, mojarra, pastescript, plexus-utils2, powerline, python-psutil, python-sfml, python-tldap, pythondialog, tox, trident, truffle, zookeeper. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: diffoscope development The changes to make diffoscope run under Python 3, along with many small fixes, entered the archive with version 35 on September 21th. Another release was made the very next day fixed two encoding-related issues discovered when running diffoscope on more Debian packages. strip-nondeterminism development Version 0.12.0 now preserves file permissions on modified zip files and dh_strip_nondeterminism has been made compatible with older debhelper. disorderfs development Version 0.3.0 implemented a multi-user mode that was required to build Debian packages using disorderfs. It also added command line options to control the ordering of files in directory (either shuffled or reversed) and another to do arbitrary changes to the reported space used by files on disk. A couple days later, version 0.4.0 was released to support locks, flush, fsync, fsyncdir, read_buf, and write_buf. Almost all known issues have now been fixed. reproducible.debian.net disorderfs is now used during the second build. This makes file ordering issue very easy to identify as such. (h01ger) Work has been done on making the distributed build setup more reliable. (h01ger) Documentation update Matt Kraii fixed the example on how to fix issues related to dates in Sphinx. Recent Sphinx versions should also be compatible with SOURCE_DATE_EPOCH. Package reviews 53 reviews have been removed, 85 added and 13 updated this week. 46 packages failing to build from source has been identified by Chris Lamb, Chris West, and Niko Tyni. Chris Lamb was the lucky reporter of bug #800000 on vdr-plugin-prefermenu. Issues related to disorderfs are being tracked with a new issue.

24 September 2015

Joachim Breitner: The Incredible Proof Machine

In a few weeks, I will have the opportunity to offer a weekend workshop to selected and motivated high school students1 to a topic of my choice. My idea is to tell them something about logic, proofs, and the joy of searching and finding proofs, and the gratification of irrevocable truths. While proving things on paper is already quite nice, it is much more fun to use an interactive theorem prover, such as Isabelle, Coq or Agda: You get immediate feedback, you can experiment and play around if you are stuck, and you get lots of small successes. Someone2 once called interactive theorem proving the worlds most geekiest videogame . Unfortunately, I don t think one can get high school students without any prior knowledge in logic, or programming, or fancy mathematical symbols, to do something meaningful with a system like Isabelle, so I need something that is (much) easier to use. I always had this idea in the back of my head that proving is not so much about writing text (as in normally written proofs) or programs (as in Agda) or labeled statements (as in Hilbert-style proofs), but rather something involving facts that I have proven so far floating around freely, and way to combine these facts to new facts, without the need to name them, or put them in a particular order or sequence. In a way, I m looking for labVIEW wrestled through the Curry-Horward-isomorphism. Something like this:
A proof of implication currying

A proof of implication currying

So I set out, rounded up a few contributors (Thanks!), implemented this, and now I proudly present: The Incredible Proof Machine3 This interactive theorem prover allows you to do perform proofs purely by dragging blocks (representing proof steps) onto the paper and connecting them properly. There is no need to learn syntax, and hence no frustration about getting that wrong. Furthermore, it comes with a number of example tasks to experiment with, so you can simply see it as a challenging computer came and work through them one by one, learning something about the logical connectives and how they work as you go. For the actual workshop, my plan is to let the students first try to solve the tasks of one session on their own, let them draw their own conclusions and come up with an idea of what they just did, and then deliver an explanation of the logical meaning of what they did. The implementation is heavily influenced by Isabelle: The software does not know anything about, say, conjunction ( ) and implication ( ). To the core, everything is but an untyped lambda expression, and when two blocks are connected, it does unification4 of the proposition present on either side. This general framework is then instantiated by specifying the basic rules (or axioms) in a descriptive manner. It is quite feasible to implement other logics or formal systems on top of this as well. Another influence of Isabelle is the non-linear editing: You neither have to create the proof in a particular order nor have to manually manage a proof focus . Instead, you can edit any bit of the proof at any time, and the system checks all of it continuously. As always, I am keen on feedback. Also, if you want to use this for your own teaching or experimenting needs, let me know. We have a mailing list for the project, the code is on GitHub, where you can also file bug reports and feature requests. Contributions are welcome! All aspects of the logic are implemented in Haskell and compiled to JavaScript using GHCJS, the UI is plain hand-written and messy JavaScript code, using JointJS to handle the graph interaction. Obviously, there is still plenty that can be done to improve the machine. In particular, the ability to create your own proof blocks, such as proof by contradiction, prove them to be valid and then use them in further proofs, is currently being worked on. And while the page will store your current progress, including all proofs you create, in your browser, it needs better ways to save, load and share tasks, blocks and proofs. Also, we d like to add some gamification, i.e. achievements ( First proof by contradiction , 50 theorems proven ), statistics, maybe a share theorem on twitter button. As the UI becomes more complicated, I d like to investigating moving more of it into Haskell world and use Functional Reactive Programming, i.e. Ryan Trickle s reflex, to stay sane. Customers who liked The Incredible Proof Machine might also like these artifacts, that I found while looking whether something like this exists:

  1. Students with migration background supported by the START scholarship
  2. Does anyone know the reference?
  3. We almost named it Proofcraft , which would be a name our current Minecraft-wild youth would appreciate, but it is alreay taken by Gerwin Kleins blog. Also, the irony of a theorem prover being in-credible is worth something.
  4. Luckily, two decades ago, Tobias Nipkow published a nice implementation of higher order pattern unification as ML code, which I transliterated to Haskell for this project.

21 September 2015

Lunar: Reproducible builds: week 21 in Stretch cycle

If you see someone on the Debian ReproducibleBuilds project, buy him/her a beer. This work is awesome. What happened in the reproducible builds effort this week: Media coverage Nathan Willis covered our DebConf15 status update in Linux Weekly News. Access to non-LWN subscribers will be given on Thursday 24th. Linux Journal published a more general piece last Tuesday. Unexpected praise for reproducible builds appeared this week in the form of several iOS applications identified as including spyware. The malware was undetected by Apple screening. This actually happened because application developers had simply downloaded a trojaned version of XCode through an unofficial source. While reproducible builds can't really help users of non-free software, this is exactly the kind of attacks that we are trying to prevent in our systems. Toolchain fixes Niko Tyni wrote and uploaded a better patch for the source order problem in libmodule-build-perl. Tristan Seligmann identified how the code generated by python-cffi could be emitted in random order in some cases. Upstream has already fixed the problem. Packages fixed The following 24 packages became reproducible due to changes in their build dependencies: apache-curator, checkbox-ng, gant, gnome-clocks, hawtjni, jackrabbit, jersey1, libjsr305-java, mathjax-docs, mlpy, moap, octave-geometry, paste, pdf.js, pyinotify, pytango, python-asyncssh, python-mock, python-openid, python-repoze.who, shadow, swift, tcpwatch-httpproxy, transfig. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net Tests for Coreboot, OpenWrt, NetBSD, and FreeBSD now runs weekly (instead of monthly). diffoscope development Python 3 offers new features (namely yield from and concurrent.futures) that could help implement parallel processing. The clear separation of bytes and unicode strings is also likely to reduce encoding related issues. Mattia Rizolo thus kicked the effort of porting diffoscope to Python 3. tlsh was the only dependency missing a Python 3 module. This got quickly fixed by a new upload. The rest of the code has been moved to the point where only incompatibilities between Python 2.7 and Pyhon 3.4 had to be changed. The commit stream still require some cleanups but all tests are now passing under Python 3. Documentation update The documentation on how to assemble the weekly reports has been updated. (Lunar) The example on how to use SOURCE_DATE_EPOCH with CMake has been improved. (Ben Beockel, Daniel Kahn Gillmor) The solution for timestamps in man pages generated by Sphinx now uses SOURCE_DATE_EPOCH. (Mattia Rizzolo) Package reviews 45 reviews have been removed, 141 added and 62 updated this week. 67 new FTBFS reports have been filled by Chris Lamb, Niko Tyni, and Lisandro Dami n Nicanor P rez Meyer. New issues added this week: randomness_in_r_rdb_rds_databases, python-ply_compiled_parse_tables. Misc. The prebuilder script is now properly testing umask variations again. Santiago Villa started a discussion on debian-devel on how binNMUs would work for reproducible builds.

20 September 2015

Russ Allbery: Review: Shady Characters

Review: Shady Characters, by Keith Houston
Publisher: W.W. Norton
Copyright: 2013
ISBN: 0-393-34972-1
Format: Trade paperback
Pages: 250
Subtitled The Secret Life of Punctuation, Symbols & Other Typographical Marks, Shady Characters is one of those delightfully quirky books that examines the history of something you would not normally connect to history. It's an attempt to document, and in some cases reconstruct, the history of several specific punctuation marks. If you've read and enjoyed Lynn Truss's Eats, Shoots & Leaves, this is a near-perfect complement, focusing on more obscure marks and adding in a more detailed and coherent history. Houston has some interest in the common and quotidian punctuation marks, with chapters on the hyphen, dash, and quotation mark, but he avoids giving a full-chapter treatment to the most obvious periods and commas. (Although one learns quite a bit about them as asides in other histories.) The rest of the book focuses on the newly-popular (the at symbol), the recognizable but less common (the hash mark, the asterisk and dagger symbols used for footnotes, and the ampersand), and the historical but now obscure (the pilcrow or paragraph mark, and the manicule or pointing finger). He even devotes two chapters to unsuccessful invented punctuation: the interrobang and the long, failed history of irony and sarcasm punctuation. And this is an actual history, not just a collection of anecdotes and curious facts. Houston does a great job of keeping the text light, engaging, and readable, but he's serious about his topic. There are many reproductions of ancient documents showing early forms of punctuation, several serious attempts to adjudicate between competing theories of origin, a few well-marked and tentative forays into guesswork, and an open acknowledgment of several areas where we simply don't know. Along the way, the reader gets a deeper feel for the chaos and arbitrary personal stylistic choices of the days of hand-written manuscripts and the transformation of punctuation by technology. So much of what we now use in punctuation was shaped and narrowed by the practicalities of typesetting. And then modern technology revived some punctuation, such as the now-ubiquitous at sign, or the hash mark that graces every telephone touchpad. I think my favorite part of this history was using punctuation as perspective from which to track the changing relationship between people and written material. It's striking how much early punctuation was primarily used for annotations and commentary on the text, as opposed to making the text itself more readable. Much early punctuation was added after the fact, and then slowly was incorporated into the original manuscript, first via recopying and then via intentional authorial choice. Texts started including their own pre-emptive commentary, and brought the corresponding marks along with the notes. And then printing forced a vast simplification and standardization of punctuation conventions. Full compliments to W.W. Norton, as well, for putting the time and effort into making the paper version of this book a beautiful artifact. Punctuation is displayed in red throughout when it is the focus of the text. Endnotes are properly split from footnotes, with asides and digressions presented properly at the bottom of the page, and numbered endnotes properly reserved solely for citations. There is a comprehensive index, list of figures, and a short and curated list of further readings. I'm curious how much of the typesetting care was preserved in the ebook version, and dubious that all of it would have been possible given the current state of ebook formatting. Typesetting, obscure punctuation, and this sort of popular history of something quotidian are all personal interests of mine, so it's unsurprising I found this book delightful. But it's so well-written and engaging that I can recommend it even to people less interested in those topics. The next time you're in the mood for learning more about a corner of human history that few people consider, I recommend Shady Characters to your attention. Rating: 8 out of 10

Dirk Eddelbuettel: RQuantLib 0.4.1

Right before heading off to last week's excellent EARL 2015 conference in London, a new minor release of RQuantLib was released onto CRAN and into Debian. The changes are detailed below.
Changes in RQuantLib version 0.4.1 (2015-09-11)
  • Changes in RQuantLib code:
    • A simple shiny application is now included in the directory shiny/DiscountCurve/ and accessible via the new demo function ShinyDiscountCurve.
    • The option surface plotting example in arrays.R now checks for rgl by using requireNamespace.
    • The files NAMESPACE and DESCRIPTION have been updated to reflect all the suggestions of R CMD check.
    • The Travis CI tests now use binary Debian packages for all package dependencies making the tests a little faster.
Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the rquantlib-devel mailing list off the R-Forge page. Issue tickets can be filed at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

19 September 2015

Jakub Wilk: Executable URLs

or how to write shell scripts without using whitespace.

Jakub Wilk: Executable URLs

or how to write shell scripts without using whitespace.

17 September 2015

Lunar: A key signing party keyserver as a Tor hidden service

Key signing parties are a pain and hopefully, one day, we will have better ways to authentication keys than reading hexadecimal strings out loud. The Zimmermann Sassaman key-signing protocol makes them much more bearable already by having only one single hexadecimal string read out loud. That string is the cryptographic hash of a document given to every participant listing all participants and their fingerprints. If everyone has the same hash, then we assume that everyone has the same document. Then, participants in turn will confirm that they fully recognize the fingerprint listed in the document. Alexander Wirt wrote a small key server dedicated to receive keys from the participants. There is also a script that will generate the document from the submitted keys and a ready-to-use keyring. The latter can be run automatically using inoticoming when a new key arrives. Finally, it would be nice if participants could confirm that their key has been properly added to the document, e.g. by making the list available on a web server. Setting all this up seemed like a good opportunity to play with Tor hidden services and systemd-nspawn. Here's the setup log with some comments. This was done on a small armhf device with Debian Jessie. Create a new hidden service Edit /etc/tor/torrc on the host to setup the hidden service:
HiddenServiceDir /var/lib/tor/ksp/
HiddenServicePort 80 10.0.0.2:80
HiddenServicePort 11371 10.0.0.2:11371
Run:
host# systemctl reload tor.service
Then, to learn the name of the newly created hidden service name:
host# cat /var/lib/tor/ksp/hostname
ksp123456789abcd.onion
Install the container debootstrap as always:
host# debootstrap --variant=minbase jessie /var/lib/container/ksp
Preliminary container configuration We do the following step simply using chroot as we are going to use the host network configuration for this stage. The container itself will not have access to the Internet.
host# chroot ksp
Let's set the hostname:
ksp-chroot# echo 'ksp' > /etc/hostname
Set up APT:
ksp-chroot# echo 'deb http://httpredir.debian.org/debian jessie main' > /etc/apt/sources.list
ksp-chroot# apt update
We need dbus to get systemd to work well:
ksp-chroot# apt-get install dbus
Make sure that we can resolve our own hostname:
ksp-chroot# apt-get install libnss-myhostname
ksp-chroot# sed -e '/^hosts:/s/files/myhostname \0/' -i /etc/nsswitch.conf
These are dependencies of the keyserver:
ksp-chroot# apt-get install --no-install-recommends libhttp-daemon-perl \
                liblog-loglite-perl libproc-reliable-perl
These ones are needed for the script generating the list:
ksp-chroot# apt-get install bzip2 inoticoming
And we will use the smallest HTTP server available:
ksp-chroot# apt-get install netcat-traditional micro-httpd
Finally, let's unconfigure all DNS resolvers:
ksp-chroot# echo > /etc/resolv.conf
And we are done with the chroot:
ksp-chroot# exit
Let's retrieve the ksp-tools repository now:
host# cd /var/lib/container/srv
host# git clone https://github.com/formorer/ksp-tools
Container setup We will now start the container with a shell to configure it:
host# systemd-nspawn -D ksp --network-veth
Let's ask systemd to configure the network for us:
ksp# systemctl enable systemd-networkd
Let's not forget to set a root password:
ksp# passwd
We add a dedicated user to run the keyserver and the list generation script:
ksp# adduser --system --group --disabled-password --disabled-login --home /var/lib/ksp ksp
Let's configure the keyserver:
ksp# cp /srv/ksp-tools/keyserver.conf /var/lib/ksp/keyserver.conf
Let's edit /var/lib/ksp/keyserver.conf:
homedir = /var/lib/ksp
Now create the GnuPG homedir for the keyserve:
ksp# mkdir /var/lib/ksp/keys
ksp# install -d -o ksp -g ksp -m 0700 /var/lib/ksp/keys/gpg
Copy the template list generator:
ksp# cp -r /srv/ksp-tools/example /var/lib/ksp/keys/ksp123456789abcd_onion
Create the key repository:
ksp# install -d -o ksp -g ksp -m 0700 /var/lib/ksp/keys/ksp123456789abcd_onion/keys
Create a directory accessible to the web server where the participant list will be generated:
ksp# mkdir -p /var/www
ksp# install -d -o ksp -g ksp -m 0755 /var/www/keys
Let's configure the list generation script by editing /var/lib/ksp/keys/ksp123456789abcd_onion/conf/vars:
KS=ksp123456789abcd.onion
export GNUPGHOME=/tmp/ksp-gpg
KSPFILE="/var/www/keys/ksp-event.txt"
Don't forget to adjust the header in /var/lib/ksp/keys/ksp123456789abcd_onion/conf/list-header. Now we create a unit file for the keyserver in /etc/systemd/system/keyserver.service:
[Unit]
Description=Key signing party keyserver
[Service]
Type=simple
Environment="KSP_HOMEDIR=/var/lib/ksp"
ExecStart=/srv/ksp-tools/bin/kspkeyserver.pl --nodaemonize
User=ksp
Group=ksp
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=full
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/ksp
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
Another unit for the list generator as /etc/systemd/system/ksp-list-generator.service:
[Unit]
Description=Key signing party list generator
[Service]
Type=simple
EnvironmentFile=/var/lib/ksp/keys/ksp123456789abcd_onion/conf/vars
ExecStart=/usr/bin/inoticoming --foreground /var/lib/ksp/keys/ksp123456789abcd_onion/keys --chdir /var/lib/ksp/keys/ksp123456789abcd_onion bin/generate-list \;
User=ksp
Group=ksp
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=full
ReadOnlyDirectories=/
ReadWriteDirectories=/var/www/keys
CapabilityBoundingSet=
[Install]
WantedBy=multi-user.target
For the web server, we first configure a socket listening on port 80 in /etc/systemd/system/micro-httpd.socket:
[Unit]
Description=micro-httpd socket
[Socket]
ListenStream=80
Accept=yes
[Install]
WantedBy=sockets.target
And then the web server in /etc/systemd/system/micro-httpd@.service:
[Unit]
Description=micro-httpd server
[Service]
ExecStart=-/usr/sbin/micro-httpd /var/www/ksp
StandardInput=socket
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=full
ReadOnlyDirectories=/
CapabilityBoundingSet=
Let's now ask systemd to start all of these at boot time:
ksp# systemctl daemon-reload
ksp# systemctl enable keyserver.service
ksp# systemctl enable ksp-list-generator.service
ksp# systemctl enable micro-httpd.socket
One way to kill the container is to type Control+] three times. Boot the container Let's get this party started!
host# systemd-nspawn -b -D /var/lib/container/ksp --network-veth
Hopefully, things should work now. Participants to the KSP should then be able to send their key with:
$ torsocks gpg --keyserver ksp123456789abcd.onion --send-key $KEYID
(Sadly, this is broken with GnuPG 2.1 at the moment.) The participant list should be available at http://ksp123456789abcd.onion/ksp-event.txt. Final steps We need to tell systemd to start the container started at boot time:
host# systemctl enable systemd-nspawn@ksp.service
But the default command-line will not use a dedicated network, so we need to override that part of the configuration. First create a directory:
host# mkdir /etc/systemd/system/systemd-nspawn@ksp.service.d
And edit /etc/systemd/system/systemd-nspawn@ksp.service.d/use-network-veth.conf:
[Service]
# The empty line because we want to override all previous ExecStart
# and not add an extra command
ExecStart=
ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --directory=/var/lib/container/%i --network-veth
Let's reload systemd and verify that our snippet is there:
host# systemctl daemon-reload
host# systemctl cat systemd-nspawn@ksp.service
All good? Let's start it:
host# systemctl start systemd-nspawn@ksp.service
One should also add a firewall to disallow any outgoing connections from the ve-ksp interface as an extra protection.

Julien Danjou: My interview in le Journal du Hacker

A few days ago, the French equivalent of Hacker News, called "Le Journal du Hacker", interviewed me about my work on OpenStack, my job at Red Hat and my self-published book The Hacker's Guide to Python. I've spent some time translating it into English so you can read it if you don't understand French! I hope you'll enjoy it.
Hi Julien, and thanks for participating in this interview for the Journal du Hacker. For our readers who don't know you, can you introduce you briefly?
You're welcome! My name is Julien, I'm 31 years old, and I live in Paris. I now have been developing free software for around fifteen years. I had the pleasure to work (among other things) on Debian, Emacs and awesome these last years, and more recently on OpenStack. Since a few months now, I work at Red Hat, as a Principal Software Engineer on OpenStack. I am in charge of doing upstream development for that cloud-computing platform, mainly around the Ceilometer, Aodh and Gnocchi projects.
Being myself a system architect, I follow your work in OpenStack since a while. It's uncommon to have the point of view of someone as implied as you are. Can you give us a summary of the state of the project, and then detail your activities in this project?
The OpenStack project has grown and changed a lot since I started 4 years ago. It started as a few projects providing the basics, like Nova (compute), Swift (object storage), Cinder (volume), Keystone (identity) or even Neutron (network) who are basis for a cloud-computing platform, and finally became composed of a lot more projects. For a while, the inclusion of projects was the subject of a strict review from the technical committee. But since a few months, the rules have been relaxed, and we see a lot more projects connected to cloud-computing joining us. As far as I'm concerned, I've started with a few others people the Ceilometer project in 2012, devoted to handling metrics of OpenStack platforms. Our goal is to be able to collect all the metrics and record them to analyze them later. We also have a module providing the ability to trigger actions on threshold crossing (alarm). The project grew in a monolithic way, and in a linear way for the number of contributors, during the first two years. I was the PTL (Project Technical Leader) for a year. This leader position asks for a lot of time for bureaucratic things and people management, so I decided to leave my spot in order to be able to spend more time solving the technical challenges that Ceilometer offered. I've started the Gnocchi project in 2014. The first stable version (1.0.0) was released a few months ago. It's a timeseries database offering a REST API and a strong ability to scale. It was a necessary development to solve the problems tied to the large amount of metrics created by a cloud-computing platform, where tens of thousands of virtual machines have to be metered as often as possible. This project works as a standalone deployment or with the rest of OpenStack. More recently, I've started Aodh, the result of moving out the code and features of Ceilometer related to threshold action triggering (alarming). That's the logical suite to what we started with Gnocchi. It means Ceilometer is to be split into independent modules that can work together with or without OpenStack. It seems to me that the features provided by Ceilometer, Aodh and Gnocchi can also be interesting for operators running more classical infrastructures. That's why I've pushed the projects into that direction, and also to have a more service-oriented architecture (SOA)
I'd like to stop for a moment on Ceilometer. I think that this solution was very expected, especially by the cloud-computing providers using OpenStack for billing resources sold to their customers. I remember reading a blog post where you were talking about the high-speed construction of this brick, and features that were not supposed to be there. Nowadays, with Gnocchi and Aodh, what is the quality of the brick Ceilometer and the programs it relies on?
Indeed, one of the first use-case for Ceilometer was tied to the ability to get metrics to feed a billing tool. That's now a reached goal since we have billing tools for OpenStack using Ceilometer, such as CloudKitty. However, other use-cases appeared rapidly, such as the ability to trigger alarms. This feature was necessary, for example, to implement the auto-scaling feature that Heat needed. At the time, for technical and political reasons, it was not possible to implement this feature in a new project, and the functionality ended up in Ceilometer, since it was using the metrics collected and stored by Ceilometer itself. Though, like I said, this feature is now in its own project, Aodh. The alarm feature is used since a few cycles in production, and the Aodh project brings new features on the table. It allows to trigger threshold actions and is one of the few solutions able to work at high scale with several thousands of alarms. It's impossible to make Nagios run with millions of instances to fetch metrics and triggers alarms. Ceilometer and Aodh can do that easily on a few tens of nodes automatically. On the other side, Ceilometer has been for a long time painted as slow and complicated to use, because its metrics storage system was by default using MongoDB. Clearly, the data structure model picked was not optimal for what the users were doing with the data. That's why I started Gnocchi last year, which is perfectly designed for this use case. It allows linear access time to metrics (O(1) complexity) and fast access time to the resources data via an index. Today, with 3 projects having their own perimeter of features defined and which can work together Ceilometer, Aodh and Gnocchi finally erased the biggest problems and defects of the initial project.
To end with OpenStack, one last question. You're a Python developer for a long time and a fervent user of software testing and test-driven development. Several of your blogs posts point how important their usage are. Can you tell us more about the usage of tests in OpenStack, and the test prerequisites to contribute to OpenStack?
I don't know any project that is as tested on every layer as OpenStack is. At the start of the project, there was a vague test coverage, made of a few unit tests. For each release, a bunch of new features were provided, and you had to keep your fingers crossed to have them working. That's already almost unacceptable. But the big issue was that there was also a lot of regressions, et things that were working were not anymore. It was often corner cases that developers forgot about that stopped working. Then the project decided to change its policy and started to refuse all patches new features or bug fix that would not implement a minimal set of unit tests, proving the patch would work. Quickly, regressions were history, and the number of bugs largely reduced months after months. Then came the functional tests, with the Tempest project, which runs a test battery on a complete OpenStack deployment. OpenStack now possesses a complete test infrastructure, with operators hired full-time to maintain them. The developers have to write the test, and the operators maintain an architecture based on Gerrit, Zuul, and Jenkins, which runs the test battery of each project for each patch sent. Indeed, for each version of a patch sent, a full OpenStack is deployed into a virtual machine, and a battery of thousands of unit and functional tests is run to check that no regressions are possible. To contribute to OpenStack, you need to know how to write a unit test the policy on functional tests is laxer. The tools used are standard Python tools, unittest for the framework and tox to run a virtual environment (venv) and run them. It's also possible to use DevStack to deploy an OpenStack platform on a virtual machine and run functional tests. However, since the project infrastructure also do that when a patch is submitted, it's not mandatory to do that yourself locally.
The tools and tests you write for OpenStack are written in Python, a language which is very popular today. You seem to like it more than you have to, since you wrote a book about it, The Hacker's Guide to Python, that I really enjoyed. Can you explain what brought you to Python, the main strong points you attribute to this language (quickly) and how you went from developer to author?
I stumbled upon Python by chance, around 2005. I don't remember how I hear about it, but I bought a first book to discover it and started toying with that language. At that time, I didn't find any project to contribute to or to start. My first project with Python was rebuildd for Debian in 2007, a bit later. I like Python for its simplicity, its object orientation rather clean, its easiness to be deployed and its rich open source ecosystem. Once you get the basics, it's very easy to evolve and to use it for anything, because the ecosystem makes it easy to find libraries to solve any kind of problem. I became an author by chance, writing blog posts from time to time about Python. I finally realized that after a few years studying Python internals (CPython), I learned a lot of things. While writing a post about the differences between method types in Python which is still one of the most read post on my blog I realized that a lot of things that seemed obvious to me where not for other developers. I wrote that initial post after thousands of hours spent doing code reviews on OpenStack. I, therefore, decided to note all the developers pain points and to write a book about that. A compilation of what years of experience taught me and taught to the other developers I decided to interview in the book.
I've been very interested by the publication of your book, for the subject itself, but also the process you chose. You self-published the book, which seems very relevant nowadays. Is that a choice from the start? Did you look for an editor? Can you tell use more about that?
I've been lucky to find out about others self-published authors, such as Nathan Barry who even wrote a book on that subject, called Authority. That's what convinced me it was possible and gave me hints for that project. I've started to write in August 2013, and I ran the firs interviews with other developers at that time. I started to write the table of contents and then filled the pages with what I knew and what I wanted to share. I manage to finish the book around January 2014. The proof-reading took more time than I expected, so the book was only released in March 2014. I wrote a complete report on that on my blog, where I explain the full process in detail, from writing to launching. I did not look for editors though I've been proposed some. The idea of self-publishing really convince me, so I decided to go on my own, and I have no regret. It's true that you have to wear two hats at the same time and handle a lot more things, but with a minimal audience and some help from the Internet, anything's possible! I've been reached by two editors since then, a Chinese and Korean one. I gave them rights to translate and publish the books in their countries, so you can buy the Chinese and Korean version of the first edition of the book out there. Seeing how successful it was, I decided to launch a second edition in Mai 2015, and it's likely that a third edition will be released in 2016.
Nowadays, you work for Red Hat, a company that represents the success of using Free Software as a commercial business model. This company fascinates a lot in our community. What can you say about your employer from your point of view?
It only has been a year since I joined Red Hat (when they bought eNovance), so my experience is quite recent. Though, Red Hat is really a special company on every level. It's hard to see from the outside how open it is, and how it works. It's really close to and it really looks like an open source project. For more details, you should read The Open Organization, a book wrote by Jim Whitehurst (CEO of Red Hat), which he just published. It describes perfectly how Red Hat works. To summarize, meritocracy and the lack of organization in silos is what makes Red Hat a strong organization and puts them as one of the most innovative company. In the end, I'm lucky enough to be autonomous for the project I work on with my team around OpenStack, and I can spend 100% working upstream and enhance the Python ecosystem.

Next.

Previous.