I have previously written about the gitlab CI runners that use docker. Yesterday I made some changes to procps and pushed them to gitlab which would then start the CI. This morning I checked and it said build failed ok, so that s not terribly unusual. The output from the runner was:
gitlab-ci-multi-runner 0.3.3 (dbaf96f)
Using Docker executor with image csmall/testdebian ...
Pulling docker image csmall/testdebian ...
Build failed with Error: image csmall/testdebian: not found
Hmm, I know I have that image, it just must be the runner so, let s see what images I have:
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
Now, I know I have images, I had about 10 or so of them, where did they go? I even looked in the /var/lib/docker directories and can see the json configs, what have you done with my images docker?
Storage Drivers
The first hint I got from stackexchange where someone lost their AUFS images and needed to load the aufs kernel module. Now I know there are two places or methods where docker stores its images. They are called aufs and devicemapper. There is some debate around which one is better and to be honest with what I do I don t much care, I just want it to work.
The version of kernel is significant. It seems the default storage container was AUFS and this requires the aufs.ko kernel module. Linux 4.0 (the version shipped with Debian) does NOT have that module, or at least I couldn t find it.
For new images, this isn t a problem. Docker will just create the new images using devicemapper and everyone is happy. The problem is where you have old aufs images, like me. I want those images.
Rescue the Images
I m not sure if this is the best or most correct way of getting your images, but for me it worked. I got the idea basically from someone who wanted to switch from aufs to devicemapper images for other reasons.
You first need to reboot and select at the grub prompt a 3.x kernel that has aufs support. Then when the system comes up, you should see all your images, like this:
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
csmall/testdebian latest 6979033105a4 5 weeks ago 369.4 MB
gcc5.1 b063030b23b8 5 weeks ago 1.225 GB
gcc 5.1.0 b063030b23b8 5 weeks ago 1.225 GB
gcc latest b063030b23b8 5 weeks ago 1.225 GB
ruby 2.1 236bf35223e7 6 weeks ago 779.8 MB
ruby 2.1.6 236bf35223e7 6 weeks ago 779.8 MB
debian jessie 41b730702607 6 weeks ago 125.1 MB
debian latest 41b730702607 6 weeks ago 125.1 MB
debian 8 41b730702607 6 weeks ago 125.1 MB
debian 8.0 41b730702607 6 weeks ago 125.1 MB
busybox buildroot-2014.02 8c2e06607696 8 weeks ago 2.433 MB
busybox latest 8c2e06607696 8 weeks ago 2.433 MB
What a relief to see this! Work out what images you need to transfer over. In my case it was just the csmall/testdebian one. You need to save it to a tar file.
$ docker save csmall/testdebian > csmall-testdebian.tar.gz
Once you have all your images you want, reboot back to your 4.x kernel. You then need to load each image back into docker.
$ docker load csmall-testdebian.tar.gz
and then test to see its there
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
csmall/testdebian latest 6979033105a4 5 weeks ago 368.3 MB
The size of my image was slightly different. I m not too sure why this is the case but assume it is to do with the storage types. My CI builds now run, they re failing because my test program is trying to link with the library (it shouldn t be) but at least its not a docker problem.
I get quite a bit of recruitment spam, especially via my LinkedIn profile,
but today's Twitter-madness (recruiter scraped my twitter and then contacted
me) really took the biscuit. I include my response (stripped of identifying
marks) for your amusement:
On Tue, Jun 09, 2015 at 10:30:35 +0000, Silly Recruiter wrote:
> I have come across your profile on various social media platforms today and
> after looking through them I feel you are a good fit for a permanent Java
> Developer Role I have available.
Given that you followed me on Twitter I'm assuming you found a tweet or two in
which I mention how much I hate Java?
> I can see you are currently working at Codethink and was wondering if you
> were considering a change of role?
I am not.
> The role on offer is working as a Java Developer for a company based in
> Manchester. You will be maintaining and enhancing the company's core websites
> whilst using the technologies Java, JavaScript, JSP, Struts, Hibernate XML
> and Grails.
This sounds like one of my worst nightmares.
> Are you interested in hearing more about the role? Please feel free to call
> or email me to discuss it further.
Thanks, but no.
> If not, do you know someone that is interested? We offer a 500 referral fee
> for any candidate that is successful.
I wouldn't inflict that kind of Lovecraftian nightmare of a software stack on
anyone I cared about, sorry.
D.
I then decided to take a look back over my Twitter and see if I could find
what might have tripped this. There's some discussion of Minecraft modding but
nothing which would suggest JavaScript, JSP, Struts, Hibernate XML or Grails.
Indeed my most recent tweet regarding Java could hardly be construed as
positive towards it.
Sigh.
What happened about the reproducible
builds effort for this week:
Presentations
On May 26th,Holger Levsen presented reproducible builds in Debian at CCC Berlin for the Datengarten 52. The presentation was in German and the slides in English. Audio and video recordings are available.
Toolchain fixes
Dmitry Shachnev uploaded pyqt5/5.4.1+dfsg-3 which makes pyuic output imports in stable order. Original patch by Reiner Herrmann.
Lunar uploaded mozilla-devscripts/0.41 which uses the UTC timezone when calling zip or unzip.
Patches submitted which did not make their way to the archive yet:
#787327 on vim by Reiner Herrmann: set a constant user and set modified-by option.
#787650 on lush by Daniel Kahn Gillmor: remove __DATE__ and __TIME__ macros from source.
#787669 on cloc by Daniel Kahn Gillmor: use time of latest debian/changelog entry in manpage.
#787675 on ricochet by Daniel Kahn Gillmor: patch configure to allow an external build date and set it to the time of latest debian/changelog entry.
#787804 on clipper by akira: set HTML_TIMESTAMP to NO in Doxygen configuration.
#787829 on colobot by akira: set HTML_TIMESTAMP to NO in Doxygen configuration.
#787865 on fastjet by akira: call Doxygen with HTML_TIMESTAMP set to NO.
#787916 on cal3d by akira: remove $datetime from the file api_footer.html.
#787918 on dime by akira: remove $datetime from the file footer.html.
Daniel Kahn Gilmor also started discussions for emacs24 and the unsorted lists in generated .el files, the recording of a PID number in lush, and the reproducibility of ISO images in grub2.
reproducible.debian.net
Notifications are now sent when the build environment for a package has changed between two builds. This is a first step before automatically building the package once more. (Holger Levsen)
jenkins.debian.net was upgraded to Debian Jessie. (Holger Levsen)
A new variation is now being tested: $PATH. The second build will be done with a /i/capture/the/path added. (Holger Levsen)
Holger Levsen with the help of Alexander Couzens wrote extra job to test the reproducibility of coreboot. Thanks James McCoy for helping with certificate issues.
Mattia Rizollo made some more internal improvements.
strip-nondeterminism development
Andrew Ayer released strip-nondeterminism/0.008-1. This new version fixes the gzip handler so that it now skip adding a predetermined timestamp when there was none.
Holger Levsen sponsored the upload.
Documentation update
The pages about timestamps in manpages generated by Doxygen, GHC .hi files, and Jar files have been updated to reflect their status in upstream.
Markus Koschany documented an easy way to prevent Doxygen to write timestamps in HTML output.
Package reviews
83 obsolete
reviews have
been removed, 71 added and 48 updated this week.
Meetings
A meeting was held on 2015-06-03. Minutes and full logs are available.
It was agreed to hold such a meeting every two weeks for the time being. The time of the next meeting should be announced soon.
I recently obtained the newest Dell's Ubuntu developer offering, XPS 13 (2015, model 9343). I opted in for FullHD non-touch display, mostly because of better battery life, the actual no need for higher resolution, and matte screen which is great outside. Touch would have been "nice-to-have", but in my work I don't really need it.
The other specifications include i7-5600U CPU, 8GB RAM, 256GB SSD [edit: lshw], and of course Ubuntu 14.04 LTS pre-installed as OEM specific installation. It was not possible to directly order it from Dell site, as Finland is reportedly not online market for Dell... The wholesale company however managed to get two models on their lists and so it's now possible to order via retailers. [edit: here are some country specific direct web order links however US, DE, FR, SE, NL]
In this blog post I give a quick look on how I started up using it, and do a few observations on the pre-installed Ubuntu included. I personally was interested in using the pre-installed Ubuntu like a non-Debian/Ubuntu developer would use it, but Dell has also provided instructions for Ubuntu 15.04, Debian 7.0 and Debian 8.0 advanced users among else. Even if not using the pre-installed Ubuntu, the benefit from buying an Ubuntu laptop is obviously smaller cost and on the other hand contributing to free software (by paying for the hardware enablement engineering done by or purchased by Dell). Unboxing
The Black Box. (and white cat)
Opened box.
First time lid opened, no dust here yet!
First time boot up, transitioning from the boot logo to a first time Ubuntu video.
A small clip from the end of the welcoming video.
First time setup. Language, Dell EULA, connecting to WiFi, location, keyboard, user+password.
Creating recovery media. I opted not to do this as I had happened to read that it's highly recommended to install upgrades first, including to this tool.
Finalizing setup.
Ready to log in!
It's alive!
Not so recent 14.04 LTS image... lots of updates.
Problems in the First BatchUnfortunately the first batch of XPS 13:s with Ubuntu are going to ship with some problems. They're easy to fix if you know how to, but it's sad that they're there to begin with in the factory image. There is no knowledge when a fixed batch will start shipping - July maybe?
First of all, installing software upgrades stops. You need to run the following command via Dash Terminal once: sudo apt-get install -f (it suggests upgrading libc-dev-bin, libc6-dbg, libc6-dev and udev). After that you can continue running Software Updater as usual, maybe rebooting in between.
Secondly, the fixed touchpad driver is included but not enabled by default. You need to enable the only non-enabled Additional Driver as seen in the picture below or instructed in Youtube.
Dialog enabling the touchpad driver.
Clarification: you can safely ignore the two paragraphs below, they're just for advanced users like me who want to play with upgraded driver stacks.
Optionally, since I'm interested in the latest graphics drivers especially in case of a brand new hardware like Intel Broadwell, I upgraded my Ubuntu to use the 14.04.2 Hardware Enablement stack (matches 14.10 hardware support): sudo apt install --install-recommends libgles2-mesa-lts-utopic libglapi-mesa-lts-utopic linux-generic-lts-utopic xserver-xorg-lts-utopic libgl1-mesa-dri-lts-utopic libegl1-mesa-drivers-lts-utopic libgl1-mesa-glx-lts-utopic:i386
Even though it's much better than a normal Ubuntu 14.10 would be since many of the Dell fixes continue to be in use, some functionality might become worse compared to the pre-installed stack. The only thing I have noticed though is the internal microphone not working anymore out-of-the-box, requiring a kernel patch as mentioned in Dell's notes. This is not a surprise since the real eventual upstream support involves switching from HDA to I2S and during 14.10 kernel work that was not nearly done. If you're excited about new drivers, I'd recommend waiting until August when the 15.04 based 14.04.3 stack is available (same package names, but 'vivid' instead of 'utopic'). [edit: I couldn't resist myself when I saw linux-generic-lts-vivid (3.19 kernel) is already in the archives. 14.04.2 + that gives me working microphone again!] ConclusionDell XPS 13 Developer Edition with Ubuntu 14.04 LTS is an extremely capable laptop + OS combination nearing perfection, but not quite there because of the software problems in the launch pre-install image. The laptop looks great, feels like a quality product should and is very compact for the screen size.
I've moved over all my work onto it and everything so far is working smoothly in my day-to-day tasks. I'm staying at Ubuntu 14.04 LTS and using my previous LXC configuration to run the latest Ubuntu and Debian development versions. I've also done some interesting changes already like LUKS In-Place Conversion, converting the pre-installed Ubuntu into whole disk encrypted one (not recommended for the faint hearted, GRUB reconfiguration is a bit of a pain).
I look happily forward to working a few productive years with this one!
Nota: Este art culo, en espa ol, aqu .
I m not a gamer, although probably I ve played with machines/computers more than most of the girls of my age. My path has been: handheld machine, Pong and tenis in my uncle s console, MSX (with that one, in addition to playing, I learnt what was an algorithm and a program, and I started to write small programs in Basic, and I copied and ran the source code of small games and programs to make graphics, which I found in MSX magazines). My parents considered that arcade machines in bars were like slot machines so they were banned for us (even pinball, only table soccer was saved from them, and only if my father was playing with us).
In the MSX I played Magical Tree, Galaxians, Arkanoid, Konami s Sports Games, Spanish games from Dinamic, and some arcades like Golden Axe, Xenon, and maybe some more. The next computers (PC AT, later a 286) were not so for gaming; let s say that we played more with the printer (Harvard Graphics, Bannermania ). Later, I was interested in other things more than in computer games, and later there were highschool homework, dBase III, and later the University and programming again and more, and it was the end of gaming, computer was for office and Uni homework.
Later, it came the internet and since then, reading and writing and communicating was more interesting for me than playing.
I was not good at playing, and if you are not good, you play less, and you don t get better, so you begin to find other ways to loose your time, or to win it :)
The new generation
My son is 6 years old now, and I m living with him a second adventure about games. Games have changed a lot, and the family computing try to stay in the libre software side whenever I am the one that can decide, so sometimes some challenges arise.
Android (phone and tablet)
The kid has played games in the phone and tablet with Android since he was a baby. We tried some of the last years popular games. I am not so keen of banning things, but I don t feel comfortable with the popular games for Android (advertisements, nonfree software, addictive elements, massive data recollection and possible surveillance ), so I try to control without being Cruella de Vil . Some techniques I use:
We agree in the amount of time for using the table, setting a tomato to control the time (thanks Pomodoro in F-Droid)
I set the airplane mode and shut down internet everytime that I can. I never register account or login to play (if it s mandatory to login in Google or create an account, sorry but we cannot play, or we create a new empty profile).
I put barriers to installing games (for example, I say that he should uninstall 2 or 3 games first if he wants a new one, or he should explain well why he wants that game and why the ones that were installed became boring suddenly).
I ve never bought games for Android, and I m not thinking about buying them in the future. I prefer to donate for some libre game project.
I try to divert attention to other games (not computer games, usually).
If the equivalent non-computer game exists, we play that one (tic-tac-toe, hungman, battleship )
I seldom play in the phone/tablet, unless we play together.
On the other side, in my phone there is no Google Play, so we have been able to discover the section Games of F-Droid.
We have tried (all of them available in F-Droid, emphasis in the ones that he liked best): 2048, AndroFish, Bomber, Coloring for Kids, Core, Dodge, Falling Blocks, Free Fall, Frozen Bubble, HeriSwap (this one in the tablet), Hex, HyperRogue, Meerkat Challenge, Memory, Pixel Dungeon, Robotfindskitten, Slow it!, Tux Memory, Tux Rider, Vector Pinball.
Playing in my phone with CyanogenMod and having downloaded the games from F-Droid provides a relief similar to the one when playing a non-computer game. At least with the games that I have listed above. Maybe it is because they are simpler games, or they make me remember the ones that I played time ago. But it s also because of the peace of mind of knowing that they are libre software, that have been audited by the F-Droid community, that they don t abuse the user.
The same happens with Debian, what takes me to the next part of this blogpost.
Computer games: Debian
The kid has learnt to play with the tablet and the phone before than with the computer, because our computers have no joysticks nor touchscreens. He learnt to use the touchpad before than the mouse, because it s easier and we have no mouse at home. He learnt to use the mouse at school, where they work with educative games via CD or via web, using Flash :(
So Flash player appears and dissapears from my Debian setup depending on his willness to play with the school games .
Until few time ago, in the computer we played with GCompris, ChildsPlay, and TuxPaint.
When he learnt to use the arrow keys, I installed Hannah and he liked a lot, specially when we learnt to do hannah -l 900 :)
Later, in ClanTV there were advertisements about some online computer games about their favorite series, resulting in that they need Flash or a framework called Unity3D (no, it s not Ubuntu s Unity), and after digging a bit I decided that I was not going to install that #@%! in my Debian, so when he insisted in play those games, I booted the Windows 7 partition in his father s laptop and I installed it there.
Windows is slow and sad in the computer, and those web games with that framework are not very light, so luckily they have not become very interesting.
We have not played in the computer much more, maybe some incursion in Minetest, what takes me to the next section.
(Not without stating my eternal thanks to the Games Team in Debian. I think they do a very important work and I think that next year I ll try to get involved in some way, because I know that the future of our family computer games is tied to libre games in Debian).
PlayStation 3 and Minecraft
Some time ago my husband bought a PlayStation 3 for home. The shop had discounts prices and so. He would play together with the kid an so.
The machine came home with some games for free (included in the price), but most of them were classified for +13 or so, so the only two left were Pro Evolution Soccer, and Minecraft.
I decided not to connect the machine to the network. Maybe we are loosing cool things, but I feel safer like that. So, no ethernet cable plugged, no registration in the Sony shop (or whatever its name is).
The controllers are quite complex for the three of us. They are DualShock don t-know-what, and I think there is something (software) that makes the game adaptative to the person playing, because my husband is worse player after the son plays, if they play in turns and use the same controller.
The kid liked Minecraft. I didn t know anything about that game (well, I knew that there was a libre clone called Minetest), so, for learning the basics I had a look at the wiki and searched videos about how to and we began learning. Now, the kid can read a bit so he needs less help, and he has watched a lot of videos about Minecraft, so he is interested in exploring and building.
I had a look at Minetest, and I installed it in Debian. Having to use the keyboard is a disadvantage, and we didn t know how to dig, so it was not much attractive at first sight. I have looked a bit about how to use the PS3 controller in the computer, via USB, and it seems to work, but I suppose I need to write something to match each controller button with the corresponding key and subsequent action in Minetest. This is work, and I am lazy, and the boy seems not very interested in playing with the computer.
Watching the videos we have infered that it s possible to download saved games and worlds to upload them in the videogame console. We have done some tests. I wanted to upload a saved game about an amusement park, but the file was in a folder of name NPEB01899* and even when the PS3 saw it to copy it from USB to the console, later it didn t appear in the list of saved games (our sved games were in folders named BLES01976). And renaming the folder didn t work, of course. I understood that we had met Sony s restrictions, so I searched for more info. The games are saved using an encryption key and you are not able to use saved games from consoles in other world zone or using a media different than ours (the game can be played using a disc or purchasing it in the digital shop, it seems). Very ugly all of this! I read somewhere that there is certain software (libre software, BTW) that allows to break the encryption and re-encrypt the saved game with the zone and type of media of your console, but it seems the program only works for Windows, and it needs a console ID that we have not, because we didn t register the console in the PlayStation network. All these things look shaky grounds for me, unpleasant stuff, I don t want to spend time on this, maybe I should learn a bit more about Minetest and make it work and interesting and tell Sony go fly a kite. Finally, I found a saved game in the same format as ours (BLES01976), it s not an amusement park but it is a world with interesting places to explore and many things already built, so I ve tried to import it and it worked, so my son will be happy for some time, I suppose.
We have tried Minetest in the tablet too, but the touchscreen is not comfortable for this kind of games.
I feel quite frustrated and angry about this issue of Sony s restrictions on saved games. So I suppose that in the next months I ll try to learn more about Minetest in Debian, game controllers in Debian, and games in Debian in general. So I hope to be able to offer cool stuff to my son, and he becomes more interested in playing in a safe environment which does not abuse the user.
And with this, I finish
Libre games in GNU/Linux, Debian, and info about games in internet
When we have searched info about games in internet, I found that many times you need to go out from the secure environment: webpages with links to downloads that who knows if they contain what they say they contain, advertisements, videoblogs with a language not adequate for kids (or any person that loves their mother language) That s why I believe the path is to go into detail about libre games provided by the distro you use (Debian in my case). Here I bookmark a list of website with info that surely will be useful for me, to read in depth:
Been writing photo viewer in nodejs.
I have a raspberrypi with 3TB HDD to archive camera photo data. It's not easy to retrieve data from it.
I tried running geeqie via ssh and it was slow waiting for it to repaint the display. shotwell seems to be too heavyweight.
I tried running sshfs and running geeqie but it was very slow waiting for IO to complete.
I tried implementing simple web service using nodejs that retrieves a day of photos, and it just crashed browser due to too much data, and the download size was huge.
I tried thumbnailing and added on-demand loading, and now browser was surviving better but raspberrypi was spending seconds converting photos to thumbnails.
I haven't solved my itch yet.
You might have already read this
comment by RMS in the Guardian. That comment and a recent discussion about
the relevance of GPL changes post GPLv2 made me think again about the
battle RMS started to fight. While some think RMS should "retire", at
least I still fail on my personal goal to not depend on non-free software
and services. So for me this battle is far from over, and here is my
personal list of "non-free debt" I've to pay off.
general purpose systems aka your computer
Looking at the increasing list of firmware blobs required to use a GPU, wireless chipsets and
more and more wired NICs, the situation seems to be worse then in the late 90s. Back then
the primary issue was finding supported hardware, but the driver was free. Nowadays even the
open sourced firmware often requires obscure patched compilers to build. If I look at this
stuff I think the OpenBSD project got that right with the more radical position.
Oh and then there is CPU microcode. I'm not yet sure what to think about it, but in the end
it's software and it's not open source. So it's non-free software running on my system.
Maybe my memory is blurred due to the fact, that the seperation of firmware from the Linux kernel,
and proper firmware loading got implemented only years later. I remember the discussion
about the pwc driver and its removal from Linux.
Maybe the situation wasn't better at that time but the firmware was just hidden inside the Linux
driver code?
On my system at work I've to add the Flash plugin to the list due to my latest test with
Prezi which I'll touch later.
I also own a few Humble Indie bundles.
I played parts of Osmos after a recommendation by
Joey Hess, I later finished to play
through Limbo and I got pretty far with
Machinarium
on a Windows system I still had at that time. I also tried a few others but never got
far or soon lost interest.
Another thing I can not really get rid of is unrar because of stuff I need to pull
from xda-developer links just to keep a cell phone running.
Update: Josh Triplett pointed out that there is
unar available in the Debian archive. And indeed that one works on the rar file I just extracted.
Android ecosystem
I will soon get rid of a stock S3 mini and try to replace it with a
moto g
loaded with CyanogenMod.
That leaves me with a working phone with a OS that just works
because of a shitload of non-free blobs. The time and work required
to get there is another story. Among others you need a new bootloader
that requires a newer fastboot compared to what we have in Jessie, and later
you also need the newer adb to be able to sideload the CM image.
There I gave in and just downloaded the pre build SDK from Google. And there
you've another binary I did not even try to build from source. Same for the
CM image itself, though that's not that much different from using a GNU/Linux
distribution if you ignore the trust issues.
It's hard to trust the phone I've build that way, but it's the best I can
get at the moment with at least some bigger chunks of free software inside.
So let's move to the applications on the phone. I do not use GooglePlay,
so I rely on f-droid and freeware I can
download directly from the vendor.
Threema: a bit (a single one) more trustworthy then WhatsApp, they started
around the company of Michael Kasper
Wunderlist: well done shared shopping list, also non-free webservice
Opera: the compression proxy is awesome, also kind of a non-free webservice
"Cloud" services
This category mixes a lot with the stuff listed above, most of them are not only an application,
in fact Threema and Wunderlist are useless without the backend service. And Opera is just
degraded to a browser - and to be replaced with Firefox - if you discount the compression proxy.
The other big addition in this category is Prezi. We tried it out at work
after it got into my focus due to a post
by Dave Aitel. It's kind of the poster child of non-freeness. It requires a non-free, unstable,
insecure and half way deprecated browser plugin to work, you can not download your result in a
useful format, you've to buy storage for your presentation at this one vendor, you've to
pay if you want to keep your presentation private. It's the perfect lockin situation. But still
it's very convenient, prevents a lot of common mistakes you can make when you create a presentation
and they invented a new concept of presenting.
I know about impress.js(hosted on a non-free
platform by the way, but at least you can export it from there) and I also know about
hovercraft. I'm impressed by them, but it's
still not close to the ease of use of Prezi. So here you can also very prominently see the
cost of free and non-free software. Invest the time and write something cool with CSS3 and impress.js
or pay Prezi to just klick yourself through. To add something about the instability - I had to use
a windows laptop for presenting with Prezi because the Flash plugin on Jessie crashed in the presentation
mode, I did not yet check the latest Flash update. I guess that did not make the situation worse, it
already is horrible.
Update: Daniel Kahn Gillmore pointed out that you can combine
inkscape with sozi, though
the Debian package is in desperate need for an active maintainer, see also #692989.
I also use kind of database services like duden.de and dict.cc.
When I was younger you bought such things printed on dead trees but they did not update very well.
Thinking a bit further, a Certification Authority is not only questionable due to the whole
trust issue, they also provide OCSP responder as kind of a web service. And I've already
had the experience what the internet looks like when the OCSP systems of GlobalSign failed.
So there is still a lot to fight for and a lot of "personal non-free debt" to pay off.
Thanks to the reproducible-build team for running a buildd from hell. gregor herrmann
Mattia Rizzolo modified the script added last week to reschedule a package from
Alioth, a reason can now be optionally specified.
Holger Levsen splitted the package
sets page
so each set now has its own page. He also added new sets for Java
packages,
Haskell
packages,
Ruby
packages,
debian-installer
packages,
Go
packages,
and OCaml
packages.
Reiner Herrmann added locales-all to
the set of packages installed in the build environment as its needed to
properly identify variations due to the current locale.
Holger Levsen improved the scheduling so new uploads get tested sooner. He also
changed the .json output that is used by
tracker.debian.org to lists FTBFS issues again
but only for issues unrelated to the toolchain or our test setup. Amongst many
other small fixes and additions, the graph colors should now be
more friendly to red-colorblind people.
The fix for pbuilder given in
#677666 by Tim Landscheidt is now used. This
fixed several FTBFS for OCaml packages.
Work on rebuilding with different CPU has continued, a kvm-on-kvm build host
has been set been set up for this purpose.
debbindiff development
Version 19 of
debbindiff included a fix for a
regression when handling info files.
Version 20 fixes a bug when diffing
files with many differences toward a last line with no newlines. It also now
uses the proper encoding when writing the text output to a
pipe, and detects info files better.
Documentation update
Thanks to Santiago Vila, the unneeded -depth option used with find when
fixing mtimes has been removed from the examples.
Package reviews
113 obsolete
reviews have
been removed this week while 77 has been added.
Christian Hofstaedtler uploaded yard/0.8.7.4-2 which will not write timestamps in the generated documentation. Original patch by Chris Lamb, does not write timestamps in the generated documentation anymore.
Emmanuel Bourg uploaded maven-plugin-tools/3.3-2 which removes the date from the plugin descriptor. Patch by Reiner Herrmann.
Emmanuel Bourg uploaded maven-archiver/2.6-1 which now uses the date set in the DEB_CHANGELOG_DATETIME environment variable for the timestamp in the pom.properties file embedded in the jar files. Original patch by Chris West.
Nicolas Boulenguez uploaded dh-ada-library/6.4 which will warn against non deterministic ALI for sources newer than changelog.
reproducible.debian.net
Alioth now hosts a script that can be used to redo builds and test for a package. This
was preliminary done manually through requests over the IRC channel. This should reduce
the number of interruptions for jenkins' maintainers
The graph of the oldest build per day has been fixed. Maintainance scripts will not
error out when they are no files to remove.
Holger Levsen started work on being able to test variations of CPU features and
build date (as in build in another month of 1984) by using virtual machines.
debbindiff development
Version 18 has been released. It will uses proper comparators for pk3 and info
files. Tar member names are now assumed to be UTF-8 encoded.
The limit for the maximum number of different lines has been removed. Let's see
on reproducible.debian.net how it goes for pathological cases.
It's now possible to specify both --html and --text output. When neither of
them is specified, the default will be to print a text report on the standard
output (thanks to Paul Wise for the suggestion).
Documentation update
Nicolas Boulenguez investigated Ada
libraries.
Package reviews
451 obsolete
reviews have
been removed and 156 added this week.
New identified issues: running kernel version getting captured, random filenames in GHC debug symbols, and timestamps in headers generated by qdbusxml2cpp.
Misc.
Holger Levsen went to re:publica and talked about
reproducible builds to developers and users there.
Holger also had a chance to meet FreeBSD developers and discuss the status of
FreeBSD. Investigations have
started on how it could be made part of our current test
system.
Laurent Guerby gave Lunar access to systems in the GCC Compile
Farm. Hopefully access to these powerful
machines will help to fix packages for GCC, Iceweasel, and similar packages
requiring long build times.
My LTS March and April
In March and April 2015 I sadly didn't get much LTS work done, for a variety of reasons. Most of these reasons make me happy, while at the same time I'm sad I had to reduce my LTS involvement and actually I even failed to allocate those few hours which were assigned to me. So I'll keep this blog post short too, as time is my most precious ressource atm.
In March I only sponsored the axis upload for and wrote DLA-169-1, besides that I spent some hours to implement JSON output for the security-tracker, which was more difficult than anticipated, because a.) different people had different (first) unspoken assumptions what output they wanted and b.) since the security-trackers database schema has grown over years getting the data out in a logically structured fashion ain't as easy as one would imagine...
In April I sponsored the openldap upload and wrote DLA-203-1 and then prepared debian-security-support 2015.04.04~~deb6u1 for squeeze-lts and triaged some of d-s-s's bugs. Adding support for oldoldstable (and thus keeping support for squeeze-lts) to the security-tracker was my joyful contribution for the very joyful Jessie release day.
So in total I only spent 7.5 (paid) hours in these two months on LTS, despite I should have spent 10. The only thing I can say to my defense is that I've spent more time on LTS (supporting both users as well as contributors on the list as well as on IRC) but this time ain't billable. Which I think is right, but it still eats from my "LTS time" and so sometimes I wish I could more easily ignore people and just concentrate on technical fixes...
My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me.
Debian LTS
This month I have been paid to work 26.25 hours on Debian LTS. In that time I did the following:
CVE triage: I pushed 52 commits to the security tracker. I finished a new helper script (bin/lts-cve-triage.py) that builds on the JSON output that Holger implemented recently. It helps to triage more quickly some issues based on the triaging work already done by the Debian Security team.
I filed #783005 to clarify the situation of libhtp and suricata in unstable (discovered this problem while triaging issues affecting those packages).
I reviewed and sponsored DLA-197-1 for Nguyen Cong fixing 5 CVE on libvncserver.
I released DLA-199-1 fixing one CVE on libx11. I also used codesearch.debian.net to identify all packages that had to be rebuilt with the fixed macro and uploaded them all (there was 11 of them).
I sponsored DLA-207-1 for James McCoy fixing 7 CVE on subversion.
I released DLA-214-1 fixing 1 CVE on libxml-libxml-perl.
I released DLA-215-1 fixing 1 CVE on libjson-ruby. This backport was non-trivial but luckily included some non-regression tests.
I filed #783800 about the security-tracker not handling correctly squeeze-lts/non-free.
Now, still related to Debian LTS, but on unpaid hours I did quite a few other things:
I wrote a talk on Debian LTS that I gave during the Mini-DebConf in Lyon. I took quite some time to collect some statistics about the last 10 months of work within the team.
I ensured that the Jessie press release will include a sentence saying that it would be supported for 5 years too.
Other Debian work
Feature request in update-alternatives. After a discussion with Josselin Mouette during the Mini-DebConf in Lyon, I filed #782493 to request the possibility to override at a system-wide level the default priority of alternatives recorded in update-alternatives. This would make it easier for derivatives to make different choices than Debian.
Sponsored a dnsjava NMU. This NMU introcuded a new upstream version which is needed by jitsi. And I also notified the MIA team that the dnsjava maintainers have disappeared.
python-crcmod bug fix and uploads to *-backports. A member of the Google Cloud team wanted this package (with its C extension) to be available to Wheezy users so I NMUed the package in unstable (to fix #782379) and prepared backports for wheezy-backports and jessie-backports (the latter only once the release team rejected a fix in jessie proper, see #782766).
Old and new PTS updates for Jessies s release. I took care to update tracker.debian.org and packages.qa.debian.org to take into account Jessie s release (which, most notably, introduced the oldoldstable suite as the new name for Squeeze until its end of life).
Received thanks with pleasure. This is not something that I did but I enjoyed reading so many spontaneous thanks in response to Guillem s terse and thankless notification of me stepping down from dpkg maintenance. I love the Debian community. Thank you.
Thanks
See you next month for a new summary of my activities.
Debian Jessie has been released on April 25th, 2015. This has opened the
Stretch development cycle. Reactions to the idea of making Debian
build reproducibly have
been pretty enthusiastic. As the pace is now likely to be even faster,
let's see if we can keep everyone up-to-date on the developments.
Before the release of Jessie
The story goes back a long
way
but a formal announcement to the
project
has only been sent in February 2015.
Since then, too much work has happened to make a complete report, but
to give some highlights:
New variations are now tested: umask, kernel version, domain name,
and timezone. We might only be missing CPU type and current date now.
Number of identified issues has grown from about 80 to 123 today.
Lunar did a pretty improvised lightning
talk
during the Mini-DebConf in
Lyon.
This past week
It seems changes were pilling behind the curtains given the amount of activity
that happened in just one week.
Toolchain fixes
Niels Thykier uploaded
debhelper/9.20150501
which includes fixes to dh_makeshlibs
(#774100), dh_icons
(#774102), dh_usrlocal
(#775020). Patches written
by Lunar.
Helmut Grohne uploaded
doxygen/1.8.9.1-3 which
will not generate timestamps in HTML by default. Kudos to
akira for bringing the issue
upstream.
Kenneth J. Pronovici uploaded
epydoc/3.0.1+dfsg-6
adding a --no-include-build-time option.
Patch by Jelmer
Vernooij.
David Pr vot uploaded
php-apigen/2.8.1+dfsg-2
which now has reproducible output.
C dric Boutillier uploaded
ruby-prawn/2.0.1+dfsg-1
which now produce a deterministic output when using gradients.
Patch by Lunar.
Jelmer Vernooij uploaded
samba/2:4.1.17+dfsg-4 which
contains a patch by Matthieu Patou making the output of pidl
(from libparse-pidl-perl) reproducible.
Dmitry Shachnev uploaded
sphinx/1.3.1-1 in experimental
which should produce deterministic output. The original
patch from Chris Lamb has
inspired the upstream
fix.
Improvements to reproducible.debian.net
Mattia Rizzolo has been working on compressing logs using gzip to save
disk space. The web server would uncompress them on-the-fly for clients
which does not accept gzip content.
Mattia Rizzolo worked on a new page listing various
breakage: missing
or bad debbindiff output, missing build logs, unavailable build
dependencies.
Holger Levsen added a new execution environment to run debbindiff using
dependencies from testing. This is required for packages built with
GHC as the compiler only understands interfaces built by the same
version.
debbindiff development
Version 17 has been uploaded to unstable. It now supports comparing
ISO9660 images, dictzip files and should compare identical files much
faster.
Documentation update
Various small updates and fixes to the pages about
PDF produced by LaTeX,
DVI produced by
LaTeX,
static
libraries,
Javadoc,
PE
binaries,
and Epydoc.
Package reviews
Known issues have been tagged when known to be deterministic as some
might unfortunately not show up on every single build.
For example, two new issues have been identified by building with one
timezone in April and one in May. RD
and help2man add current month and
year to the documentation they are producing.
1162 packages have been removed and 774 have been added in
the past week. Most of them are the work of proper automated
investigation done by Chris West.
Summer of code
Finally, we learned that both akira and Dhole were accepted for
this Google Summer of
Code.
Let's welcome them!
They have until May 25th before coding officialy begins. Now is the good time
to help them feel more comfortable by sharing all these little bits of
knowledge on how Debian works.
Gnome Shell, besides providing the main user interface for GNOME 3, is a Javascript shell with bindings to many native interfaces that allow e.g. Window manipulation, graphics rendering and animations, compositing, etc. It also allows developers to write extensions changing Gnome Shell s behavior.
Less known is that it is possible to replace the entire Javascript code base that Gnome Shell uses. It can be useful to hack on Gnome Shell itself (no need to fiddle with system files, or, since 3.12, no need to rebuild libgnome-shell.so), but it can also be used to implement a completely new User Interface in Javascript.
I m starting to experiment with the latter, because I want to try building a window manager that fits my needs, while keeping away the boring details of EWMH, xinerama, and other X11 things. And because it s fun.
But baby steps, first: let s bootstrap a Hello world with Gnome shell.
Create a directory that will hold your code.
In that directory, create a ui subdirectory.
In that ui directory, create a environment.js file, with the following contents:
const Shell = imports.gi.Shell;
function init()
window.global = Shell.Global.get();
In the same directory, create a main.js file, with the following contents:
const St = imports.gi.St;
function start()
let text = new St.Label( text: "Hello, world!" );
global.stage.add_actor(text);
global.stage.show();
You may want to run this in a separate X server (I use Xephyr)
I tested this with Gnome Shell 3.14. Trying various older versions, I got different results for reasons I don t know. 3.4 doesn t display anything unless, paradoxically, global.stage.show() is removed, and 3.8 doesn t display anything no matter what.
I guess the next step is to go through some Clutter tutorials and transpose them to Javascript.
Update: On the other hand, a lot of the window managing is still done by mutter under the hood, which doesn t leave a lot of space for something really different.
Well, I looked at the Bigtop efforts because I needed Hadoop packages. But they are not very
useful. They have lots of issues (including empty packages, naming conflicts etc.).
I filed a few bugs, and I even uploaded my fixes to Github. Some of that went unnoticed,
because Sean Owen of Cloudera decided to
remove all Debian packaging from Spark.
But in the end, even with these fixes, the resulting packages do not live up to
Debian quality standards (not to say, they would outright violate policy).
If you wanted to package Hadoop properly, you should ditch Apache Bigtop,
and instead use the existing best practises for packaging. Using any of the Bigtop work
just makes your job harder, by pulling in additional dependencies like their modified Groovy.
But whatever you do, you will be stuck in .jar dependency hell.
Whatever you look at, it pulls in another batch of dependencies, that all need
to be properly packaged, too. Here is the dependency chain of Hadoop:
So the first step for packaging Hadoop would be to check which of these
dependencies are not yet packaged in Debian... I guess 1/3 is not.
Maybe, we should just rip out some of these dependencies with a cluebat.
For the stupid reason of making a webfrontend (which doesn't provide a lot
of functionality, and I doubt many people use it at all), Hadoop embeds not
just one web server, but two: Jetty and Netty...
Things would also be easier if e.g. S3 support, htrace, the web frontend,
and different data serializations were properly put into modules. Then you
could postpose S3 support, for example.
As I said, the deeper you dig, the crazier it gets.
If the OpenDataPlatform efforts
of Hortonworks, Pivotal and IBM were anything but a marketing gag, they would
try to address these technical issues. Instead, they make things worse by
specifying yet another fatter core, including Ambari, Apaches attempt to
automatically make a mess out of your servers - essentially, they are now adding
the ultimate root shell, for all those cases where unaudited puppet commands
and "curl sudo bash" was not bad enough:
The closer you look, the more you want to rather die than use this.
P.S. I have updated the libtrove3-java package (Java collections for
primitive types; but no longer the fastest such library), so that it is now
in the local maven repository (/usr/share/maven-repo) and that it
can be rebuilt reproducible
(the build user name is no longer in the jar manifest).
Today, trying upload to jessie-backports from a Jessie jail, I got this message from dput-ng:
$ dput netdiscover_0.3beta7~pre-2~bpo8+1_amd64.changes
Uploading netdiscover using ftp to ftp-master (host: ftp.upload.debian.org; directory: /pub/UploadQueue/)
running allowed-distribution: check whether a local profile permits uploads to the target distribution
jessie-backports' not in the codename group
To solve this problem, you can edit the /usr/share/dput-ng/codenames/debian.json file and add jessie-backports here:
Last evening, Codingame held a Programming World Cup titled There is no Spoon . The format is that within four hours, you get to write a program that solves a given task. Submissions are first rated by completeness (there are 13 test inputs that you can check your code again, and further hidden tests that will only be checked after submission) and then by time of submission. You can only submit your code once.
What I like about Codingame is that they support a great number of programming languages in their Web- IDE , including Haskell. I had nothing better to do yesterday, so I joined. I was aiming for a good position in the Haskell-specific ranking.
After nearly two hours my code completed all the visible test cases and I submitted. I figured that this was a reasonable time to do so, as it was half-time and there are supposed to be two challenges. I turned out that the first, quite small task, which felt like a warm-up or qualification puzzle, was the first of those two, and that therefore I was done, and indeed the 5th fastest to complete a 100% solution! With only less than 5 minutes difference to the 3rd, money-winning place if I had known I had such a chance, I had started on time...
Having submitted the highest ranked Haskell code, I will get a T-Shirt. I also defended Haskell s reputation as an efficient programming language, ranked third in the contest, after C++ (rank 1) and Java (rank 2), but before PHP (9), C# (10) and Python (11), listing only those that had a 100% solution.
The task, solving a Bridges puzzle, did not feel like a great fit for Haskell at first. I was juggling Data.Maps around where otherwise I d simple attach attributes to object, and a recursive function simulated nothing but a plain loop. But it played off the moment I had to implement guessing parts of the solution, trying what happens and backtracking when it did not work: With all state in parameters and pure code it was very simple to get a complete solution.
My code is of course not very polished, and having the main loop live in the IO monad just to be able to print diagnostic commands is a bit ugly.
The next, Lord of the Ring-themed world cup will be on June 27th. Maybe we will see more than 18 Haskell entries then?
A few months ago, I wrote a long post about what I called back then the
"Gnocchi experiment".
Time passed and we me and the rest of the Gnocchi team continued to work on
that project, finalizing it.
It's with a great pleasure that we are going to release our first 1.0 version
this month, roughly at the same time that the integrated
OpenStack projects release their Kilo milestone. The
first release candidate numbered 1.0.0rc1
has been released this morning!
The problem to solve
Before I dive into Gnocchi details, it's important to have a good view of what
problems Gnocchi is trying to solve.
Most of the IT infrastructures out there consists of a set of resources. These
resources have properties: some of them are simple attributes whereas others
might be measurable quantities (also known as metrics).
And in this context, the cloud infrastructures make no exception. We talk about
instances, volumes, networks which are all different kind of resources. The
problems that are arising with the cloud trend is the scalability of storing
all this data and being able to request them later, for whatever usage.
What Gnocchi provides is a REST API that allows the user to manipulate
resources (CRUD) and their attributes, while preserving the history of those
resources and their attributes.
Gnocchi is fully documented and the
documentation is available online.
We are the first OpenStack project to require patches to integrate the
documentation. We want to raise the bar, so we took a stand on that. That's
part of our policy, the same way it's part of the OpenStack policy to require
unit tests.
I'm not going to paraphrase the whole Gnocchi documentation, which covers
things like installation (super easy), but I'll guide you through some basics
of the features provided by the REST API. I will show you some example so you
can have a better understanding of what you could leverage using Gnocchi!
Handling metrics
Gnocchi provides a full REST API to manipulate time-series that are called
metrics. You can easily create a metric using a simple HTTP request:
The archive_policy_name parameter defines how the measures that are being
sent are going to be aggregated. You can also define archive policies using the
API and specify what kind of aggregation period and granularity you want. In
that case , the low archive policy keeps 1 hour of data aggregated over 1
second and 1 day of data aggregated to 30 minutes. The functions used for
aggregations are the mathematical functions standard deviation, minimum,
maximum, and even 95th percentile. All of that is obviously customizable and
you can create your own archive policies.
If you don't want to specify the archive policy manually for each metric, you
can also create archive policy rule, that will apply a specific archive
policy based on the metric name, e.g. metrics matching disk.* will be high
resolution metrics so they will use the high archive policy.
It's also worth noting Gnocchi is precise up to the nanosecond and is not tied
to the current time. You can manipulate and inject measures that are years old
and precise to the nanosecond. You can also inject points with old timestamps
(i.e. old compared to the most recent one in the timeseries) with an archive
policy allowing it (see back_window parameter).
It's then possible to send measures to this metric:
These measures are synchronously aggregated and stored into the configured
storage backend. Our most scalable storage drivers for now are either based on
Swift or Ceph which are both
scalable storage objects systems.
It's then possible to retrieve these values:
As older Ceilometer users might notice here, metrics are only storing points
and values, nothing fancy such as metadata anymore.
By default, values eagerly aggregated using mean are returned for all supported
granularities. You can obviously specify a time range or a different
aggregation function using the aggregation, start and stop query
parameter.
Gnocchi also supports doing aggregation across aggregated metrics:
This computes the mean of mean for the metric
65071775-52a8-4d2e-abb3-1377c2fe5c55 and
9ccdd0d6-f56a-4bba-93dc-154980b6e69a starting on 6th October 2014 at 14:34
UTC.
Indexing your resources
Another object and concept that Gnocchi provides is the ability to manipulate
resources. There is a basic type of resource, called generic, which has very
few attributes. You can extend this type to specialize it, and that's what
Gnocchi does by default by providing resource types known for OpenStack such as
instance, volume, network or even image.
The resource is created with the UUID provided by the user. Gnocchi handles the
history of the resource, and that's what the revision_start and
revision_end fields are for. They indicates the lifetime of this revision of
the resource. The ETag and Last-Modified headers are also unique to this
resource revision and can be used in a subsequent request using If-Match or
If-Not-Match header, for example:
Which is useful to synchronize and update any view of the resources you might
have in your application.
You can use the PATCH HTTP method to modify properties of the resource, which
will create a new revision of the resource. The history of the resources are
available via the REST API obviously.
The metrics properties of the resource allow you to link metrics to a
resource. You can link existing metrics or create new ones dynamically:
Haystack, needle? Find!
With such a system, it becomes very easy to index all your resources, meter
them and retrieve this data. What's even more interesting is to query the
system to find and list the resources you are interested in!
You can search for a resource based on any field, for example:
That query will return a list of all resources owned by the user_idbd3a1e52-1c62-44cb-bf04-660bd88cd74d.
You can do fancier queries such as retrieving all the instances started by a
user this month:
And you can even do fancier queries than the fancier ones (still following?).
What if we wanted to retrieve all the instances that were on host foobar the
15th April and who had already 30 minutes of uptime? Let's ask Gnocchi to look
in the history!
I could also mention the fact that you can
search for value in metrics.
One feature that I will very likely include in Gnocchi 1.1 is the ability to
search for resource whose specific metrics matches some value. For example,
having the ability to search for instances whose CPU consumption was over 80%
during a month.
Cherries on the cake
While Gnocchi is well integrated and based on common OpenStack technology,
please do note that it is completely able to function without any other
OpenStack component and is pretty straight-forward to deploy.
Gnocchi also implements a full RBAC system based on the
OpenStack standard oslo.policy
and which allows pretty fine grained control of permissions.
There is also some work ongoing to have HTML rendering when browsing the API
using a Web browser. While still simple, we'd like to have a minimal Web
interface served on top of the API for the same price!
Ceilometer alarm subsystem supports Gnocchi with the Kilo release, meaning you
can use it to trigger actions when a metric value crosses some threshold. And
OpenStack Heat also supports auto-scaling your
instances based on Ceilometer+Gnocchi alarms.
And there are a few more API calls that I didn't talk about here, so don't
hesitate to take a peek at the
full documentation!
Towards Gnocchi 1.1!
Gnocchi is a different beast in the OpenStack community. It is under the
umbrella of the Ceilometer program, but it's one of the first projects that is
not part of the (old) integrated release. Therefore we decided to have a
release schedule not directly linked to the OpenStack and we'll release more
often that the rest of the old OpenStack components probably once every 2
months or the like.
What's coming next is a close integration with Ceilometer (e.g. moving the
dispatcher code from Gnocchi to Ceilometer) and probably more features as we
have more requests from our users. We are also exploring different backends
such as InfluxDB (storage) or MongoDB (indexer).
Stay tuned, and happy hacking!
Note: In this post I mention some problems and ask questions (to myself, like thinking aloud ). The goal is not to get answers to those questions (I suppose that I will find them soon or later in the internet, manuals and so), but to show the kind of problems and questions that arise in my selfhosting adventures, which I suppose are common to other people trying to administer a home server with some web services.
Am I an userop? Well I m something in the middle of (GNU/Linux) user and sysadmin: I have studied computer technical engineering but most of my experience has been in helpdesk, providing support for Windows users. I m running Debian in some LAMP boxes at work (without GUI) since 2008 or so, and in my desktops (with GUI) since 2010. I don t code nor package, but I don t mind trying to read code and understand it (or not). I know a bit of C, a bit of Python, of PHP, and enough Perl to open a Perl file and close it after two minutes, understanding that it s great, but too much for me :) I translate software, so I m not scared to clone a repository, edit files, commit or submit a patch. I m not scared of compiling a program (except if it s an Android app: I try to avoid setting up the development environment just to try some translation that I made but I built my Puma before it was the binary available for download or in F-Droid).
In conclusion, I feel more like a GNU/Linux power user than a sysadmin . Sometimes just a user or even a newbie (for example, I don t know very well the Unix/Linux folder tree where are the wallpapers stored? Does it depend on the desktop that I use?).
Anyway. I won t stop my free software + free networks digital life because I don t know many things. I bought a small server for home last September, and I wanted to try to selfhost some services, for me and for my family. I want to be a home sysadmin or something like that, so I joined the userops mailing list:)
Here you have my experiences on selfhosting/being an userop until now.
Mail
I even didn t try to setup my mail server, because many people say it s a pain (although nice articles were published about how to do it, for example this series in ArsTechnica) and I need a static IP which is 14 /month more to my ISP, and Gandi, the place where I rented my domain name, provides mail, and they use Debian and Roundcube, and sponsor Debian too, so I decided to trust on them.
So this is my strategy now, to try to keep mail under my control:
Trust my domain provider.
Backup my mail and keep local copies, removing sensible stuff from the server.
Use and spread the word about GPG encryption.
Try not to send photos or videos by mail, just send the link to my MediaGoblin instance (see below).
MediaGoblin
I ve setup two MediaGoblin instances (yes, two!). I managed to do it in Debian 7 stable (I think NodeJS npm was not needed then), but soon later I upgraded to Jessie so now it s even better.
I installed Nginx and PostgreSQL via apt, to use them for both instances (and probably some more software later).
One instance is public, and I use a Debian user, a PostgreSQL database, and it s running in http://media.larjona.net
I have requested an SSL cert to Gandi but I still didn t deployed it (lazy LArjona!!).
The other instance is private, for family photos. I didn t know very well how much of my existing setup could reuse and how to keep both instances in case of downtimes or attack I know more or less the concept or chroot but I don t know how to deploy it in my machine. So I decided to use another Debian user, another PostgreSQL database, deploy MediaGoblin in a different folder, and create another virtual server in my Nginx to serve it. I managed to setup that virtual server to http-authenticate and to serve content via a different port, and use a self-signed SSL certificate (it s only for family, so it does not matter). I created another (unprivileged) Debian user with a password for the nginx authentication, and gave my family the URL in the form https://mediaprivate.larjona.net:PortNumber and the user and password (mediaprivate is a string, and PortNumber is a number). I think they don t use the instance too much, but at least I upload photos there from time to time and email the link instead of emailing the photos themselves (they don t use GPG either ).
Upgrades
I upgraded MediaGoblin from 0.7.1 to 0.8.0 successfully, I sent a report about how I did it to the mailing list. First I upgraded the public instance, when I figure out the process, I upgraded the second instance to test my instructions, and then, I sent the report with the instructions to the mailing list.
Static site and LimeSurvey: the power of free software (with instructions)
I wanted to act as a mirror of floss2013.libresoft.es and surveys.libresoft.es since they suffer a downtime and I participated in that project (not in the sysadmin part, but in the research and content creation).
The static site floss2013.libresoft.es offered a zip with the whole website tree (since the website was licensed as AGPL), and I had access to the git repo holding the development copy of the website. So I just cloned the repository and setup another nginx virtual server in my machine, and tuned my DNS zone in Gandi website to serve floss2013.larjona.net from home. 10 minutes setup YAY! #inGitWeTrust #FreeSoftwareFTW :)
For surveys.larjona.net I had to install a LimeSurvey instance. I knew how to do it because we use LimeSurvey at work, but at home I had Nginx instead of Apache, and PostgreSQL instead of MySQL. And no PHP I searched about how to install PHP in Nginx (I can use apt-get, nice!) and how to install LimeSurvey with Nginx and PostgreSQL (I had documentation about that, so I followed, and it worked).
For making available the data (one survey and its results, so people can login as visitor to query and get statistics), I downloaded the LimeSurvey export dataset that we were providing in the static website, followed the replication instructions (hey, I wrote them!), and they worked #oleole! (And here, dear researchers, gets demonstrated that free software and free culture really empower your research and help spreading your results).
Etherpad: not so easy, it seems!
I m trying to install Etherpad-Lite, but I m suffering a bit. I think I did everything ok according to some guides but I get Bad Gateway and these kind of errors when trying to browse with Lynx in the host:
I m not sure if I need to open some port in iptables, my router, or change my nginx configuration because the guides assume you re only serving one website in the port 80 (and I have several of them, now ), or what I ve spent three chunks of time (maybe ~2h each?) on this, in different days, and couldn t figure it out, so I decided to round-robin in my TODO list.
Userops thoughts
Debian brings peace of mind (for me)
On one side, maintaining a Debian box it s quite easy, and the more software that it s packaged, the less time that I spend installing or upgrading. I like being in stable, I m in Jessie now (I migrated when it was frozen), but I ll stay in stable as much as I can.
I like that I can use the software that I installed via apt-get for several services (nginx, PostgreSQL ). About the software that is not packaged (MediaGoblin, LimeSurvey, EtherPad, maybe others later), I wonder how dependencies and updates are handled. And maybe (probably) I have installed some components several times, one for each service (this sounds like a Windows box #grr).
For example MediaGoblin uses PyPump. PyPump 0.5 is packaged in Debian Jessie. MediaGoblin uses PyPump 0.7+. What if PyPump 0.7+ gets, let s say, into Jessie-backports? Can I benefit from that?
I know that MediaGoblin upgrade instructions includes upgrading the dependencies, but what about some security patch in one dependency? Should I upgrade the pip modules periodically? How to know if some upgrade is recommended because patches a vulnerability, or it s just new features (and maybe breaking my setup)?
This kind of things are the peace of mind that Debian packaging brings to me: when some piece of software is packaged, I know maybe I need to care about proper setup and configuration, but later, it s kind-of-easy to maintain (since the Debian maintainers care about the rest). I don t mind about cloning a repo and compiling, I mind about later, or coexistance with other program/services. I trust in the MediaGoblin community and I m an active member (I m not developer, but hang on IRC, follow the mailing list, etc) but for example I don t know anything about the EtherPad project. And I don t feel like joining the community (I m already an active member in Debian, MediaGoblin, F-Droid, Pump.io, translator of LimeSurvey and many other small apps that I use, and in the future will use more services, like OwnCloud, XMPP ), joining the community of each software that I use is becoming not sustainable :s
Free software is more than software
I follow the userop mailing list, and it s becoming very technical. I mostly understand the problems (which are similar to the problems that I face: how to isolate different services, how to easily-configure them, how to make them installable by average user ) But I don t understand most of the solutions proposed, and I think that probably we need technical solutions, but in the meanwhile, some issues can be addressed not with software, but with other means: good documentation, community support, translations, beta-testers
This is my conclusion until now. When a project is well documented, I think I can find my way to selfhost no matter if the software is packaged (or contained ) or not. MediaGoblin, and LimeSurvey, are well documented, and the user support channels are very responsive.
I find lots of instructions that assume that you will use a whole machine for their service (and not for other things). And lots of documentation for the LAMP stack, but not for Nginx + PostgreSQL and Node instead of PHP So, for each particularity of my setup, I search the internet and try to pick good sources to help me do what I wanted to do.
I m kind of privileged
Some elements, not software related, to take into account as pre-requisites for succeed selfhosting services:
I knew what to search.
I knew which sites to visit from the results (arch wiki, debian wiki, stack overflow, etc: some of them were not the Top1 in the results).
I had time to read several sources and make my mind about what to do and how.
I can read, understand, and write in English.
I have no fear about my broken English.
I have no impostor syndrome.
I felt welcome in the FLOSS communities where I hanged out.
These aspects are not present in a lot of people. If I look around to the computer users that I know (mostly Windows+Android, some GNU/Linux users, some Mac OSX users, some iOS users), I find that they search things like X does not work or they cannot write a proper search query in English. Or they trust some random person writing a recipe in their blog, without trying first to understand what the recipe does. Other people just say I m not a professional sysadmin, I ll just do what everybody does (aka use Google services or whatever). What if I try and I don t succeed? . Things like that.
We may need some technical solutions (and hackers are thinking about that, and working on that). But I feel that we need (more) a huge group of beta-testers, dogfooding people, aventurers that try the half-cooked solutions and provide successful and unsuccessful experiences, to guide the research and make software technologies advance. I m not sure if I am an userop, but I feel part of that vanguard force , I want to be part of the future of free software and free networks.
Comments?
You can comment about this post on this pump.io thread. Filed under: My experiences and opinion Tagged: Communities, Contributing to libre software, Debian, Developer motivations, English, free networks, Free Software, Freedom, innovation, MediaGoblin, Moving into free software, Project Management, selfhosting, sysadmin
On Sunday I gave a talk about Debian LTS during the Mini-DebConf in Lyon. Obviously I presented the project and the way it s organized, but I also took the opportunity to compute some statistics.
You can watch the presentation (thanks to the video team!) or have a look at the slides to learn more.
Here are some extracts of the statistics I collected:
The number of the uploads per affiliation (known affiliations are recorded in the LTS/Team wiki page) is displayed on the graph below. None corresponds to packages maintainers taking care of their own packages, Debian Security corresponds to members of the security team who also contributed to LTS, Debian LTS corresponds to individual members of the LTS team without any explicit affiliation. Freexian represents in fact 29 financial sponsors (see detail here).
Top 12 contributors (in number of uploads):
Thorsten Alteholz: 66
Holger Levsen: 27
Rapha l Hertzog: 14
Rapha l Geissert: 13
Thijs Kinkhorst: 8
Kurt Roeck: 7
Christoph Biedl: 7
Nguyen Cong: 6
Ben Hutchings: 6
Michael Vogt: 5
Moritz M hlenhoff: 4
Matt Palmer: 4
The talk also contains explanations about the current funding setup. Hopefully this clears things up for people who were still wondering how the LTS project is working.
Like each month, here comes a report about the work of paid contributors to Debian LTS.
Individual reports
In February, 61 work hours have been equally split among 4 paid contributors. Their reports are available:
The remaining hours of Ben and Holger have been redispatched to other contributors for April (during which Mike Gabriel joins the set of paid contributors). BTW, if you want to join the team of paid contributors, read this and apply!
Evolution of the situation
April has seen no change in terms of sponsored hours but we have two new sponsors in the pipe and May should hopefully have a few more sponsored hours.
For the need of a LTS presentation I gave during the Mini-DebConf Lyon I prepared a small graph showing the evolution of the hours sponsored through Freexian:
The growth is rather slow and it will take years to reach our goal of funding the equivalent a full time position (176 hours per month). Even the intermediary goal of funding the equivalent of a half-time position (88h/month) is more than 6 months away given the current growth rate. But the perspective of Wheezy-LTS should help us to convince more organizations and hopefully we will reach that goal sooner. If you want to sponsor the project, check out this page.
In terms of security updates waiting to be handled, the situation looks similar to last month: the dla-needed.txt file lists 40 packages awaiting an update (exactly like last month), the list of open vulnerabilities in Squeeze shows about 56 affected packages in total (2 less than last month).
Thanks to our sponsors
The new sponsors of the month are in bold (none this month).
I get quite a bit of recruitment spam, especially via my 
What happened about the 
I recently obtained the newest Dell's Ubuntu developer offering, XPS 13 (2015, model 9343). I opted in for FullHD non-touch display, mostly because of better battery life, the actual no need for higher resolution, and matte screen which is great outside. Touch would have been "nice-to-have", but in my work I don't really need it.
Been writing photo viewer in nodejs.
I have a raspberrypi with 3TB HDD to archive camera photo data. It's not easy to retrieve data from it.
I tried running geeqie via ssh and it was slow waiting for it to repaint the display. shotwell seems to be too heavyweight.
I tried running sshfs and running geeqie but it was very slow waiting for IO to complete.
I tried implementing simple web service using nodejs that retrieves a day of photos, and it just crashed browser due to too much data, and the download size was huge.
I tried thumbnailing and added on-demand loading, and now browser was surviving better but raspberrypi was spending seconds converting photos to thumbnails.
I haven't solved my itch yet.
My LTS March and April
In March and April 2015 I sadly didn't get much LTS work done, for a variety of reasons. Most of these reasons make me happy, while at the same time I'm sad I had to reduce my LTS involvement and actually I even failed to allocate those few hours which were assigned to me. So I'll keep this blog post short too, as time is my most precious ressource atm.
In March I only sponsored the axis upload for and wrote 
My monthly report covers a large part of what I have been doing in the free software world. I write it for 
Today, trying upload to jessie-backports from a Jessie jail, I got this message from dput-ng:
Last evening, 
Gnocchi provides a full REST API to manipulate time-series that are called
metrics. You can easily create a metric using a simple HTTP request:
Another object and concept that Gnocchi provides is the ability to manipulate
resources. There is a basic type of resource, called generic, which has very
few attributes. You can extend this type to specialize it, and that's what
Gnocchi does by default by providing resource types known for OpenStack such as
instance, volume, network or even image.
Like 