Search Results: "will"

8 December 2014

Bernhard R. Link: The Colon in the Shell.

I was recently asked about some construct in a shell script starting with a colon(:), leading me into a long monologue about it. Afterwards I realized I had forgotten to mention half of the nice things. So here for your amusement some usage for the colon in the shell: To find the meaning of ":" in the bash manpage[1], you have to look at the start of the SHELL BUILTIN COMMANDS section. There you find:

: [arguments]
	No effect; the command does nothing beyond expanding arguments and performing any specified redirections.  A zero exit code is returned.

If you wonder what the difference to true is: I don't know any difference (except that there is no /bin/:) So what is the colon useful for? You can use it if you need a command that does nothing, but still is a command.

For example, if you want to avoid using a negation (for fear of history expansion still being on by default on a interactive bash or wanting to support ancient shells), you cannot simply write
```
if conditon ; then
	# this will be an error
else
	echo condition is false
fi
```
but need some command there, for which the colon can be used:
```
if conditon ; then
	: # nothing to do in this case
else
	echo condition is false
fi
```
To confuse your reader, you can use the fact that the colon ignores it's arguments and you only have normal words there:
```
if conditon ; then
	: nothing to do in this case # <- this works but is not good style
else
	echo condition is false
fi
```
though I strongly recommend against it (exercise: why did I use a # there for my remark?).
This of course also works in other cases:
```
while processnext ; do
	:
done
```
The ability to ignore the actual arguments (but still processing them as with every command that ignores it arguments) can also be used, like in:
```
: $ VARNAME:=default 
```
which sets VARNAME to a default if unset or empty. (One could also use that the first time it is used, or $ VARNAME:-default everywhere, but this can be more readable).
In other cases you do not strictly need a command, but using the colon can clear things up, like creating or truncating a file using a redirection:
```
: > /path/to/file
```

Then there is more things you can do with the colon, most I'd put under "abuse":

misuing it for comments:
```
: ====== here
```
While it has the advantage of also showing up in -x output, the to be expected confusion of the reader and the danger of using any shell active character makes this general a bad idea.
As it practically the same as true it can be used as a shorter form of true. Given that true is more readable that is a bad idea. (At least it isn't as evil as using the empty string to denote true.)
```
# bad style!
if condition ; then doit= ; doit2=: ; else doit=false ; doit2=false ; fi
if $doit ; then echo condition true ; fi
if $doit2 && true ; then echo condition true ; fi
```
Another way to scare people:
```
ignoreornot=
$ignoreornot echo This you can see.
ignoreornot=:
$ignoreornot echo This you cannot see.
```
While it works, I recommend against it: Easily confusing and any > in there or $(...) will likely rain harvoc over you.
Last and least, one can shadow the built-in colon with a different one. Only useful for obfuscation, and thus likely always evil. :() :&: ;: anyone?

This is of course not a complete list. But unless I missed something else, those are the most common cases I run into. [1] <rant>If you never looked at it, better don't start: the bash manpage is legendary for being quite useless as hiding all information in other information in a quite absurd order. Unless you look at documentation about how to write a shell script parser, then the bash manpage is really what you want to read.</rant>

Clint Adams: I don't care about the sunshine, yeah

Rhoda is guarded. She is secretly in love with her brother. When he gets a girlfriend, she finds a boyfriend. She tells no one what she's really thinking. Rhoda likes to seize the day. In the midst of evening conversation, she will excuse herself to use the bathroom or come right back , then, within literally two minutes, she will go home with an acquaintance or stranger. Often these encounters or the aftermaths thereof do not go according to her liking, and she will make veiled remarks of hostility, should she see those people again. Rhoda was unhappy so she changed everything in her life. Though multiple things remained constant, she concluded that she was, in fact, the problem. Rhoda does not make enough money on which to live. She works part-time, and turns down all other job offers. Other people make up for her financial shortcomings so that she is not homeless and starving. Rhoda is certain that it is more difficult to be female than male.

7 December 2014

Sven Hoexter: Failing with F5: Implementing HTTP session persistence

In the old days people deployed the Apache HTTPD together with mod_jk and mod_balancer in front of Apache Tomcat instances to achieve scalability and resilience. That sooner or later turns out to not provide features you nowadays would like to have to run a service 24/7. So it happened that I had to jump into the sometimes cold sea of managing services fronted by F5 BigIP devices, a somewhat common load balancing solution. Soon after you start to introduce loadbalancing, because you grow beyond the "one fat webserver" state, you'll have the requirement to implement some way of session persistence. In the old days you might have added a unique jvmRoute value to your mod_balancer pool and to your tomcat configuration to match the connection to the right pool member. If you're clever you've set it on the Tomcat as a variable that automatically builds up the jvmRoute based on the hostname, so you can add new instances without having to touch the Tomcat configuration. In case you were not that clever (like we were when we started) you might have added placeholders in the server.xml to set the jvmRoute value in your deployment script. I think in a somewhat better world your first step usually is to introduce a shared session store, be it the php session handler backed by a memcached or Hazelcast for the Java afficionados, to avoid the persistence requirement on your balancer. But if you grow even bigger the requirement will soon reappear because you might have to add some form of sharding and now have to sent users to the same cluster serving a specific shard. In the end all of that is not cool and your developers have to be aware of those issues. For all cases where fault tolerance within the session is not a priority we aimed for something that we can enable or disable on the balancer without application code changes. The "let the box do its job" solution The nicest solution from our point of view is letting the F5 inject a cookie that points the device on subsequent requests to the right pool node. If you like you can define the cookie name, a timeout, force the injection on replies and some other tunables. Take a look at the help for ltm persistence cookie if you'd like to have a look at the details. The "let's do it by hand but on the box" solution Most Java frameworks provide a session cookie called JSESSIONID. Why not use that one and add a persistence table entry on the BigIP with the cookie as the lookup key? As always if there's no solution provided out of the box you can implement an iRule for it. Kind of an F5 mantra. There are many examples out there, here is what we ended up with:

when HTTP_REQUEST  
    if   [HTTP::cookie "JSESSIONID"] ne ""  
        persist uie [string tolower [HTTP::cookie "JSESSIONID"]] 1800 
     
 
when HTTP_RESPONSE  
    if   [HTTP::cookie "JSESSIONID"] ne ""  
        persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 1800

And here is where I failed Now imagine you're running an application that so far handles only machine to machine traffic without session persistence. A few month later you're approached to enable session persistence, because someone introduced a management console for the application as a web user interface. Without further ado you jump on board and provide - showcasing the flexibility of the new balancer - session persistence based on cookies injected by the BigIP. You've done it before, you enable it for the test environment, everything looks fine, you enable it for the production setup. Everything is fine (at least for now ...), the WUI workes fine, sessions are persistent to the node the user hit first. A month or two later someone from the development team approaches you again, this time asking if the balancing setup has some issues because over 90% of the live traffic hit one of the two machines in the pool until it wents down due to a crash or a deployment. Then about 90% hit the other node, even after the crashed one is resurrected. But every time the developer tests the balancing with a "while true" loop sending several hundred requests via curl the balancing is perfectly equal. After beeing puzzled by the behaviour myself we started to look at the live traffic with tcpdump and suddenly it was all clear when I saw the BigIP cookie beeing passed around. Actually everything behaved as we configured it. The application used a proper HTTP library and that one implemented cookies: If you receive one you're polite and pass it back to the sender. The original sender of course also implements HTTP with cookie support and now, due to the natur of the traffic, we have long living connections between two applications passing around the same cookie. On every single HTTP request sent through this connection. And every time the balancer will look at the cookie and happily base the routing decision on the pool member noted in this cookie. When we tested with curl we did not have the usage of cookies on our mind. Because our false assumption was that the applications do not create cookies for machine to machine traffic. Nobody, including myself, actively remembered the decision to base the session persistence on cookies injected by the balancer. We went on and switched to the iRule mentioned above. Luckily we already had that one implemented and tested for a different use case with the requirement to not inject additional cookies.

Can we maybe improve the change process for the balancer to handle a mixed traffic scenario?
Do we need monitoring to monitor the balancing of requests?
Is it a wise idea to mix machine2machine and user2machine traffic on the same port?
Should we maybe, if we seperate properly, place management applications on a different system?
Is our documentation sufficient?

Combine 3. and 4. and ask yourself how resilient you are if someone tries to exhaust the memory of the balancer with a lot of connection attempts that have random JSESSIONID cookies? Or is it even possible to exhaust ressource with faked BigIP cookies you inject? Maybe it's even a vector for something like http://events.ccc.de/congress/2011/Fahrplan/attachments/2007_28C3_Effective_DoS_on_web_application_platforms.pdf? Quite some questions left to think about and draft answers for.

Miriam Ruiz: Falling Trees, by Robert Fulghum

In the Solomon Islands in the south Pacific some villagers practice a unique form of logging. If a tree is too large to be felled with an ax, the natives cut it down by yelling at it. (Can t lay my hands on the article, but I swear I read it.) Woodsmen with special powers creep up on a tree just at dawn and suddenly scream at it at the top of their lungs. They continue this for thirty days. The tree dies and falls over. The theory is that the hollering kills the spirit of the tree. According to the villagers, it always works. Ah, those poor nave innocents. Such quaintly charming habits of the jungle. Screaming at trees, indeed. How primitive. Too bad thay don t have the advantages of modern technology and the scientific mind. Me? I yell at my wife. And yell at the telephone and the lawn mower. And yell at the TV and the newspaper and my children. I ve been known to shake my fist and yell at the sky at times. Man next door yells at his car a lot. And this summer I heard him yell at a stepladder for most of an afternoon. We modern, urban, educated folks yell at traffic and umpires and bills and banks and machines especially machines. Machines and relatives get most of the yelling. Don t know what good it does. Machines and things just sit there. Even kicking doesn t always help. As for people, well, the Solomon Islanders may have a point. Yelling at living things does tend to kill the spirit in them. Sticks and stones may break our bones, but words will break our hearts . by Robert Fulghum (All I Really Need To Know I Learned In Kindergarten)

Steve Kemp: I eventually installed Debian on a new desktop.

Recently I build a new desktop system. The hightlights of the hardware are a pair of 512Gb SSDs, which were to be configured in software RAID for additional speed and reliability (I'm paranoid that they'd suddenly stop working one day). From power-on to the (GNOME) login-prompt takes approximately 10 seconds. I had to fight with the Debian installer to get the beast working though as only the Jessie Beta 2 installer would recognize the SSDs, which are Crucual MX100 devices. My local PXE-setup which deploys the daily testing installer, and the wheezy installer, both failed to recognize the drives at all. The biggest pain was installing grub on the devices. I think this was mostly this was due to UFI things I didn't understand. I created spare partitions for it, and messaged around with grub-ufi, but ultimately disabled as much of the "fancy modern stuff" as I could in the BIOS, leaving me with AHCI for the SATA SSDs, and then things worked pretty well. After working through the installer about seven times I also simplified things by partitioning and installing on only a single drive, and only configured the RAID once I had a bootable and working system. (If you've never done that it's pretty fun. Install on one drive. Ignore the other. Then configure the second drive as part of a RAID array, but mark the other half as missing/failed/dead. Once you've done that you can create filesystems on the various /dev/mdX devices, rsync the data across, and once you boot from the system with root=/dev/md2 you can add the first drive as the missing half. Do it patiently and carefully and it'll just work :) There were some niggles though:

Jessie didn't give me the option of the gnome desktop I know/love. So I had to install gnome-session-fallback. I also had to mess around with ~/.config/autostart because the gnome-session-properties command (which should let you tweak the auto-starting applications) doesn't exist anymore.
Setting up custom keyboard-shortcuts doesn't seem to work.
I had to use gnome-tweak-tool to get icons, etc, on my desktop.

Because I assume the SSDs will just die at some point, and probably both on the same day, I installed and configured obnam to run backups. There is more testing and similar, but this is the core of my backup script:

#!/bin/sh
# backup "/" - minus some exceptions.
obnam backup -r /media/backups/storage --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/media /
# keep files for various periods
obnam forget --keep="30d,8w,8m" --repository /media/backups/storage

5 December 2014

Joey Hess: clean OS reinstalls with propellor

You have a machine someplace, probably in The Cloud, and it has Linux installed, but not to your liking. You want to do a clean reinstall, maybe switching the distribution, or getting rid of the cruft. But this requires running an installer, and it's too difficult to run d-i on remote machines. Wouldn't it be nice if you could point a program at that machine and have it do a reinstall, on the fly, while the machine was running? This is what I've now taught propellor to do! Here's a working configuration which will make propellor convert a system running Fedora (or probably many other Linux distros) to Debian:

testvm :: Host
testvm = host "testvm.kitenet.net"
        & os (System (Debian Unstable) "amd64")
        & OS.cleanInstallOnce (OS.Confirmed "testvm.kitenet.net")
                 onChange  propertyList "fixing up after clean install"
                        [ User.shadowConfig True
                        , OS.preserveRootSshAuthorized
                        , OS.preserveResolvConf
                        , Apt.update
                        , Grub.boots "/dev/sda"
                                 requires  Grub.installed Grub.PC
                        ]
        & Hostname.sane
        & Hostname.searchDomain
        & Apt.installed ["linux-image-amd64"]
        & Apt.installed ["ssh"]
        & User.hasSomePassword "root"

And here's a video of it in action.

It was surprisingly easy to build this. Propellor already knew how to create a chroot, so from there it basically just has to move files around until the chroot takes over from the old OS. After the cleanInstallOnce property does its thing, propellor is running inside a freshly debootstrapped Debian system. Then we just need a few more Propertites to get from there to a bootable, usable system: Install grub and the kernel, turn on shadow passwords, preserve a few config files from the old OS, etc. It's really astounding to me how much easier this was to build than it was to build d-i. It took years to get d-i to the point of being able to install a working system. It took me a few part days to add this capability to propellor (It's 200 lines of code), and I've probably spent a total of less than 30 days total developing propellor in its entirity. So, what gives? Why is this so much easier? There are a lot of reasons:

Technology is so much better now. I can spin up cloud VMs for testing in seconds; I use VirtualBox to restore a system from a snapshot. So testing is much much easier. The first work on d-i was done by booting real machines, and for a while I was booting them using floppies.
Propellor doesn't have a user interface. The best part of d-i is preseeding, but that was mostly an accident; when I started developing d-i the first thing I wrote was main-menu (which is invisible 99.9% of the time) and we had to develop cdebconf, and tons of other UI. Probably 90% of d-i work involves the UI. Jettisoning the UI entirely thus speeds up development enormously. And propellor's configuration file blows d-i preseeding out of the water in expressiveness and flexability.
Propellor has a much more principled design and implementation. Separating things into Properties, which are composable and reusable gives enormous leverage. Strong type checking and a powerful programming language make it much easier to develop than d-i's mess of shell scripts calling underpowered busybox commands etc. Properties often Just Work the first time they're tested.
No separate runtime. d-i runs in its own environment, which is really a little custom linux distribution. Developing linux distributions is hard. Propellor drops into a live system and runs there. So I don't need to worry about booting up the system, getting it on the network, etc etc. This probably removes another order of magnitude of complexity from propellor as compared with d-i.

This seems like the opposite of the Second System effect to me. So perhaps d-i was the second system all along? I don't know if I'm going to take this all the way to propellor is d-i 2.0. But in theory, all that's needed now is:

Teaching propellor how to build a bootable image, containing a live Debian system and propellor. (Yes, this would mean reimplementing debian-live, but I estimate 100 lines of code to do it in propellor; most of the Properties needed already exist.) That image would then be booted up and perform the installation.
Some kind of UI that generates the propellor config file.
Adding Properties to partition the disk.

cleanInstallOnce and associated Properties will be included in propellor's upcoming 1.1.0 release, and are available in git now. Oh BTW, you could parameterize a few Properties by OS, and Propellor could be used to install not just Debian or Ubuntu, but whatever Linux distribution you want. Patches welcomed...

Richard Hartmann: Release Critical Bug report for Week 49

Look at that bug count! At that pace, Jessy will happen before FOSDEM ;) The UDD bugs interface currently knows about the following release critical bugs:

In Total: 1101 (Including 169 bugs affecting key packages)
- Affecting Jessie: 226 (key packages: 119) That's the number we need to get down to zero before the release. They can be split in two big categories:
  - Affecting Jessie and unstable: 147 (key packages: 85) Those need someone to find a fix, or to finish the work to upload a fix to unstable:
    - 28 bugs are tagged 'patch'. (key packages: 22) Please help by reviewing the patches, and (if you are a DD) by uploading them.
    - 10 bugs are marked as done, but still affect unstable. (key packages: 6) This can happen due to missing builds on some architectures, for example. Help investigate!
    - 109 bugs are neither tagged patch, nor marked done. (key packages: 57) Help make a first step towards resolution!
  - Affecting Jessie only: 79 (key packages: 34) Those are already fixed in unstable, but the fix still needs to migrate to Jessie. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
    - 54 bugs are in packages that are unblocked by the release team. (key packages: 18)
    - 25 bugs are in packages that are not unblocked. (key packages: 16)

How do we compare to the Squeeze release cycle?

Week	Squeeze	Wheezy	Jessie
43	284 (213+71)	468 (332+136)	319 (240+79)
44	261 (201+60)	408 (265+143)	274 (224+50)
45	261 (205+56)	425 (291+134)	295 (229+66)
46	271 (200+71)	401 (258+143)	427 (313+114)
47	283 (209+74)	366 (221+145)	342 (260+82)
48	256 (177+79)	378 (230+148)	274 (189+85)
49	256 (180+76)	360 (216+155)	226 (147+79)
50	204 (148+56)	339 (195+144)
51	178 (124+54)	323 (190+133)
52	115 (78+37)	289 (190+99)
1	93 (60+33)	287 (171+116)
2	82 (46+36)	271 (162+109)
3	25 (15+10)	249 (165+84)
4	14 (8+6)	244 (176+68)
5	2 (0+2)	224 (132+92)
6	release!	212 (129+83)
7	release+1	194 (128+66)
8	release+2	206 (144+62)
9	release+3	174 (105+69)
10	release+4	120 (72+48)
11	release+5	115 (74+41)
12	release+6	93 (47+46)
13	release+7	50 (24+26)
14	release+8	51 (32+19)
15	release+9	39 (32+7)
16	release+10	20 (12+8)
17	release+11	24 (19+5)
18	release+12	2 (2+0)

Graphical overview of bug stats thanks to azhag:

Iustin Pop: Jump!

Jump!!! What

Tandem skydive! or alternatively: a real "one-way" plane ticket
Start point: ~4'250 meters above the ocean (14'000ft)
End point: on the beach
Total time: less than ten minutes
Time in "real" free fall: according to Wikipedia, around 12 seconds, with most of the terminal velocity being reached at about 8 seconds
Terminal velocity: ~190Km/h, ~120mph (again, according to Wikipedia)
Time in "fake" free fall until the chute opened: around one minute
Time spent going slowly down, with a few fun manoeuvres: don't remember, 3-5 minutes?

Why At the end of November, we had a team offsite planned, with lots of fun and exciting activities in a somewhat exotic location. I was quite looking forward to it, when - less than two weeks before the event - a colleague asked if anyone is interested in going skydiving as an extra activity. Without thinking too much, I said "yes" immediately, because: a) I've never done it before, and b) it sounded really cool! Other people said yes as well, so we were set up to have a really good time! Of course, as the time counted down and we were approaching the offsite, I was thinking: OK, this sounds cool, but: will I be fine? do I have altitude sickness? All kinds of such, rather logistical, questions. In order to not think too much, I did exactly zero research on the topic (all mentions of Wikipedia above are from post-fact reading). So, we went on the offsite - which was itself cool - and then, on the last day, right before going back, the skydive event! How it went The weather on the day of the jump was nice, the sky not perfectly clear, just a bit of small clouds and some haze. We waited for our turn, got the instruction for what to do (and not to do!), got hooked into the harness, prepared everything, and then boarded the plane; it needed only a very short run before taking off the ground. It took around ten minutes or so to get to the jump altitude, which I spent partially looking forward to it, partially trying to calm the various emotions I had - a very interesting mix. It was actually annoying just having to wait and wait the ten long minutes, I wished that we actually get to the jumping altitude faster. The altimeter on the instructor's hand was showing 4'000, then 4'100, 4'150, then he reminded me again what I need to do (or rather, not to do), and then - people were already jumping from the plane! I was third from our team to jump, and I had the opportunity to see how people were not simply "exiting" the plane, but rather - exiting and then immediately disappearing from view! Finally we were on the edge of the door, a push and then - I'm looking down, more than two and a half miles of nothing between me and the ground. Just air and the thought - "Why did I do this"? - as I start falling. For the first around ten seconds, it's actually a free fall, gaining speed almost at standard acceleration, and the result - Weightlessness - it's the weirdest feeling ever: all your organs floating in your body, no compression or torsion forces. Much more weird a roller-coaster that never ends; then most I had on roller-coasters was around one second of such acceleration, and you still are in contact with the chair or the restraints, whereas this long fall was very confusing for my brain - it felt somewhat like when you're tripping and you need to do something to regain balance, except in sky-diving you can't do anything, of course. There's nothing to grab, nothing to hold on, and you keep falling. After ten long seconds we reached terminal velocity, phase 1 ends, and phase 2 begins, in which - while still falling - the friction with the air compensates exactly the earth's pull and one is falling at a constant speed and it's the most wonderful state ever. Like floating on the air, except that you're actually falling at almost 200kph, and yes, the closest feeling to flying, I guess. It doesn't hurt that you're no longer weightless, which means back to some level of normality. The location of the skydive was very beautiful: the blue ocean beneath, the blue sky above, somewhere to the side the beach, and the air filling the mouth and lungs without any effort is the only sign that I'm moving really fast. The way this whole thing feels is very alien if you never jumped before, but one gets accustomed to it quite fast - and that means I got too comfortable and excited and forgot the correct position to keep my legs in, the instructor reminded me, and as I put my legs back in the correct position, which is (among others) with the soles of the feet pointing up, I felt again the air going strongly into my shoes, and a thought crossed my mind: what if I the air blows off one of my shoes (the right one, more precisely) and I lose it? How do I get to the airport for the trip back? Will I look suspicious at the security check? The banality of this thought, given that I was still up in the air somewhere and travelling quite fast, was so comical that I started laughing And then, an unexpected noise, the chute opens, and I feel like someone is pulling me strongly up. Of course nobody is pulling "up", I'm just slowing down very fast on this final phase (Wikipedia says: 3 to 4g). And then, once at the new terminal velocity, the lack of wind noise and the quietness of everything around gives a different kind of awesome - more majestic and serene this time, rather than the adrenaline-filled moments before. Because one is still up and the beach looks small, you actually feel that you're suspended in the air, almost frozen. Of course, that feeling goes away quickly when the instructor start telling me to pull the strings, and we enter a fast spin - so fast that my body is almost horizontal again - a reminder that we're still in the air, going somewhat fast, and not in "normal" conditions. I'm again reminded of the speed once we get closer to the earth, the people on the beach start to get bigger fast, and now we're gliding over the beach and finally land in the sand. The adventure is over, but I'm still pumped up and my body is still full of adrenaline, and I feel like you've just been in heaven - which is true, for some definitions of . The first thing I realise is that the earth is very solid. And not moving at all. Everything is very very slow which is both good and bad. My body is confused at the very fast sequence of events, and why did everything stop?? Conclusion I learned all about the terminal velocity, how fast you get there, and so on a day later, from Wikipedia and other sources. It helped explain and clarify the things I experienced during the dive, because there in the air I was quite confused (and my body even more so). Knowing this in advance would have spoiled the surprise, but on the other hand would have allowed me to enjoy the experience slightly better. Looking back, I can say a few of things. First, it was really awesome - not what I was expecting, much more awesome (in the real sense of awe) than I thought, but also not as easy or trivial as I believed from just seeing videos of people "floating" during their dive. Yep, worth doing, and hard to actually put in words (I tried to, but I think this rambling is more confusing than helping). Phase one was too long (and a bit scary), phase two was too short (and the best thing), phase three was relaxing (and just the right length). I also wonder how it is to jump alone - without the complicated and heavy harness, without an instructor, just you up there. Oh, and the parachute. And the reserve parachute , of course. Point is, this was awesome, but I was mostly a passive spectator, so I wonder what it feels like to be actually in control (as much as one can be, falling down) and responsible. And finally, as we left the offsite location just a couple of hours after the skydive, and we had a 4 hours flight back, I couldn't believe myself how slow everything was. I never experienced quite such a thing, I was sitting in this normal airplane flying high and fast, but for me everything was going in slow motion and I was bored out of my mind. Adrenaline aftershock or something like that? Also interesting!

Stefano Zacchiroli: Shuttleworth Foundation Flash Grant

Shuttleworth Foundation Flash Grant I'm glad to announce that I've been awarded a 5,000 USD "Flash Grant" by the Shuttleworth Foundation. Flash grants are an interesting funding model, which I've just learned about. You don't need to apply for them. Rather, you get nominated by current fellows, and then selected and approached by the foundation for funding. The grant amount is smaller than actual fellowships, but it comes with very few strings attached: furthering open knowledge (which is the foundation's core mission) and being transparent about how you use the money. I'm lucky enough to already have a full-time job to pay my bills, and I do my Free Software activism mostly in my spare time. So I plan to use the money not to pay my bills, but rather to boost the parts of my Free Software activities that could benefit from some funding. I don't have a fully detailed budget yet but, tentatively: some money will go to fund Debsources development (by others), some into promoting my thoughts on the dark ages of Free Software, and maybe some into helping the upcoming release of Debian. I'll provide a public report at the end of the funding period (~6 months from now). I'd like to thank the Shuttleworth Foundation for the grant and foundation's fellow Jonas berg for making this possible.

Michal Čihař: Weblate 2.1

Weblate 2.1 has been released today. It comes with native Mercurial support, user interface cleanup and various other fixes. Full list of changes for 2.1:

Added support for Mercurial repositories.
Replaced Glyphicon font by Awesome.
Added icons for social authentication services.
Better consistency of button colors and icons.
Documentation improvements.
Various bugfixes.
Automatic hiding of columns in translation listing for small screens.
Changed configuration of filesystem paths.
Improved SSH keys handling and storage.
Improved repository locking.
Customizable quality checks per source string.

You can find more information about Weblate on http://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances will be soon available in SUSE Studio Gallery. Weblate is also being used https://hosted.weblate.org/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others. If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you. Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far!

Filed under: English phpMyAdmin SUSE Weblate 0 comments Flattr this!

Enrico Zini: the-smell-of-email

The smell of email This was written in response to a message with a list of demotivating behaviours in email interactions, like fingerpointing, aggressiveness, resistance when being called out for misbehaving, public humiliation for mistakes, and so on There are times when I stumble on an instance of the set of things that were mentioned, and I think "ok, today I feel like doing some paid work rather than working on Debian". If another day I wake up deciding to enjoy working on Debian, which I greatly do, I try and make sure that I can focus on bits of Debian where I don't stumble on any instances of the set of things that were mentioned. Then I stumble on Gregor's GDAC and I feel like I'd happily lose one day of pay right now, and have fun with Debian. I feel like Debian is this big open kitchen populated by a lot of people:

some dump shit
some poke the shit with a stick, contributing to the spread of the smell
some carefully clean up the shit, which in the short term still contributes to the smell, but makes things better in the long term
some prepare and cook, making a nice smell of food and NOMs
some try out the food and tell us how good it was

I have fun cooking and tring out the food. I have fun being around people who cook and try out the food. The fun in the kitchen seems to be correlated to several things, one of which is that it seems to be inversely proportional to the stink. I find this metaphore interesting, and I will start thinking about the smell of a mailing list post. I expect it should put posts into perspective, I expect I will develop an instinct for it, so that I won't give a stinky post the same importance of a post that smells of food. I also expect that the more I learn to tell the smell of food from the smell of shit, the more I can help cleaning it, and the more I can help telling people who repeatedly contribute to the stink to please try cooking instead, or failing that, just try and stay out of the kitchen.

Russell Coker: BTRFS Status Dec 2014

My last problem with BTRFS was in August [1]. BTRFS has been running mostly uneventfully for me for the last 4 months, that s a good improvement but the fact that 4 months of no problems is noteworthy for something as important as a filesystem is a cause for ongoing concern. A RAID-1 Array A week ago I had a minor problem with my home file server, one of the 3TB disks in the BTRFS RAID-1 started giving read errors. That s not a big deal, I bought a new disk and did a btrfs replace operation which was quick and easy. The first annoyance was that the output of btrfs device stats reported an error count for the new device, it seems that btrfs device replace copies everything from the old disk including the error count. The solution is to use btrfs device stats -z to reset the count after replacing a device. I replaced the 3TB disk with a 4TB disk (with current prices it doesn t make sense to buy a new 3TB disk). As I was running low on disk space I added a 1TB disk to give it 4TB of RAID-1 capacity, one of the nice features of BTRFS is that a RAID-1 filesystem can support any combination of disks and use them to store 2 copies of every block of data. I started running a btrfs balance to get BTRFS to try and use all the space before learning from the mailing list that I should have done btrfs filesystem resize to make it use all the space. So my balance operation had configured the filesystem to configure itself for 2*3TB+1*1TB disks which wasn t the right configuration when the 4TB disk was fully used. To make it even more annoying the btrfs filesystem resize command takes a devid not a device name. I think that when BTRFS is more stable it would be good to have the btrfs utility warn the user about such potential mistakes. When a replacement device is larger than the old one it will be very common to want to use that space. The btrfs utility could easily suggest the most likely btrfs filesystem resize to make things easier for the user. In a disturbing coincidence a few days after replacing the first 3TB disk the other 3TB disk started giving read errors. So I replaced the second 3TB disk with a 4TB disk and removed the 1TB disk to give a 4TB RAID-1 array. This is when it would be handy to have the metadata duplication feature and copies= option of ZFS. Ctree Corruption 2 weeks ago a basic workstation with a 120G SSD owned by a relative stopped booting, the most significant errors it gave were BTRFS: log replay required on RO media and BTRFS: open_ctree failed . The solution to this is to run the command btrfs-zero-log , but that initially didn t work. I restored the system from a backup (which was 2 months old) and took the SSD home to work on it. A day later btrfs-zero-log worked correctly and I recovered all the data. Note that I didn t even try mounting the filesystem in question read-write, I mounted it read-only to copy all the data off. While in theory the filesystem should have been OK I didn t have a need to keep using it at that time (having already wiped the original device and restored from backup) and I don t have confidence in BTRFS working correctly in that situation. While it was nice to get all the data back it s a concern when commands don t operate consistently. Debian and BTRFS I was concerned when the Debian kernel team chose 3.16 as the kernel for Jessie (the next Debian release). Judging by the way development has been going I wasn t confident that 3.16 would turn out to be stable enough for BTRFS. But 3.16 is working reasonably well on a number of systems so it seems that it s likely to work well in practice. But I m still deploying more ZFS servers. The Value of Anecdotal Evidence When evaluating software based on reports from reliable sources (IE most readers will trust me to run systems well and only report genuine bugs) bad reports have a much higher weight than good reports. The fact that I ve seen kernel 3.16 to work reasonably well on ~6 systems is nice but that doesn t mean it will work well on thousands of other systems although it does indicate that it will work well on more systems than some earlier Linux kernels which had common BTRFS failures. But the annoyances I had with the 3TB array are repeatable and will annoy many other people. The ctree coruption problem MIGHT have been initially caused by a memory error (it s a desktop machine without ECC RAM) but the recovery process was problematic and other users might expect problems in such situations.

[1] http://etbe.coker.com.au/2014/08/01/more-btrfs-fun/

4 December 2014

Tomasz Buchert: BSP in Munich

My first plan was to participate in the Bug Squashing Party (BSP) in Paris, but finally I didn t manage to organize the travel and proper stay in the capital of France. Finally I settled on Munich and below you will find a summary of my stay there. While organizing my trip, the Munich natives offered me a couch to surf during the stay, which was very kind. In the end, however, I stayed at my friend s place - thanks a lot Mehdi! The BSP gathered Debian people interested in preparing Jessie release, but also KDE, Kolab and LibreOffice people. The host for the meeting was LiMux (on of the most known large-scale Linux deployments) which provided a venue, but also food (free!) and other attractions. Organizers, if you read this, I thank you for the amazing work that you have done. I arrived on late Friday s evening and so I didn t participate that day in bug squashing and took a good sleep instead. On Saturday and Sunday we were, well, squashing bugs. I haven t done NMUs before and it was a new experience to me, even though theoretically I know how it works. The DDs present at the BSP were more than helpful to help me when I was stuck. Here is a short description of what I have done:

#770085 - This is a bug I reported myself before BSP. I picked it as a low-hanging fruit and I had it fixed after 2 hours. It fixes a Python module that couldn t be imported in Python 2.
#768690 - This was a tricky one. latex-mk depends on tgif which is not going in jessie. A lazy workaround I made in the patch is the disable tgif-related functionality completely. It is not essential to the package, so it should be ok (a shameless publicity here: I use and recommend latex-make).
#768615 - A simple one - the package testsuite is tightly wired with pygments version (actually, the package is a ruby wrapper around pygments) so it had to be fixed to reflect the version in jessie.
#768695 - I ve spent nearly a whole day understanding and writing a patch. Long story short - statsmodels uses numpy, pandas and scipy, and its testsuite depends a lot of interfaces of these libraries. For example, it relies on pandas to have DateRange class which is not true for the version in jessie. We uploaded the patch which fixed most of the problems, but the build still failed on i386 due to some precision in floating point operations. The new patch was prepared and awaits upload.
#768673 - I ve traced the problem to the recent POODLE attack (Wikipedia). Ruby-httpclient uses SSLv3 by default which is deprecated and disabled on the server. The SSL negotiation ends with a nasty ECONNRESET. This problem has been fixed in the newer version of the library so I just backport in my NMU. Note: I suspect that #770616 is related.
#768905 - keyutils testsuite breaks on newer versions of the kernel (i.e., > 3.16 which is in Jessie). The bug remains, but by testing we confirmed that it won t affect jessie.

I also got my GPG keys signed and vice-versa, so I m much better connected to the web of trust now (7 signatures of DDs so far). All in all, it was a great experience which I recommend to everyone. The atmosphere is very motivating and there are many people who will gladly help you if you don t know or understand something. You don t have to be DD to participate even, but a minimal packaging experience is definitely very useful. I m quite sure that people would be glad to help even a complete newcomer (I certainly would do), but being able to build a package using apt-get source and debuild won t hurt anybody. At the very least you can always triage bugs: reproduce them, find the cause of the bug, propose a way to fix it, etc. As always you should be open-minded and willing to get your hands dirty. I want also thank Debian for sponsoring me which finally convinced me to participate (Munich is around 500 km from where I live and the travel is fairly expensive). Some links of interest:

Terse report of the BSP
Website of the BSP
LiMux
Squash (not really related, but still interesting)

Many bugs still wait to be squashed (120 at the time of writing this), so let s get back to work!

Chris Lamb: Don't ask your questions in private

(If I've linked you to this page, it is my feeble attempt to provide a more convincing justification.)

I often receive instant messages or emails requesting help or guidance at work or on one of my various programming projects. When asked why they asked privately, the responses vary; mostly along the lines of it simply being an accident, not knowing where else to ask, as well as not wishing to "disturb" others with their bespoke question. Some will be more candid and simply admit that they were afraid of looking unknowledgable in front of others. It is always tempting to simply reply with the answer, especially as helping another human is inherently rewarding unless one is a psychopath. However, one can actually do more good overall by insisting the the question is re-asked in a more public forum.

This is for many reasons. Most obviously, public questions are simply far more efficient as soon as more than one person asks that question the response can be found in a search engine or linked to in the future. These time savings soon add up, meaning that simply more stuff can be done in any given day. After all, most questions are not as unique as people think. Secondly, a private communication cannot be corrected or elaborated on if someone else notices it is incorrect or incomplete. Even this rather banal point is more subtle that it first appears the lack of possible corrections deprives both the person asking and the person responding of the true and correct answer. Lastly, conversations that happen in private are depriving others of the answer as well. Perhaps someone was curious but hadn't got around to asking? Maybe the answer or even the question! contains a clue to solving some other issue. None of this can happen if this is occurs behind closed doors. (There are lots of subtler reasons too in a large organisation or team, simply knowing what other people are curious about can be curiously valuable information.)

Note that this is not as you might immediately suspect simply a way of ensuring that one gets the public recognition or "kudos" from being seen helping others. I wouldn't deny that technical communities work on a gift economy basis to some degree, but to attribute all acts of assistance as "selfish" and value-extracting would be to take the argument too far in the other direction. Saying that, the lure and appeal of public recognition should not be understated and can certainly provide an incentive to elaborate and provide a generally superior response.

More philosophically, there's also something fundamentally "honest" about airing issues in an appropriately public and transparent manner. I feel it promotes a culture of egoless conversations, of being able to admit one's mistakes and ultimately a healthy personal mindset. So please, take care not only in the way you phrase and frame your question, but also consider wider context in which you are asking it. And don't take it too personally if I ask you to re-ask elsewhere...

Olivier Berger: Shell script to connecting to a Shibboleth protected web app with curl

Here s a shell script I ve created (reusing one meant for CAS protected resources), which will allow to connect to a Web application protected by the Shibboleth SSO mechanism.
It uses cURL to navigate through the various jumps required by the protocol, perform the necessary posts, etc.
I haven t read the Shibboleth specs, so it may not be the best way, and may not work in all cases, but that was enough for my case, at least. Feel free to improve it on Github Gists. After the connection is succesful, one may reuse the .cookieJar file to perform further cURL connections, or even some automated content mirroring with httrack, for instance (see a previous experiment of mine with httrack for Moodle).

MJ Ray: Autumn Statement #AS2014, the Google tax and how it relates to Free Software

One of the attention-grabbing measures in the Autumn Statement by Chancellor George Osborne was the google tax on profits going offshore, which may prove unworkable (The Independent). This is interesting because a common mechanism for moving the profits around is so-called transfer pricing, where the business in one country pays an inflated price to its sibling in another country for some supplies. It sounds like the intended way to deal with that is by inspecting company accounts and assessing the underlying profits. So what s this got to do with Free Software? Well, one thing the company might buy from itself is a licence to use some branding, paying a fee for reachuse. The main reason this is possible is because copyright is usually a monopoly, so there is no supplier of a replacement product, which makes it hard to assess how much the price has been inflated. One possible method of assessing the overpayment would be to compare with how much other businesses pay for their branding licences. It would be interesting if Revenue and Customs decide that there s lots of Royalty Free licensing out there including Free Software and so all licence fees paid to related companies are a tax avoidance ruse. Similarly, any premium for a particular self-branded product over a generic equivalent could be classed as profit transfer. This could have amusing implications for proprietary software producers who sell to sister companies but I doubt that the government will be that radical, so we ll continue to see absurdities like Starbucks buying all their coffee from famous coffee producing countries Switzerland and the Netherlands. Shouldn t this be stopped, really?

3 December 2014

Michael Stapelberg: Announcing the new Debian Code Search Instant

For the last few months, I have been working on a new version of Debian Code Search, and today it s going live! I call it Debian Code Search Instant, for multiple reasons, see below. A lot faster The new Debian Code Search is still hosted by Rackspace (thank you!), but our new architecture finally allows us to use their Performance Cloud Servers hardware generation. The old code search architecture was split across 5 different servers, however the search itself was only performed on a single machine with a network attached block volume backed by SSD. The new Code Search spreads out both the trigram index and the source code onto 6 different servers, and thanks to the new hardware generation, we have a locally connected SSD drive in each of them. So, we now get more than 6 times the IOPS we ve had before, at much lower latency :). Grouping by Debian source package The first feature request we ever got for Code Search was to group results by Debian source package. I ve tried tackling that before, but it s a pretty hard problem. With the new architecture and capacity, this finally becomes feasible. After your query completes, there is a checkbox called Group search results by Debian source package . Enable it, and you ll get neatly grouped results. For each package, there is a link at the bottom to refine the search to only consider that package. In case you are more interested in the full list of packages, for example because you are doing large-scale changes across Debian, that s also possible: Click on the symbol right next to the Filter by package list, and a curl command will be revealed with which you can download the full list of packages. Expensive queries (with lots of results) now possible Previously, we had a 60 second timeout during which queries must complete. This timeout has been completely lifted, and long-running queries with tons and tons of results are now possible. We kindly ask you to not abuse this feature it s not very exciting to play with complexity explosion in regular expression engines by preventing others from doing their work. If you re interested in that sort of analysis, go grab the source code and play with it on your own machine :). Instant results While your query is running, you will almost immediately see the top 10 results, even though the search may still take a while. This is useful to figure out if your regular expression matches what you thought it would, before having to wait until your 5 minute query is finished. Also, the new progress bar tells you how far the system is with your search. Instant indexing Previously, Code Search was deployed to a new set of machines from scratch every week in order to update the index. This was necessary because the performance would severely degrade while building the index, so we were temporarily running with twice the amount of machines until the new version was up. In the new architecture, we store an index for each source package and then merge these into one big index shard. This currently takes about 4 minutes with the code I wrote, but I m sure this can be made even faster if necessary. So, whenever new packages are uploaded to the Debian archive, we can just index the new version and trigger a merge. We get notifications about new package uploads from FedMsg. Packages that are not seen on FedMsg for some reason are backfilled every hour. The time between uploading a package and being able to find it in Debian Code Search therefore now ranges from a couple of minutes to about an hour, instead of about a week! New, beautiful User Interface Since we needed to rewrite the User Interface anyway thanks to the new architecture, we also spent some effort on making it modern, polished and beautiful. Smooth CSS animations make you aware of what s going on, and the search results look cleaner than ever. Conclusion At least to me, the new Debian Code Search seems like a significant improvement over the old one. I hope you enjoy the new features into which I put a lot of work. In case you have any feedback, I m happy to hear it (see the contact link at the bottom of every Code Search site).

Holger Levsen: 20141201-lts-november-2014

My LTS November In November I resumed work on Debian LTS and worked on the following packages:

DLA 88-1 for ruby1.8 fixing several CVEs as described in the announcement.
DLA 91-1 for tomcat6, mostly prepared by one of it's maintainers, Tony Mancill, also fixing several issues by upgrading to a new upstream version. I just did review, testing, release and announcement, and then figured out the proper versioing for Wheezy, which turned out to be problematic because using the recommend versioning breaks the upgrade paths, when new upstream versions are introdued to squeeze-lts which basically have same version as in will be (or are) in wheezy-security. One cause (besides the wrong recommendation which still needs fixing in our wiki page are non enforcable version constraints: the suites wheezy and squeeze-lts reside on ftp-master.debian.org (and wheezy is only updated on point releases), while wheezy-security resides on security.debian.org. Most probably we will still leave things as they are for squeeze-lts and do changes for wheezy-lts only. Oh, and the release of the tomcat6 update for wheezy is currently stalled by #770769.
DLA 92-1 for tomcat-native was also done in cooperation with Tony and is also a new upstream release, which is needed as the old version of tomcat-native doesn't function with the new tomcat6 version.
The 2.6.32.64 update of linux-2.6 has not happened yet, but is planned for the coming weekend. So far it has been done in collaboration of Moritz M hlenhoff from the security team, Ben Hutchings from the kernel team, and Rapha l Hertzog and myself from the LTS team, which I consider to be quite nice. As Rapha l had already explained, Ben has joined the LTS team and so far his contribution was to identify a problem in patch related to openvz, so I haven't published this kernel update yet. Also, there was zero feedback from testers for the openvz flavor packages - so if you are using openvz and squeeze kernels, please contact us. For all the other flavors there was positive feedback to the call for testing (thanks!) - so you might want to give these kernels a try too!

Thanks to everyone who is supporting Squeeze LTS in whatever form, according to the wide feedback there are quite many people appreciating the work! (As this was asked on IRC: if you are a maintainer preparing something for squeeze-lts, that's totally great, thanks!, just please tell us about it as this is the assumption we are working under: if noone tells the LTS team, we think we need to prepare upgrades for squeeze-lts.)

Diego Escalante Urrelo: Link pack #01

Following the lead of my dear friend Daniel and his fantastic and addictive Summing up series, here s a link pack of recent stuff I read around the web. Link pack is definitely a terrible name, but I m working on it.

How to Silence Negative Thinking
On how to avoid the pitfall of being a Negatron and not an Optimist Prime. You might be your own worst enemy and you might not even know it:

Psychologists use the term automatic negative thoughts to describe the ideas that pop into our heads uninvited, like burglars, and leave behind a mess of uncomfortable emotions. In the 1960s, one of the founders of cognitive therapy, Aaron Beck, concluded that ANTs sabotage our best self, and lead to a vicious circle of misery: creating a general mindset that is variously unhappy or anxious or angry (take your pick) and which is (therefore) all the more likely to generate new ANTs. We get stuck in the same old neural pathways, having the same negative thoughts again and again.

Meet Harlem s Official Street Photographer
A man goes around Harlem with his camera, looking to give instead of taking. Makes you think about your approach to people and photography, things can be simpler. Kinda like Humans of New York, but in Harlem. And grittier, and on film but as touching, or more:

I tell people that my camera is a healing mechanism, Allah says. Let me photograph it and take it away from you.

What Happens When We Let Industry and Government Collect All the Data They Want
Why having nothing to hide is not about the now, but about the later. It s not that someone is going to judge for pushing every detail of your life to Twitter and Instagram, it s just that something you do might be illegal a few years later:

There was a time when it was essentially illegal to be gay. There was a time when it was legal to own people and illegal for them to run away. Sometimes, society gets it wrong. And it s not just nameless bureaucrats; it s men like Thomas Jefferson. When that happens, strong privacy protections including collection controls that let people pick who gets their data, and when allow the persecuted and unpopular to survive.

The Sex-Abuse Scandal Plaguing USA Swimming
Abusive coaches and a bullying culture in sports training are the perfect storm for damaging children. And it s amazing the extent to which a corporation or institution is willing to look the other way, as long as they save face. Very long piece, but intriguing to read. What Cities Would Look Like if Lit Only by the Stars
Thierry Cohen goes around the world and builds beautiful and realistic composite images of how would big cities look like if lit only by stars. The original page has some more cities: Villes teintes (Darkened Cities). On Muppets & Merchandise: How Jim Henson Turned His Art into a Business
Lessons from how Jim Henson managed to juggle both art and business without selling out for the wrong reasons. Really interesting, and reminds you to put Henson in perspective as a very smart man who managed to convince everyone to give him money for playing with muppets. The linked video on How the Muppet Show is Made is also cool. Made me curious enough to get the book. Barbie, Remixed: I (really!) can be a computer engineer
Mattel launched the most misguided book about empowering Barbie to be anything but a computer engineer in a book about being a computer engineer. The internet did not disappoint and fixed the problem within hours. There s now even an app for that (includes user submitted pages).

Craig Small: WordPress 4.0.1 fixes for Debian stable

Previously I posted a short article about the WordPress package for Debian and how that SID was getting the updated WordPress 4.0.1 which had some security fixes. The question a lot of people were asking was: What about stable (or Wheezy). After way too much time due to other pressing issues, I have just uploaded the patched WordPress debian package for stable. The fixed version has the catchy number of 3.6.1~deb7u5. This package has all of the relevant patches that went in from WordPress 3.7.4 to 3.7.5 and there are even CVE IDs for this package (and 4.0.1 which all this stems from). Stolen from the 3.6.1 changelog, these are the fixes:

CVE-2014-9031 XSS in wptexturize() via comments or posts
CVE-2014-9033 CSRF in the password reset process
CVE-2014-9034 Denial of service for giant passwords
CVE-2014-9035 XSS in Press This
CVE-2014-9036 XSS in HTML filtering of CSS in posts
CVE-2014-9037 Hash comparison vulnerability in old passwords
CVE-2014-9038 SSRF: Safe HTTP requests did not sufficiently block the loopback IP address space
CVE-2014-9039 Email address change didn t invalidate previously sent password reset

I d like to thank the Debian security team especially Salvatore for their assistance and checking the package looked ok. Backporting and Git Part of the delay in getting the wordpress stable package out is that backporting is fiddly. I m currently using pdebuild with a custom pbuilderrc file that points to wheezy. Getting things to that point took a lot of trial and error; with one of the errors being that the pbuilder puts the files in a result directory, not the parent. This also means that the wheezy backports are out of the git repository. I see that there is a git-pbuild but to me it looks like yet another workflow which will slow me right down. Anyone got some good and simple suggestions on having a wheezy track (branch?) and requiring backporting that doesn t get complicated or broken quick? sbuild died in a wave of permission denieds within the chroot.

Next.

Previous.