Search Results: "cas"

Two months ago I asked for help porting python-debianbts, to Python3. Python-debianbts is a tiny little library that allows for querying Debian s bug tracking system. Since Debian s reportbug depends on it, the library is installed on ca 80% of the Debian installations. The main blocker back then was that python-debianbts depended on SOAPpy which was not available for Python3. So before we could port python-debianbts to Python3 we had to migrate from SOAPpy to pysimplesoap a daunting task given the beast that SOAP is. Fortunately, Gaetano Guerriero heard my call and helped a lot. First he migrated python-debianbts to pysimplesoap. Then, he ported python-debianbts to Python3 and now he is still busy fixing bugs and providing me with pull requests. I m very satisfied with the outcome. His pull requests are very high-quality and come usually with the appropriate unit tests included. While he is doing the major grunt work, I merely do some occasional nitpicking and uploading to Debian/unstable. Thank you very much Gaetano! If you ever come to Berlin, please drop me a note so I can invite you to a beer or two.

We got pretty good feedback on Gnocchi so far, even if we only had little. Recently, in order to have a better feeling of where we were at, we wanted to know how fast (or slow) Gnocchi was. The early benchmarks that some of the Mirantis engineers ran last year showed pretty good signs. But a year later, it was time to get real numbers and have a good understanding of Gnocchi capacity. Benchmark tools The first thing I realized when starting that process, is that we were lacking of tools to run benchmarks. Therefore I started to write some benchmark tools in python-gnocchiclient, which provides a command line tool to interrogate Gnocchi. I added a few basic commands to measure metric performance, such as:

$ gnocchi benchmark metric create -w 48 -n 10000 -a low
+----------------------+------------------+
  Field                  Value             
+----------------------+------------------+
  client workers         48                
  create executed        10000             
  create failures        0                 
  create failures rate   0.00 %            
  create runtime         8.80 seconds      
  create speed           1136.96 create/s  
  delete executed        10000             
  delete failures        0                 
  delete failures rate   0.00 %            
  delete runtime         39.56 seconds     
  delete speed           252.75 delete/s   
+----------------------+------------------+

The command line tool supports the --verbose switch to have detailed progress report on the benchmark progression. So far it supports metric operations only, but that's the most interesting part of Gnocchi. Spinning up some hardware I got a couple of bare metal servers to test Gnocchi on. I dedicated the first one to Gnocchi, and used the second one as the benchmark client, plugged on the same network. Each server is made of 2 Intel Xeon E5-2609 v3 (12 cores in total) and 32 GB of RAM. That provides a lot of CPU to handle requests in parallel. Then I simply performed a basic RHEL 7 installation and ran devstack to spin up an installation of Gnocchi based on the master branch, disabling all of the others OpenStack components. I then tweaked the Apache httpd configuration to use the worker MPM and increased the maximum number of clients that can sent request simultaneously. I configured Gnocchi to use the PostsgreSQL indexer, as it's the recommended one, and the file storage driver, based on Carbonara (Gnocchi own storage engine). That means files were stored locally rather than in Ceph or Swift. Using the file driver is less scalable (you have to run on only one node or uses a technology like NFS to share the files), but it was good enough for this benchmark and to have some numbers and profiling the beast. The OpenStack Keystone authentication middleware was not enabled in this setup, as it would add some delay validating the authentication token. Metric CRUD operations Metric creation is pretty fast. I managed to attain 1500 metric/s created pretty easily. Deletion is now asynchronous, which means it's faster than in Gnocchi 1.2, but it's still slower than creation: 300 metric/s can be deleted. That does not sound like a huge issue since metric deletion is actually barely used in production. Retrieving metric information is also pretty fast and goes up to 800 metric/s. It'd be easy to achieve very higher throughput for this one, as it'd be easy to cache, but we didn't feel the need to implement it so far. Another important thing is that all of these numbers are constant and barely depends on the number of the metric already managed by Gnocchi.

Operation	Details	Rate
Create metric	Created 100k metrics in 77 seconds	1300 metric/s
Delete metric	Deleted 100k metrics in 190 seconds	524 metric/s
Show metric	Show a metric 100k times in 149 seconds	670 metric/s

Sending and getting measures Pushing measures into metrics is one of the hottest topic. Starting with Gnocchi 1.1, the measures pushed are treated asynchronously, which makes it much faster to push new measures. Getting new numbers on that feature was pretty interesting. The number of metric per second you can push depends on the batch size, meaning the number of actual measurements you send per call. The naive approach is to push 1 measure per call, and in that case, Gnocchi is able to handle around 600 measures/s. With a batch containing 100 measures, the number of calls per second goes down to 450, but since you push 100 measures each time, that means 45k measures per second pushed into Gnocchi! I've pushed the test further, inspired by the recent blog post of InfluxDB claiming to achieve 300k points per second with their new engine. I ran the same benchmark on the hardware I had, which is roughly two times smaller than the one they used. I achieved to push Gnocchi to a little more than 120k measurement per second. If I had same hardware as they used, I could interpolate the results to achieve almost 250k measures/s pushed. Obviously, you can't strictly compare Gnocchi and InfluxDB since they are not doing exactly the same thing, but it still looks way better than what I expected. Using smaller batch sizes of 1k or 2k improve the throughput further to around 125k measures/s.

Operation	Details	Rate
Push metric 5k	Push 5M measures with batch of 5k measures in 40 seconds	122k measures/s
Push metric 4k	Push 5M measures with batch of 4k measures in 40 seconds	125k measures/s
Push metric 3k	Push 5M measures with batch of 3k measures in 40 seconds	123k measures/s
Push metric 2k	Push 5M measures with batch of 2k measures in 41 seconds	121k measures/s
Push metric 1k	Push 5M measures with batch of 1k measures in 44 seconds	113k measures/s
Push metric 500	Push 5M measures with batch of 500 measures in 51 seconds	98k measures/s
Push metric 100	Push 5M measures with batch of 100 measures in 112 seconds	45k measures/s
Push metric 10	Push 5M measures with batch of 10 measures in 852 seconds	6k measures/s
Push metric 1	Push 500k measures with batch of 1 measure in 800 seconds	624 measures/s
Get measures	Push 43k measures of 1 metric	260k measures/s

What about getting measures? Well, it's actually pretty fast too. Retrieving a metric with 1 month of data with 1 minute interval (that's 43k points) takes less than 2 second. Though it's actually slower than what I expected. The reason seems to be that the JSON is 2 MB big and encoding it takes a lot of time for Python. I'll investigate that. Another point I discovered, is that by default Gnocchi returns all the datapoints for each granularities available for the asked period, which might double the size of the returned data for nothing if you don't need it. It'll be easy to add an option to the API to only retrieve what you need though! Once benchmarked, that meant I was able to retrieve 6 metric/s per second, which translates to around 260k measures/s. Metricd speed New measures that are pushed into Gnocchi are processed asynchronously by the gnocchi-metricd daemon. When doing the benchmarks above, I ran into a very interesting issue: sending 10k measures on a metric would make gnocchi-metricd uses up to 2 GB RAM and 120 % CPU for more than 10 minutes. After further investigation, I found that the naive approach we used to resample datapoints in Carbonara using Pandas was causing that. I reported a bug on Pandas and the upstream author was kind enough to provide a nice workaround, that I sent as a pull request to Pandas documentation. I wrote a fix for Gnocchi based on that, and started using it. Computing the standard aggregation methods set (std, count, 95pct, min, max, sum, median, mean) for 10k batches of 1 measure (worst case scenario) for one metric with 10k measures now takes only 20 seconds and uses 100 MB of RAM 45 faster. That means that in normal operations, where only a few new measures are processed, the operation of updating a metric only takes a few milliseconds. Awesome! Comparison with Ceilometer For comparison sake, I've quickly run some read operations benchmark in Ceilometer. I've fed it with one month of samples for 100 instances polled every minute. That represents roughly 4.3M samples injected, and that took a while almost 1 hour whereas it would have taken less than a minute in Gnocchi. Then I tried to retrieve some statistics in the same way that we provide them in Gnocchi, which mean aggregating them over a period of 60 seconds over a month.

Operation	Details	Rate
Read metric SQL	Read measures for 1 metric	2min 58s
Read metric MongoDB	Read measures for 1 metric	28s
Read metric Gnocchi	Read measures for 1 metric	2s

Obviously, Ceilometer is very slow. It has to look into 4M of samples to compute and return the result, which takes a lot of time. Whereas Gnocchi just has to fetch a file and pass it over. That also means that the more samples you have (so the more time you collect data and the more resources you have), slower Ceilometer will become. This is not a problem with Gnocchi, as I emphasized when I started designing it. Most Gnocchi operations are O(log R) where R is the number of metrics or resources, whereas most Ceilometer operations are O(log S) where S is the number of samples (measures). Since is R millions of time smaller than S, Gnocchi gets to be much faster. And what's even more interesting, is that Gnocchi is entirely scalable horizontally. Adding more Gnocchi servers (for the API and its background processing worker metricd) will multiply Gnocchi performances by the number of servers added. Improvements There are several things to improve in Gnocchi, such as splitting Carbonara archives to make them more efficient, especially from drivers such as Ceph and Swift. It's already on my plate, and I'm looking forwarding to working on that! And if you have any questions, feel free to shoot them in the comment section.

I m currently on the way home, returning from a four-day workshop for high school students; ages 13 to 20, grade 9 to 13. My goal was to teach them something about proofs and logic, in a ambitiously formal and abstract way. For this workshop, I created a computer program called The Incredible Proof Machine that allows the kids to explore creating a proof, by placing blocks on a canvas and drawing connections between them (see my previous blog post for an introduction). Subtracting time spent on on breaks, organizational stuff, the time the student needed to prepare a final presentation, I think we were working for a total of 14 hours, during which we covered propositional logic. I generally let the students explore and try to solve the tasks of one session mostly on their own, followed by a common discussion of what they have just done, what it means why it makes sense etc. The sessions were: assumptions and conclusions in general, with conjunction; implication; disjunction; falsum and tertium non datur. We also briefly discussed paper proofs with the student: how they look, and how they relate to the Proof Machine proofs. We had some lecture notes that we handed out pice-wise after each session. The sections were mildly¹ password-protected to avoid that the quicker students would work ahead, thus keeping the group together. One or two of the 13 students were doing very well and eventually, I gave them the passwords to the predicate logic section and let them work them on them on their own. The quickest managed to solve almost all of these as well, but (as far I as I can tell) without a deeper understanding of the quantifiers, and more a mechanical intuition. As expected, the students were able to solve most of the exercises, even when a proper understanding of the logical semantics was not yet fully developed. This was by design: I believe that this way it was more motivating and encouraging, as they could make it work , compared to a more traditional approach of first throwing a lot of theory at them and then expecting them to apply it. This was confirmed by their feedback after the workshop. I was happy with my implementation. The students immediately could work with it with very few hick-ups, and only one minor bug occurred², which I could fix on the spot. Having everything run on in the browser was a big plus, given that we had no stable internet connection for everyone: Once the Incredible Proof Machine was loaded, the student could continue to work offline. Personally, I find that the UI is occasionally too sluggish, especially on the weaker laptops, but it seems that the students did not seem to mind. Some students tried to connect outputs with outputs or inputs with inputs and the visualization did make it clearly visible that such a link is not actually connected to the block. The predicate logic part is a bit less convincing, with e.g. scoping of local constants not easily understood. I would say that this part would work better if some explanation is given before the students start working on the more involved rules. Our room was equipped with a smartboard, and I was delighted when I found out, mostly by accident, that I could actually use my finger to drag the blocks of the proof and to draw connections. This not only gave me a Minority Report -like feeling, but also meant that it was much easier for the students to follow my explanations when they could just watch my hand, instead of trying to locate the mouse pointer on a regular projector. I m generally doubtful whether such fancy technological toys are useful in the class room, but in this case at least I liked it. The ability to scribble anywhere on the screen was occasionally a plus as well. All in all I believe the Proof Machine was a useful tool, and I am sure that without it, it would have been tricky to have students voluntarily spend 14 hours on such a relatively dry and abstract topic. Given the amount of work that went into the development, I hope that this will not be the last occasion where it is put to good use. So if you have to teach formal logic and natural deduction-style proofs, you are welcome to use the Incredible Proof Machine to get your students excited. It is purely static, i.e. needs no special server-side infrastructure, and you can define your own logic (i.e. proof rules), sessions and tasks. Also, there are a few interesting way in which the Proof Machine could be extended. In particular, I d like it to be able to generate a normal , natural-language proof from a given proof even if it will sound a bit mechanical and then use hover-highlight effects to relate the formulas and sentences in the text proof to the connections and blocks in the graphical proof. Contributions are welcome!

JavaScript based with no crypto or even obfuscation in the code.
Due to differences between the task specification as given in the code, and the task specification when normalizing the syntax of the terms, the proofs that the students created were lost when they re-loaded the page.

Last night I saw Johnny Marr play at Newcastle University Student's Union. This is the first time I've been to a gig at the refurbished NUSU. I think the last gig I saw there was Nouvelle Vague in around 2007. It was a great gig! The accoustics in the Union are pretty good. I seem to recall they've always been pretty good. It used to be very cheap, too, which is sadly no longer true. Johnny Marr is an eminently likeable character. He did a set with a fairly balanced mix of his solo work from across two albums, Smiths songs and a few other bits and pieces, including a surprise cover of "Crash" by the Primitives, which by a seeming coincidence I had re-discovered a few weeks ago and can't stop listening to. The stand-out tracks for me were

Speak Out Reach Out, from Playland
Panic, by The Smiths (short but sweet, one of my favourite Smiths songs)
New Town Velocity, From The Messenger
Crash, by The Primitives

* Strikingly similar to Gary Numan's Metal. I think this must be intentional, because the guitar work at the end of the verses also sounds a lot like Numan's Complex, which is the track that immediately follows Metal on The Pleasure Principle. Speak Out Reach Out is a great song, and not just due to the similarity.)

My previous blog post related to using ssh-keygen to generate fingerprints from SSH public keys. At the back of my mind was the fear that running the command against untrusted, user-supplied, keys might be a bad plan. So I figured I'd do some fuzzing to reassure myself. The most excellent LWN recently published a piece on Fuzzing with american fuzzy lop, so with that to guide me I generated a pair of SSH public keys, and set to work. Two days later I found an SSH public key that would make ssh-keygen segfault, and equally the SSH client (same parser), so that was a shock. The good news is that my Perl module to fingerprint keys is used like so:

my $helper = SSHKey::Fingerprint->new( key => "ssh ...." );
if ( $helper->valid() )  
   my $fingerprint = $helper->fingerprint();
   ...

The validity-test catches my bogus key, so in my personal use-cases this OK. That said it's a surprise to see this:

skx@shelob ~ $ ssh -i key.trigger.pub steve@ssh.steve.org.uk 
Segmentation fault

Similarly running "ssh-keygen -l -f ~/key.trigger.pub" results in an identical segfault. In practice this is a low risk issue, hence mentioning it, and filing the bug-report publicly, even if code execution is possible. Because in practice how many times do people fingerprint keys from unknown sources? Except for things like githubs key management page? Some people probably do it, but I assume they do it infrequently and only after some minimal checking. Anyway we'll say this is my my first security issue of 2015, we'll call it #roadhouse, and we'll get right on trademarking the term, designing the logo, and selling out for all the filthy filthy lucre ;)

The definition of thick-skinned in different dictionaries ranges from not easily offended to largely unaffected by the needs and feelings of other people; insensitive , going through able to ignore personal criticism , ability to withstand criticism and show no signs of any criticism you may receive getting to you , an insensitive nature or impervious to criticism . It essentially describes an emotionally detached attitude regarding one s social environment, the capacity or ignoring or minimizing the effects of others criticism and the priorization of the protection of one s current state over the capacity of empathizing and taking into account what others may say that don t conform to one s current way of thinking. It is essentially setting up barriers against whatever others may do that might provoke any kind of crisis or change in you. There are a few underlying assumptions in the use of this term as a something good to have, when it comes to interactions with your own community: In the first place, it assumes that your own community is essentially hostile to you, and you will have to be constantly in guard against them. It assumes that it is better to set up barriers against the influence of others within your own comunity, because in fact your own peers are out there essentially to hurt you. Or, at least, they do not care a damn about you. In second place, it assumes that changes are wrong, that personal evolution is wrong, and that the more insensitive you are to your peer s opinions, the best, because they really have nothing to contribute to help you grow as a person. I m smart; you re dumb. I m big; you re little. I m right; you re wrong; And there s nothing you can do about it. (from the film Matilda). Matilda s dad is in fact the first reference that comes to my mind when we re talking about really thick skin. Scene of the film Matilda: I'm smart, You're dumb. I'm big, You're little. I'm right, You're wrong

Scene of the film Matilda: I'm smart, You're dumb. I'm big, You're little. I'm right, You're wrong

When the main recommendation when the level of aggressiveness within a community is that someone has to make their skin thicker, they are assuming that a bullying environment will help the results. This is nothing new. It s the same theoretical base that you can see in hazing and in other activities involving harassment, abuse or humiliation in college, when initiating a person into some groups. It s supposed to build character, to make someone closer to the alpha male stereotype and, in essence, make us better men (yes, I am using the word men on purpose, because insensitiveness is not usually seen as a positive trait in females). The assumption is that a community with a hard environment and individuals prepared for the war is more effective than a more civilized one. Luckily, that s not the point of view of most members of the Debian Community, and many other Free/Open Source projects. The Code of Conduct is very explicit when it says that a community in which people feel threatened is not a healthy community , and that is good. The Debian Project welcomes and encourages participation by everyone (Diversity Statement), including those with a thin skin, and I m happy about that. There are still a lot of things to improve, of course, but I have the feeling that -despite the occasional complains that having to be respectful to others take the fun away- we re moving in the right direction. The best tip I can give you on thickening your skin = don t. That is, don t thicken your skin. Having a thin skin means you re letting the world in, you re letting what s out there affect what s in you. It means you re connected. You re open. You re considerate and you ll consider it whatever it might be. Having a thin skin may be dangerous, sure, because you might take in so much that you pop, like that blueberry girl from Willy Wonka. But life is dangerous. A thick skin protects you from everything, but it also protects you from everything from the gentle touches of life, from the subtle emotions of others, the deep connections, the meaningful interactions. (Top Ten Tips on how to Thicken your Skin).

I got home from a law lecture[1] today and took apart my TV. As you do. I d a couple of motivations for this. The TV is an LG 55LM8600-UC. That -UC is important - it means it s the US model. I shipped it back to the UK along with the rest of my belongings at the start of the year. It s a lovely TV, and I m very happy with the picture quality. Up until now I ve been running it through a 240v->120v transformer, because the back clearly states 120v 50/60Hz. Having had success converting my Roomba PSU to UK voltage I was wondering if it might be easy enough to do the same for the TV. So the first reason for taking it apart was to look at the PSU board and work out if I could replace it with a UK version, or fix up a handful of components myself. Turns out its rated for 100-240v 50/60Hz so it looks like it should be fine to plug into UK mains directly. And, having done so, it all seems fine. However I don t take any responsibility if you try this yourself and it all goes up in smoke. The other reason was to have a general poke at trying to get access to the onboard CPU - the firmware updates are signed and encrypted so I m a bit stuck working from just those. My TV is from the US; it therefore has an ATSC tuner, and is no good for picking up the DVB-T/T2 broadcast in the UK. It also has a range of smart apps built in. Of course are also tailored to the US, so the Amazon Instant Video app connects to amazon.com and is no use with my Amazon UK account. Netflix is smarter about this and detects where you re located rather than requiring logging in to the appropriate geographic site. However the Netflix app is old and doesn t support user profiles. Which causes a problem when I spend an afternoon watching Unbreakable Kimmy Schmidt and it screws up the recommendations for my girlfriend. In an ideal world I d easily be able to update the apps on my TV to recent versions appropriate for my geographic location. Instead, the current state of home entertainment software platforms is at least 10 years behind mobile phones. Some devices allow you to get hold of SDKs even as an end user, but you re stuck with a multiple ecosystems even between different models from the same manufacturer. LG has no interest in updating my 2012-era TV. They d rather I buy a nice new 2015 TV. Which means at present my TV is mostly a really nice monitor for my XBMC/Kodi box. I can get away with using my Blu-ray player for Netflix, as it does user profile, but it s also a US model and again Amazon Instant Video doesn t work with a UK account. I should point out /that/ box is from Panasonic, just in case anyone thinks I m needlessly picking on LG. My amp is an Onkyo and it would be nice if it could stream music from Amazon/Google as well (it supports Spotify and a couple of others already), but again no joy there. I understand there are DRM issues around video and audio content, but these problems seem to have largely been solved in the Android world while still providing a fairly standard development environment and allowing end users to install their own software. It would seem to make more sense if there was a standard TV / Blu-ray platform that the likes of Netflix and Amazon could write apps once for, reducing the effort required on both sides to get the latest and greatest services on end user equipment. I guess this is some of what Google was trying to achieve with Google TV, but it s probably less interesting to them than mobile devices as I certainly don t tend to do things like search on my TV; I just want to view content and would be mighty pissed off if it started advertising things at me unexpectedly. While I m not planning to buy a new TV any time soon, I could do with a UK spec 3D-capable Blu-ray player. Is there anything out there with some commitment to open apps / proper updates? I haven t found anything, but I d love to be proved wrong. [1] It was on the duty of care and negligence, if you care.

We all live beneath the great Big Dipper. So goes a line in a song I once heard the great Tony Brown sing. As I near the completion of my private pilot s training, I ve had more and more opportunities to literally see the wisdom in those words. Here s a story of one of them. Night

A shining beacon in space all alone in the night. Babylon 5

A night cross-country flight, my first, taking off from a country airport. The plane lifts into the dark sky. The bright white lights of the runway get smaller, and disappear as I pass the edge of the airport. Directly below me, it looks like a dark sky; pitch black except for little pinpoints of light at farmhouses and the occasional car. But seconds later, an expanse of light unfolds, from a city it takes nearly an hour to reach by car. Already it is in sight, and as I look off to other directions, other cities even farther away are visible, too. The ground shows a square grid, the streets of the city visible for miles. There are no highway signs in the sky. There are no wheels to keep my plane pointed straight. Even if I point the plane due south, if there is an east wind, I will actually be flying southwest. I use my eyes, enhanced by technology like a compass, GPS, and VHF radio beacons, to find my way. Before ever getting into the airplane, I have carefully planned my route, selecting both visual and technological waypoints along the way to provide many ways to ensure I am on course and make sure I don t get lost. Soon I see a flash repeating every few seconds in the distance an airport beacon. Then another, and another. Little pinpoints of light nestled in the square orange grid. Wichita has many airports, each with its beacon, and one of them will be my first visual checkpoint of the night. I make a few clicks in the cockpit, and soon the radio-controlled lights at one of the airports spring to life, illuminating my first checkpoint. More than a mile of white lights there to welcome any plane that lands, and to show a point on the path of any plane that passes. I continue my flight, sometimes turning on lights at airports, other times pointing my plane at lights from antenna towers (that are thousands of feet below me), sometimes keeping a tiny needle on my panel centered on a radio beacon. I land at a tiny, deserted airport, and then a few minutes later at a large commercial airport. On my way back home, I fly solely by reference to the ground directly over a freeway. I have other tools at my disposal, but don t need them; the steady stream of red and white lights beneath me are all I need. From my plane, there is just red and white. One after another, passing beneath me as I fly over them at 115 MPH. There is no citizen or undocumented immigrant, no rich or poor, no atheist or Christian or Muslim, no Democrat or Replubican, no American or Mexican, no adult or child, no rich or poor, no Porsche or Kia. Just red and white points of light, each one the same as the one before and the one after, stretching as far as I can see into the distance. All alike in the night. You only need to get a hundred feet off the ground before you realize how little state lines, national borders, and the machinery of politics and exclusion really mean. From the sky, the difference between a field of corn and a field of wheat is far more significant than the difference between Kansas and Missouri. This should be a comforting reminder to us. We are all unique, and beautiful in our uniqueness, but we are all human, each as valuable as the next. Up in the sky, even though my instructor was with me, during quiet times it is easy to feel all alone in the night. But I know it is not the case. Only a few thousand feet separate my plane from those cars. My plane, too, has red and white lights.

How often at night, when the heavens were bright,
With the light of the twinkling stars
Have I stood here amazed, and asked as I gazed,
If their glory exceed that of ours. John A. Lomax

Strange as it may be, it turns out I never wrote about dh-exec yet, even though it is close to being four years old. Gosh, time flies so fast when you're having fun! Since its first introduction, there's been a reasonable uptake in dh-exec use: as of this writing, 129 packages build-depend on dh-exec. One might think this would be a cause for celebration, that the package is put to great use. But it's not.You see, a significant number of those 129 packages are doing it wrong, and need not build-depend on dh-exec at all.The purpose of dh-exec is to allow one to do things stock debhelper can't do, such as renaming files during the dh_install phase, or applying architecture or build profile based filtering, or doing environment variable substitution. There are many legit uses for all of these features, but there are some which can be easily solved without using dh-exec. So first, I'll talk a bit about the don'ts.What not to use dh-exec forOne of the most abused part of dh-exec is its variable substitution feature, and it is often used without need, to install multiarch-related files. While that is one intended use-case, there are few situations currently in the archive that stock debhelper can't handle. Let me explain the situation!Lets assume we have an upstream package, where the build system is something as common as autotools or CMake, for which debhelper can automatically set the appropriate flags and paths. Furthermore, lets assume that the upstream build system would install the following files:

/usr/lib/x86_64-linux-gnu/libalala.so.1.2.3
/usr/lib/x86_64-linux-gnu/libalala.so.1
/usr/lib/x86_64-linux-gnu/libalala.so
/usr/lib/x86_64-linux-gnu/libalala.a
/usr/lib/x86_64-linux-gnu/pkgconfig/libalala.pc
/usr/include/lalala/lalala.h

We want to include the first two in the libalala1 package, the rest in libalala-dev, so what do we do? We use stock debhelper, of course!

libalala1.install:
```
usr/lib/*/lib*.so.*
```

libalala-dev.install:

usr/lib/*/lib*.a
usr/lib/*/lib*.so
usr/lib/*/pkgconfig/*.pc
usr/include/*

That is all you need. In this case, there is absolutely no need for dh-exec. While using dh-exec without need is not much of an issue, because it only increases the space required for the build and build times by a tiny bit, I would still strongly recommend not introducing dh-exec needlessly. Why? Because of simplicity and aesthetics.So, if you find yourself doing any of these, or similar, that's a sign you are doing things wrong:

usr/lib/$ DEB_HOST_MULTIARCH 
usr/lib/$ DEB_HOST_MULTIARCH /*.so.* /usr/lib/$ DEB_HOST_MULTIARCH /
usr/lib/$ DEB_HOST_MULTIARCH /pkgconfig/*.pc /usr/lib/$ DEB_HOST_MULTIARCH /pkgconfig
usr/lib/$ DEB_HOST_MULTIARCH /package/*.so /usr/lib/$ DEB_HOST_MULTIARCH /package

Unless there are other directories under usr/lib that are not the multiarch triplet, using stock debhelper and wildcards is not only more succinct, simpler and more elegant, but also lighter on resources required.When dh-exec becomes usefulChanging installation pathsOnce you want to change where things get installed, then dh-exec becomes useful:

usr/lib/*.so.* /usr/lib/$ DEB_HOST_MULTIARCH 
usr/lib/$ DEB_HOST_MULTIARCH /package/* /usr/lib/$ DEB_HOST_MULTIARCH /package-plugins/
some/dir/*.so.* /usr/lib/$ DEB_HOST_MULTIARCH

This usually happens when upstream's build system can't easily be taught about multiarch paths. For most autotools and CMake-based packages, this is not the case.Variable substitutionConsider this case:

#!/usr/bin/dh-exec
/usr/share/octave/packages/mpi-$ DEB_VERSION_UPSTREAM /hello2dimmat.m /usr/share/doc/octave-mpi/examples/hello2dimmat.m
/usr/share/octave/packages/mpi-$ DEB_VERSION_UPSTREAM /hellocell.m /usr/share/doc/octave-mpi/examples/hellocell.m

Here, the build supplies $ DEB_VERSION_UPSTREAM , and using dh-exec allows one to have a generic debian/links file, that does not need updating whenever the upstream version changes. We can't use wildcards here, because dh_link does not expand them.Renaming filesIn case one needs to rename files during the dh_install phase, dh-exec can be put to use for great results:

ssh-agent-filter.bash-completion => usr/share/bash-completion/completions/ssh-agent-filter

FilteringSometimes one would wish to conditionally install something based on the architecture, or the build profile. In this case, dh-exec is the tool to turn to:

<!stage1 !stage2> ../../libdde-linux26/Makeconf* usr/share/libdde_linux26
usr/lib/gvfs/gvfsd-afc                                          [!hurd-any]
usr/lib/gvfs/gvfsd-gphoto2                                      [linux-any]

Last week, I've been invited to the OpenStack Paris meetup #16, whose subject was about metrics in OpenStack. Last time I spoke at this meetup was back in 2012, during the OpenStack Paris meetup #2. A very long time ago!

I talked for half an hour about Gnocchi, the OpenStack project I've been running for 18 months now. I started by explaining the story behind the project and why we needed to build it. Ceilometer has an interesting history and had a curious roadmap these last year, and I summarized that briefly. Then I talk about how Gnocchi works and what it offers to users and operators. The slides where full of JSON, but I imagine it offered a interesting view of what the API looks like and how easy it is to operate. This also allowed me to emphasize how many use cases are actually really covered and solved, contrary to what Ceilometer did so far. The talk has been well received and I got a few interesting questions at the end. The video of the talk (in French) and my slides are available on my talk page and below. I hope you'll enjoy it.

Two s390x changes landed in Debian unstable today:

A new version of zipl-installer that will infer the root filesystem to pass to the kernel using mapdevfs instead of enforcing a single DASD partition for it.
A new version of sysconfig-hardware that contains a hook that activates all configured disks in the initramfs instead of a single one passed in as the root parameter in the kernel command-line. Thanks to Stephen Powell for the patch.

With this it should be possible to install Debian on s390x with root on LVM. I'd be happy to hear feedback about installations with any configuration, be it root on a single DASD or root on LVM. Unless you set both mirror/udeb/suite and mirror/suite to unstable you'll need to wait until the changes are in testing, though. (The debian-installer build does not matter as zipl-installer is not part of the initrd and sysconfig-hardware is part of the installation.)

Furthermore I uploaded a new version of Hercules - a z/Architecture emulator - to get a few more years of maintenance into Debian. See its upstream changelog for details on the changes (old 3.07 new 3.11).

At this point qemu at master is also usable for s390x emulation. It is much faster than Hercules, but it uses newfangled I/O subsystems like virtio. Hence we will need to do some more patching to make debian-installer just work. One patch for netcfg is in to support virtio networking correctly, but then it forces the user to configure a DASD. (Which would be as wrong if Fibre Channel were to be used.) In the end qemu and KVM on s390x look so much like a normal x86 VM that we could drop most of the special-casing of s390x (netcfg-static instead of netcfg; network-console instead of using the VM console; DASD configuration instead of simply using virtio-blk devices; I guess we get to keep zIPL for booting).

What happened in the reproducible builds effort this week: Toolchain fixes

Barry Warsaw uploaded wheel/0.26.0-1 which now uses SOURCE_DATE_EPOCH instead of WHEEL_FORCE_TIMESTAMP and uses time.gmtime() to avoid timezone issues. Patches by Chris Lamb and Reiner Herrmann.

Andreas Metzler uploaded autogen/1:5.18.6-1 in experimental with several patches for reproducibility issues written by Valentin Lorentz. Groovy upstream has merged a change proposed by Emmanuel Bourg to remove timestamps generated by groovydoc. Ben Hutchings submitted a patch to add support for SOURCE_DATE_EPOCH in linux-kbuild as an alternate way to specify the build timestamp. Reiner Herrman has sent a patch adding support for SOURCE_DATE_EPOCH in docbook-utils. Packages fixed The following packages became reproducible due to changes in their build dependencies: commons-csv. fest-reflect, sunxi-tools, xfce4-terminal, The following packages became reproducible after getting fixed:

httpcomponents-client/4.5.1-1 by Emmanuel Bourg.
jhead/1:3.00-2 by Ludovic Rousseau.
libvigraimpex/1.10.0+dfsg-10 by Daniel Stender.
linux/4.2-1~exp1 by Ben Hutchings.
maelstrom/1.4.3-L3.0.6+main-7 by Santiago Vila.
nedit/1:5.6a-3 by Paul Gevers.
pitivi/0.94-4 by Sebastian Dr ge, reported by Scott Kitterman.
procenv/0.40-2 by James Hunt.
seyon/2.20c-32 by Santiago Vila.
slib/3b1-5 by Santiago Vila.
spock/0.7-groovy-2.0-1 by Emmanuel Bourg.
u-boot/2015.10~rc4+dfsg1-1 by Vagrant Cascadian.
vdr-plugin-remote/0.7.0-1 by Tobias Grimm.

Some uploads fixed some reproducibility issues but not all of them:

dutch/1:2.10-4 uploaded by Thijs Kinkhorst, original patch by Chris Lamb.

Patches submitted which have not made their way to the archive yet:

#800776 on cluster-glue: exports SOURCE_DATE_EPOCH in debian/rules.

Tomasz Rybak uploaded pycuda/2015.1.3-1 which should fix reproducibility issues. The package has not been tested as it is in contrib. akira found an embedded code copy of texi2html in fftw. reproducible.debian.net Email notifications are now only sent once a day per package, instead of on each status change. (h01ger) disorderfs has been temporarily disabled to see if it had any impact on the disk space issues. (h01ger) When running out of disk space, build nodes will now automatically detect the problem. This means test results will not be recorded as FTBFS and the problem will be reported to Jenkins maintainers. (h01ger) The navigation menu of package pages has been improved. (h01ger) The two amd64 builders now use two different kernel versions: 3.16 from stable and 4.1 from backports on the other. (h01ger) We now graph the number of packages which needs to be fixed. (h01ger) Munin now creates graphs on how many builds were performed by build nodes (example). (h01ger) A migration plan has been agreed with DSA on how to turn Jenkins into an official Debian service. A backport of jenkins-job-builder for Jessie is currently missing. (h01ger) Package reviews 119 reviews have been removed, 103 added and 45 updated this week. 16 fail to build from source issues were reported by Chris Lamb and Mattia Rizzolo. New issue this week: timestamps_in_manpages_generated_by_docbook_utils. Misc. Allan McRae has submitted a patch to make ArchLinux pacman record a .BUILDINFO file.

I've mentored a number of students in 2013, 2014 and 2015 for Debian and Ganglia and most of the companies I've worked with have run internships and graduate programs from time to time. GSoC 2015 has just finished and with all the excitement, many students are already asking what they can do to prepare and be selected for Outreachy or GSoC in 2016. My own observation is that the more time the organization has to get to know the student, the more confident they can be selecting that student. Furthermore, the more time that the student has spent getting to know the free software community, the more easily they can complete GSoC. Here I present a list of things that students can do to maximize their chance of selection and career opportunities at the same time. These tips are useful for people applying for GSoC itself and related programs such as GNOME's Outreachy or graduate placements in companies. Disclaimers There is no guarantee that Google will run the program again in 2016 or any future year until the Google announcement. There is no guarantee that any organization or mentor (including myself) will be involved until the official list of organizations is published by Google.

Do not follow the advice of web sites that invite you to send pizza or anything else of value to prospective mentors. Following the steps in this page doesn't guarantee selection. That said, people who do follow these steps are much more likely to be considered and interviewed than somebody who hasn't done any of the things in this list. Understand what free software really is You may hear terms like free software and open source software used interchangeably. They don't mean exactly the same thing and many people use the term free software for the wrong things. Not all projects declaring themselves to be "free" or "open source" meet the definition of free software. Those that don't, usually as a result of deficiencies in their licenses, are fundamentally incompatible with the majority of software that does use genuinely free licenses. Google Summer of Code is about both writing and publishing your code and it is also about community. It is fundamental that you know the basics of licensing and how to choose a free license that empowers the community to collaborate on your code well after GSoC has finished. Please review the definition of free software early on and come back and review it from time to time. The The GNU Project / Free Software Foundation have excellent resources to help you understand what a free software license is and how it works to maximize community collaboration. Don't look for shortcuts There is no shortcut to GSoC selection and there is no shortcut to GSoC completion. The student stipend (USD $5,500 in 2014) is not paid to students unless they complete a minimum amount of valid code. This means that even if a student did find some shortcut to selection, it is unlikely they would be paid without completing meaningful work. If you are the right candidate for GSoC, you will not need a shortcut anyway. Are you the sort of person who can't leave a coding problem until you really feel it is fixed, even if you keep going all night? Have you ever woken up in the night with a dream about writing code still in your head? Do you become irritated by tedious or repetitive tasks and often think of ways to write code to eliminate such tasks? Does your family get cross with you because you take your laptop to Christmas dinner or some other significant occasion and start coding? If some of these statements summarize the way you think or feel you are probably a natural fit for GSoC. An opportunity money can't buy The GSoC stipend will not make you rich. It is intended to make sure you have enough money to survive through the summer and focus on your project. Professional developers make this much money in a week in leading business centers like New York, London and Singapore. When you get to that stage in 3-5 years, you will not even be thinking about exactly how much you made during internships. GSoC gives you an edge over other internships because it involves publicly promoting your work. Many companies still try to hide the potential of their best recruits for fear they will be poached or that they will be able to demand higher salaries. Everything you complete in GSoC is intended to be published and you get full credit for it. Imagine a young musician getting the opportunity to perform on the main stage at a rock festival. This is how the free software community works. It is a meritocracy and there is nobody to hold you back.

Having a portfolio of free software that you have created or collaborated on and a wide network of professional contacts that you develop before, during and after GSoC will continue to pay you back for years to come. While other graduates are being screened through group interviews and testing days run by employers, people with a track record in a free software project often find they go straight to the final interview round. Register your domain name and make a permanent email address Free software is all about community and collaboration. Register your own domain name as this will become a focal point for your work and for people to get to know you as you become part of the community. This is sound advice for anybody working in IT, not just programmers. It gives the impression that you are confident and have a long term interest in a technology career. Choosing the provider: as a minimum, you want a provider that offers DNS management, static web site hosting, email forwarding and XMPP services all linked to your domain. You do not need to choose the provider that is linked to your internet connection at home and that is often not the best choice anyway. The XMPP foundation maintains a list of providers known to support XMPP. Create an email address within your domain name. The most basic domain hosting providers will let you forward the email address to a webmail or university email account of your choice. Configure your webmail to send replies using your personalized email address in the From header. Update your ~/.gitconfig file to use your personalized email address in your Git commits. Create a web site and blog Start writing a blog. Host it using your domain name. Some people blog every day, other people just blog once every two or three months. Create links from your web site to your other profiles, such as a Github profile page. This helps reinforce the pages/profiles that are genuinely related to you and avoid confusion with the pages of other developers. Many mentors are keen to see their students writing a weekly report on a blog during GSoC so starting a blog now gives you a head start. Mentors look at blogs during the selection process to try and gain insight into which topics a student is most suitable for. Create a profile on Github Github is one of the most widely used software development web sites. Github makes it quick and easy for you to publish your work and collaborate on the work of other people. Create an account today and get in the habbit of forking other projects, improving them, committing your changes and pushing the work back into your Github account. Github will quickly build a profile of your commits and this allows mentors to see and understand your interests and your strengths. In your Github profile, add a link to your web site/blog and make sure the email address you are using for Git commits (in the ~/.gitconfig file) is based on your personal domain. Start using PGP Pretty Good Privacy (PGP) is the industry standard in protecting your identity online. All serious free software projects use PGP to sign tags in Git, to sign official emails and to sign official release files. The most common way to start using PGP is with the GnuPG (GNU Privacy Guard) utility. It is installed by the package manager on most Linux systems. When you create your own PGP key, use the email address involving your domain name. This is the most permanent and stable solution. Print your key fingerprint using the gpg-key2ps command, it is in the signing-party package on most Linux systems. Keep copies of the fingerprint slips with you.

This is what my own PGP fingerprint slip looks like. You can also print the key fingerprint on a business card for a more professional look. Using PGP, it is recommend that you sign any important messages you send but you do not have to encrypt the messages you send, especially if some of the people you send messages to (like family and friends) do not yet have the PGP software to decrypt them. If using the Thunderbird (Icedove) email client from Mozilla, you can easily send signed messages and validate the messages you receive using the Enigmail plugin. Get your PGP key signed Once you have a PGP key, you will need to find other developers to sign it. For people I mentor personally in GSoC, I'm keen to see that you try and find another Debian Developer in your area to sign your key as early as possible. Free software events Try and find all the free software events in your area in the months between now and the end of the next Google Summer of Code season. Aim to attend at least two of them before GSoC. Look closely at the schedules and find out about the individual speakers, the companies and the free software projects that are participating. For events that span more than one day, find out about the dinners, pub nights and other social parts of the event. Try and identify people who will attend the event who have been GSoC mentors or who intend to be. Contact them before the event, if you are keen to work on something in their domain they may be able to make time to discuss it with you in person. Take your PGP fingerprint slips. Even if you don't participate in a formal key-signing party at the event, you will still find some developers to sign your PGP key individually. You must take a photo ID document (such as your passport) for the other developer to check the name on your fingerprint but you do not give them a copy of the ID document. Events come in all shapes and sizes. FOSDEM is an example of one of the bigger events in Europe, linux.conf.au is a similarly large event in Australia. There are many, many more local events such as the Debian UK mini-DebConf in Cambridge, November 2015. Many events are either free or free for students but please check carefully if there is a requirement to register before attending. On your blog, discuss which events you are attending and which sessions interest you. Write a blog during or after the event too, including photos.

Quantcast generously hosted the Ganglia community meeting in San Francisco, October 2013. We had a wild time in their offices with mini-scooters, burgers, beers and the Ganglia book. That's me on the pink mini-scooter and Bernard Li, one of the other Ganglia GSoC 2014 admins is on the right. Install Linux GSoC is fundamentally about free software. Linux is to free software what a tree is to the forest. Using Linux every day on your personal computer dramatically increases your ability to interact with the free software community and increases the number of potential GSoC projects that you can participate in. This is not to say that people using Mac OS or Windows are unwelcome. I have worked with some great developers who were not Linux users. Linux gives you an edge though and the best time to gain that edge is now, while you are a student and well before you apply for GSoC. If you must run Windows for some applications used in your course, it will run just fine in a virtual machine using Virtual Box, a free software solution for desktop virtualization. Use Linux as the primary operating system. Here are links to download ISO DVD (and CD) images for some of the main Linux distributions:

If you are nervous about getting started with Linux, install it on a spare PC or in a virtual machine before you install it on your main PC or laptop. Linux is much less demanding on the hardware than Windows so you can easily run it on a machine that is 5-10 years old. Having just 4GB of RAM and 20GB of hard disk is usually more than enough for a basic graphical desktop environment although having better hardware makes it faster. Your experiences installing and running Linux, especially if it requires some special effort to make it work with some of your hardware, make interesting topics for your blog. Decide which technologies you know best Personally, I have mentored students working with C, C++, Java, Python and JavaScript/HTML5. In a GSoC program, you will typically do most of your work in just one of these languages. From the outset, decide which language you will focus on and do everything you can to improve your competence with that language. For example, if you have already used Java in most of your course, plan on using Java in GSoC and make sure you read Effective Java (2nd Edition) by Joshua Bloch. Decide which themes appeal to you Find a topic that has long-term appeal for you. Maybe the topic relates to your course or maybe you already know what type of company you would like to work in. Here is a list of some topics and some of the relevant software projects:

System administration, servers and networking: consider projects involving monitoring, automation, packaging. Ganglia is a great community to get involved with and you will encounter the Ganglia software in many large companies and academic/research networks. Contributing to a Linux distribution like Debian or Fedora packaging is another great way to get into system administration.
Desktop and user interface: consider projects involving window managers and desktop tools or adding to the user interface of just about any other software.
Big data and data science: this can apply to just about any other theme. For example, data science techniques are frequently used now to improve system administration.
Business and accounting: consider accounting, CRM and ERP software.
Finance and trading: consider projects like R, market data software like OpenMAMA and connectivity software (Apache Camel)
Real-time communication (RTC), VoIP, webcam and chat: look at the JSCommunicator or the Jitsi project
Web (JavaScript, HTML5): look at the JSCommunicator

Before the GSoC application process begins, you should aim to learn as much as possible about the theme you prefer and also gain practical experience using the software relating to that theme. For example, if you are attracted to the business and accounting theme, install the PostBooks suite and get to know it. Maybe you know somebody who runs a small business: help them to upgrade to PostBooks and use it to prepare some reports. Make something Make some small project, less than two week's work, to demonstrate your skills. It is important to make something that somebody will use for a practical purpose, this will help you gain experience communicating with other users through Github. For an example, see the servlet Juliana Louback created for fixing phone numbers in December 2013. It has since been used as part of the Lumicall web site and Juliana was selected for a GSoC 2014 project with Debian. There is no better way to demonstrate to a prospective mentor that you are ready for GSoC than by completing and publishing some small project like this yourself. If you don't have any immediate project ideas, many developers will also be able to give you tips on small projects like this that you can attempt, just come and ask us on one of the mailing lists. Ideally, the project will be something that you would use anyway even if you do not end up participating in GSoC. Such projects are the most motivating and rewarding and usually end up becoming an example of your best work. To continue the example of somebody with a preference for business and accounting software, a small project you might create is a plugin or extension for PostBooks. Getting to know prospective mentors Many web sites provide useful information about the developers who contribute to free software projects. Some of these developers may be willing to be a GSoC mentor. For example, look through some of the following:

Planet / Blog aggregation sites: these sites all have links to the blogs of many developers. They are useful sources of information about events and also finding out who works on what.
Developer profile pages. Many projects publish a page about each developer and the packages, modules or other components he/she is responsible for. Look through these lists for areas of mutual interest.
- my package list in Debian
- my package list in Fedora
Developer github profiles. Github makes it easy to see what projects a developer has contributed to. To see many of my own projects, browse through the history at my own Github profile

Getting on the mentor's shortlist Once you have identified projects that are interesting to you and developers who work on those projects, it is important to get yourself on the developer's shortlist. Basically, the shortlist is a list of all students who the developer believes can complete the project. If I feel that a student is unlikely to complete a project or if I don't have enough information to judge a student's probability of success, that student will not be on my shortlist. If I don't have any student on my shortlist, then a project will not go ahead at all. If there are multiple students on the shortlist, then I will be looking more closely at each of them to try and work out who is the best match. One way to get a developer's attention is to look at bug reports they have created. Github makes it easy to see complaints or bug reports they have made about their own projects or other projects they depend on. Another way to do this is to search through their code for strings like FIXME and TODO. Projects with standalone bug trackers like the Debian bug tracker also provide an easy way to search for bug reports that a specific person has created or commented on. Once you find some relevant bug reports, email the developer. Ask if anybody else is working on those issues. Try and start with an issue that is particularly easy and where the solution is interesting for you. This will help you learn to compile and test the program before you try to fix any more complicated bugs. It may even be something you can work on as part of your academic program. Find successful projects from the previous year Contact organizations and ask them which GSoC projects were most successful. In many organizations, you can find the past students' project plans and their final reports published on the web. Read through the plans submitted by the students who were chosen. Then read through the final reports by the same students and see how they compare to the original plans. Start building your project proposal now Don't wait for the application period to begin. Start writing a project proposal now. When writing a proposal, it is important to include several things:

Think big: what is the goal at the end of the project? Does your work help the greater good in some way, such as increasing the market share of Linux on the desktop?
Details: what are specific challenges? What tools will you use?
Time management: what will you do each week? Are there weeks where you will not work on GSoC due to vacation or other events? These things are permitted but they must be in your plan if you know them in advance. If an accident or death in the family cut a week out of your GSoC project, which work would you skip and would your project still be useful without that? Having two weeks of flexible time in your plan makes it more resilient against interruptions.
Communication: are you on mailing lists, IRC and XMPP chat? Will you make a weekly report on your blog?
Users: who will benefit from your work?
Testing: who will test and validate your work throughout the project? Ideally, this should involve more than just the mentor.

If your project plan is good enough, could you put it on Kickstarter or another crowdfunding site? This is a good test of whether or not a project is going to be supported by a GSoC mentor. Learn about packaging and distributing software Packaging is a vital part of the free software lifecycle. It is very easy to upload a project to Github but it takes more effort to have it become an official package in systems like Debian, Fedora and Ubuntu. Packaging and the communities around Linux distributions help you reach out to users of your software and get valuable feedback and new contributors. This boosts the impact of your work. To start with, you may want to help the maintainer of an existing package. Debian packaging teams are existing communities that work in a team and welcome new contributors. The Debian Mentors initiative is another great starting place. In the Fedora world, the place to start may be in one of the Special Interest Groups (SIGs). Think from the mentor's perspective After the application deadline, mentors have just 2 or 3 weeks to choose the students. This is actually not a lot of time to be certain if a particular student is capable of completing a project. If the student has a published history of free software activity, the mentor feels a lot more confident about choosing the student. Some mentors have more than one good student while other mentors receive no applications from capable students. In this situation, it is very common for mentors to send each other details of students who may be suitable. Once again, if a student has a good Github profile and a blog, it is much easier for mentors to try and match that student with another project.

Conclusion Getting into the world of software engineering is much like joining any other profession or even joining a new hobby or sporting activity. If you run, you probably have various types of shoe and a running watch and you may even spend a couple of nights at the track each week. If you enjoy playing a musical instrument, you probably have a collection of sheet music, accessories for your instrument and you may even aspire to build a recording studio in your garage (or you probably know somebody else who already did that). The things listed on this page will not just help you walk the walk and talk the talk of a software developer, they will put you on a track to being one of the leaders. If you look over the profiles of other software developers on the Internet, you will find they are doing most of the things on this page already. Even if you are not selected for GSoC at all or decide not to apply, working through the steps on this page will help you clarify your own ideas about your career and help you make new friends in the software engineering community.

Playing with FUSE and git. I've been playing with FUSE and git to make a file system, for fun. There's already many filesystems that are implemented with FUSE, and there are quite a few ones that implement filesystem for git, but I don't use any of them. I wondered why that is the case but tried to build one anyway. It's in github repository gitlstreefs. I have created several toy file systems in C++. ninjafs is one where it shows ninja targets as files and builds the file target when file is actually needed. They aren't quite as useful yet but an interesting excercise, FUSE was reasonably straightforward to implement simple filesystems with.

It has been bugging me for a while. When responding to an email, you often want to delete all the content (or part of the previous content) until the end of the email's body. However it would be nice to leave your signature in place. For that I came up with this nifty little vim trick:

nnoremap <silent> <leader>gr <Esc>d/--\_.*Dariusz<CR>:nohl<CR>O

Assuming that your signature starts with -- and the following line starts with your name (in my case it is Dariusz), this will delete all the content from the current line until the signature. Then it will remove search highlighting, and finally move one line up.

Updates for my multi-boot/multi-purpose USB stick: All components have been updated to the latest versions and I have confirmed that all of them still boot properly although changes in the grub.cfg file are necessary. So going through these explanations one will end up with a usable USB stick that can boot you into TAILS, System Rescue CD, GNU Parted Live CD, GRML, and also can boot into an installation of Debian 8.2 Jessie installation. All this while still being able to use the USB stick as normal media. multi-usb-disk

Since there have been a lot of updates, and also changes in the setup and grub config file, I include the full procedure here, that is, merging and updating these previous posts: USB stick with Tails and SystemRescueCD, Tails 1.2.1, Debian jessie installer, System Rescue CD on USB, USB stick update: TAILS 1.4, GParted 0.22, SysResCD 4.5.2, ~~Debian Jessie~~, and USB stick update: Debian is back, plus GRML. Let us repeat some things from the original post concerning the wishlist and the main players: I have a long wishlist of items a boot stick should fulfill

boots into Tails, SystemRescueCD, GParted, and GRML
boots on both EFI and legacy systems
uses the full size of the USB stick (user data!)
allows installation of Debian
if possible, preserve already present user data on the stick

Requirements A USB stick, the iso images of TAILS 1.6, SystemRescueCD 4.6.0, GParted Lice CD 0.23.0, GRML 2014.11, and some tool to access iso images, for example ISOmaster (often available from your friendly Linux distribution). I assume that you have already an USB stick prepared as described previously. If this is not the case, please go there and follow the section on preparing your usb stick. Three types of boot options We will employ three different approaches to boot special systems: the one is directly from an iso image (easiest, simple to update), the other via extraction of the necessary kernels and images (bit painful, needs some handwork), and the last one is a mixture necessary to get Debian booting (most painful, needs additional downloads and handwork). At the moment we have the following status with respect to boot methods:

Booting directly from ISO image: System Rescue CD, GNOME Parted Live CD, GRML
Extraction of kernels/images: TAILS
Mixture: Debian Jessie install

Booting from ISO image Grub has gained quite some time ago the ability to boot directly from an ISO image. In this case the iso image is mounted via loopback, and the kernel and initrd are specified relatively to the iso image root. This system makes it extremely easy to update the respective boot option: just drop the new iso image onto the USB stick, and update the isofile setting. One could even use some -latest method, but I prefer to keep the exact name. For both SystemRescueCD, GNOME Partition Live CD, and GRML, just drop the iso files into /boot/iso/, in my case /boot/iso/systemrescuecd-x86-4.6.0.iso and /boot/iso/gparted-live-0.23.0-1-i586.iso. After that, entries like the following have to be added to grub.cfg. For the full list see grub.cfg:

submenu "System Rescue CD 4.6.0 (via ISO) ---> "  
  set isofile="/boot/iso/systemrescuecd-x86-4.6.0.iso"
  menuentry "SystemRescueCd (64bit, default boot options)"  
        set gfxpayload=keep
        loopback loop (hd0,1)$isofile
        linux   (loop)/isolinux/rescue64 isoloop=$isofile
        initrd  (loop)/isolinux/initram.igz
   
  ...
 
 
submenu "GNU/Gnome Parted Live CD 0.23.0 (via ISO) ---> "  
  set isofile="/boot/iso/gparted-live-0.23.0-1-i586.iso"
  menuentry "GParted Live (Default settings)" 
    loopback loop (hd0,1)$isofile
    linux (loop)/live/vmlinuz boot=live union=overlay username=user config components quiet noswap noeject  ip= net.ifnames=0 nosplash findiso=$isofile
    initrd (loop)/live/initrd.img
   
  ...
 
 
submenu "GRML 2014.11 ---> "  
  menuentry "Grml Rescue System 64bit"  
        iso_path="/boot/iso/grml64-full_2014.11.iso"
        export iso_path
        loopback loop (hd0,1)$iso_path
        set root=(loop)
        kernelopts=" ssh=foobarbaz toram  "
        export kernelopts
        configfile /boot/grub/loopback.cfg

Note the added isoloop=$isofile and findiso=$isofile that helps the installer find the iso images. Booting via extraction of kernels and images This is a bit more tedious, but still not too bad. Installation of TAILS files Assuming you have access to the files on the TAILS CD via the directory ~/tails, execute the following commands:

mkdir -p /usbstick/boot/tails
cp -a ~/tails/live/* /usbstick/boot/tails/

The grub.cfg entries look now similar to the following:

submenu "TAILS Environment 1.6 ---> "  
  menuentry "Tails64 Live System"  
        linux   /boot/tails/vmlinuz2 boot=live live-media-path=/boot/tails config live-media=removable nopersistent noprompt timezone=Etc/UTC block.events_dfl_poll_msecs=1000 splash noautologin module=Tails
        initrd  /boot/tails/initrd2.img
 
   
  ...

The important part here is the live-media-path=/boot/tails, otherwise TAILS will not find the correct files for booting. The rest of the information was extracted from the boot setup of TAILS itself. Mixture of iso image and extraction Debian jessie As mentioned in the previous post, booting Debian/Jessie installation images via any method laid out above didn t work, since the iso images is never found. It turned out that the current installer iso images do not contain the iso-scan package, which is responsible for searching and loading of iso images. But with a small trick one can overcome this: One needs to replace the initrd that is on the ISO image with one that contains the iso-scan package. And we do not need to create these initrd by ourselves, but simply use the ones from hd-media type installer. I downloaded the following four gzipped initrds from one of the Debian mirrors: i386/initrd text mode, i386/initrd gui mode, amd64/initrd text mode, amd64/initrd gui mode, and put them into the USB stick s boot/debian/install.386, boot/debian/install.386/gtk, boot/debian/install.amd, boot/debian/install.amd/gtk, respectively. Finally, I added entries similar to this one (rest see the grub.cfg file):

submenu "Debian 8.2 Jessie NetInstall ---> "  
    set isofile="/boot/iso/firmware-8.2.0-amd64-i386-netinst.iso"
    menuentry '64 bit Install'  
        set background_color=black
        loopback loop (hd0,1)$isofile
        linux    (loop)/install.amd/vmlinuz iso-scan/ask_second_pass=true iso-scan/filename=$isofile vga=788 -- quiet 
        initrd   /boot/debian/install.amd/initrd.gz
     
    ...

Again an important point, don t forget the two kernel command line options: iso-scan/ask_second_pass=true iso-scan/filename=$isofile, otherwise you probably will have to make the installer scan all disks and drives completely, which might take ages. Current status of USB stick Just to make sure, the usb stick should contain at the current stage the following files:

/boot/
    iso/
        firmware-8.2.0-amd64-i386-netinst.iso
        gparted-live-0.23.0-1-i586.iso
        grml64-full_2014.11.iso
        systemrescuecd-x86-4.6.0.iso
    tails/
        vmlinuz Tails.module initrd.img ....
    grub/
        fonts/
            lots of files
        locale/
            lots of files
        x86_64-efi/
            lots of files
        font.pf2
        grubenv
        grub.cfg            *this file we create in the next step!!*
/EFI
    BOOT/
        BOOTX64.EFI

The Grub config file grub.cfg The final step is to provide a grub config file in /usbstick/boot/grub/grub.cfg. I created one by looking at the isoboot.cfg files both in the SystemRescueCD, TAILS iso images, GParted iso image, and the Debian/Jessie image, and converting them to grub syntax. Excerpts have been shown above in the various sections. I spare you all the details, grab a copy here: grub.cfg Conclusion That s it. Now you can anonymously provide data about your evil government, rescue your friends computer, fix a forgotten Windows password, and above all, install a proper free operating system. If you have any comments, improvements or suggestions, please drop me a comment. I hope this helps a few people getting a decent USB boot stick running. Enjoy.

The automotive industry has been in the spotlight after a massive scandal at Volkswagen, using code hidden in the engine management software to cheat emissions tests. What else is hidden in your new car's computer? Every technology we use in our lives is becoming more computerized, even light bulbs and toilet seats. In a large piece of equipment like a car, there are many opportunities for computerization. In most cases, consumers aren't even given a choice whether or not they want software in their car. It has long been known that such software is spying on the habits of the driver and this data is extracted from the car when it is serviced and uploaded to the car company. Car companies are building vast databases about the lifestyles, habits and employment activities of their customers. Computers aren't going away, so what can be done? Most people realize that computers aren't going to go away any time soon. That doesn't mean that people have to put up with these deceptions and intrusions on our lives. For years, many leading experts in the software engineering world have been promoting the benefits and principles of free software. What we mean by free is that users, regulators and other independent experts should have the freedom to see and modify the source code in the equipment that we depend on as part of modern life. In fact, experts generally agree that there is no means other than software freedom to counter the might of corporations like Volkswagen and their potential to misuse that power, as demonstrated in the emissions testing scandal. If Governments and regulators want to be taken seriously and protect society, isn't it time that they insisted that the car industry replaces all hidden code with free and open source software?

[As You Like It, Act III, Scene V] Hey Beautiful I thought of you on the Fourth of July; which does not, in and of itself, distinguish the day from any other since I met you. It was remarkable only in that it was justifiable, given our conversation about fireworks displays. Of course, I'm happy to take the flimsiest of displays as an excuse to mark you. I'm home alone right now, after watching The Bad Seed. Truth be told, the shadows keep spooking me. I can't seem to stop myself from imagining precocious blonde murderers in them. It's a manageable silliness, but made a little less so by the fact that I forgot to lock the door. Less troublesome than the night I spent after Ringu (the original Japanese version of The Ring). I finished watching it in the wee hours of the morning, and I wanted to go to sleep. I was in half a stupor, but the incessant inner critic in me kept imagining all the changes that could have been made to make the movie more truly unsettling until visions of Obake were swimming around me. Ordinarily I doubt I'd be bothered particularly by a 50's classic, but went back up to Boston this afternoon. 's at a conference in Finland, so I invited him down to visit while I have the apartment all to myself. It was strange to have a visitor actually in my home for whom I didn't have to play at being contented. At any rate, being around for three days essentially meant carrying on a three-day-long conversaiton, and the abrupt drop of sociability makes me feel my isolation a little more acutely. We watched The Way We Were together, and agreed that it should be remade with casting that actually works. I hadn't seen it before, and was surprised to find it unusually nuanced and substantial, yet still not good. It was nice to have someone around who would dissect it with me afterward. It's been a while since I actually discussed a film with someone. Partly out of my own fault; I don't always enjoy verbalizing my opinions of movies immediately after watching them if I've found them in the least bit moving. I guess I consider the aftertaste part of the experience. In this case, we both felt the film had missed its emotional mark so it wasn't so much of an issue. On the other hand, I don't find most movies moving. I find them frustratingly flawed, and by the time they end I'm raring to rant about their petty contradictions and failures of logic. I think it might give people the impression that I don't actually make any effort to tease out the messages filmmakers weave into their work. Or maybe I'm just making excuses for having uninteresting friends. Either way, it was pleasant to be in the company of someone eager to tolerate the convolutions of my thought process. On Wednesday night, I have a date to meet up with some former co-workers/friends that I've been passively avoiding for several years now. Every time I fail to carefully manage my visibility, people seem to come flooding back into my life. This time the culprit was a day spent logged into instant messaging without stringent privacy settings. I should feel lucky for that, I suppose. I'm not sure how I actually feel. One of the formers is a woman I was very close to, as far as most of the world her included could discern. The other is a Boston boy I admired for the touch of golden child in the air that hung about him. The main theme of his life was (and I suspect still is) getting to drinks with friends at one of his regular bars at the end of every evening. Which did not at all stop him from being productive, interested in the world, and bright. If he had been a girl I would have been hateful with envy. Instead he's always stood out in my memory as the only person I've had a bit of a crush on despite not finding him particularly intellectually stimulating. A month or so ago he sold his company to google. Now he spends a lot of time out of town giving lectures. I suspect I may be generally happy for him, and I'm not quite sure what to do with that. Thursday I'm leaving for a few days in Denver. I wish I hadn't scheduled it for a time when I could have been the sole occupant of my domicile, but other than that I'm looking forward to it. I've no idea what I'll do there, but at least that means I really am going someplace that wouldn't occur to me outside of a peculiar set of constraints. I think it would be advisable to work out the transportation system before I depart, though. I hope this letter finds you relatively satisfied, at a minimum. I don't actually need to tell you how much I miss talking to you, do I? You're wonderful. Affectionally, as always,

Review: Half Life, by S.L. Huang

Series:	Russell's Attic #2
Publisher:	S.L. Huang
Copyright:	2014
ISBN:	0-9960700-5-2
Format:	Kindle
Pages:	314

This is a sequel to Zero Sum Game and the second book about Cas Russell, a mercenary superhero (in a world without the concept of superheroes) with preternatural ability to analyze anything about her surroundings with mathematics. While it reuses some personal relationships from the first book and makes a few references to the villains, it's a disconnected story. It would be possible to start here if you wanted to. Cas is now in the strange and unexpected situation of having friends, and they're starting to complicate her life. First, Arthur has managed to trigger some unexpected storehouse of morals and gotten her to try to stop killing people on jobs. That conscience may have something to do with her willingness to take a job from an apparently crazy man who claims a corporation has stolen his daughter, a daughter who appears nowhere in any official records. And when her other friend, Checker, gets in trouble with the mob, Cas tries to protect him in her own inimitable way, which poses a serious risk of shortening her lifespan. Even more than the first book, the story in Half Life is a mix of the slightly ridiculous world of superheroes with gritty (and bloody) danger. It featuring hit men, armed guards, lots of guns, and quite a lot of physical injury and blood. A nasty corporation that's obviously hiding serious secrets shares pages with the matriarch of a mob family who considers Checker sleeping with her daughter to be an abuse of her honor. The story eventually escalates into more outlandish bits of technology, an uncanny little girl, and a plot that would feel at home in a Batman comic. I like books that don't take themselves too seriously, but the contrast between the brutal treatment Cas struggles through and the outrageous mad scientist villain provokes a bit of cognitive whiplash. That said, the villains of Half Life are neither as freakish nor as disturbing as those in Zero Sum Game, which I appreciated. Huang packs in several plot twists, some inobvious decisions and disagreements between Russell and her friends about appropriate strategy, and Cas's discovery that there are certain things she cares very strongly about other than money and having jobs. Cas goes from a barely moral, very dark hero in the first book to something closer to a very grumbly chaotic good who insists she's not as good as she actually is. It's a standard character type, but Huang does a good job with it. Huang also adds a couple of supporting cast members in this book that I hope will stick around. Pilar starts as a receptionist at one of the companies Cas breaks into, and at first seems like she might be comic relief. But she ends up being considerably more competent than she first appears (or that she seems to realize); by the end of the book, I had a lot of respect for her. And Miri makes only a few appearances, but her unflappable attitude is a delight. I hope to see more of her. The biggest drawback to this book for me is that Cas gets hurt a lot. At times, the story falls into one of the patterns of urban fantasy: the protagonist gets repeatedly beaten up and abused until they figure out what's going on, and spends most of the story up against impossible odds and feeling helpless. That's not a plot pattern I'm fond of. I don't enjoy reading about physical pain, and I had trouble at some points in the story with the constant feeling of dread. Parts of the book I read in short bursts, putting it aside to look at something else. But the sense of dread falls off towards the end of the book, as Cas figures out what's actually going on, and none of it is as horrible as it felt it could be. If you have a similar problem with some urban fantasy tropes, I think it's safe to stick with the story. This was a fun story, but it doesn't develop much in the way of deeper themes in the series. There's essentially no Rio, no further discoveries about the villains of the first book, and no further details on what makes Cas tick or why she seems to be the only, or at least one of the few, super-powered people in this world. The advance publicity for the third book seems to indicate that's coming next. I'm curious enough now that I'll keep reading this series. Recommended if you liked the first book. Half Life is very similar, but I think slightly better. Followed by Root of Unity. Rating: 7 out of 10

If you see someone on the Debian ReproducibleBuilds project, buy him/her a beer. This work is awesome.

What happened in the reproducible builds effort this week: Media coverage Nathan Willis covered our DebConf15 status update in Linux Weekly News. Access to non-LWN subscribers will be given on Thursday 24th. Linux Journal published a more general piece last Tuesday. Unexpected praise for reproducible builds appeared this week in the form of several iOS applications identified as including spyware. The malware was undetected by Apple screening. This actually happened because application developers had simply downloaded a trojaned version of XCode through an unofficial source. While reproducible builds can't really help users of non-free software, this is exactly the kind of attacks that we are trying to prevent in our systems. Toolchain fixes

Mathieu Malaterre uploaded abi-compliance-checker/1.99.11-1 which drops the timestamps from the generated HTML reports and makes the generated .abi.tar.gz files reproducible. Original patches by Chris Lamb.

Niko Tyni wrote and uploaded a better patch for the source order problem in libmodule-build-perl. Tristan Seligmann identified how the code generated by python-cffi could be emitted in random order in some cases. Upstream has already fixed the problem. Packages fixed The following 24 packages became reproducible due to changes in their build dependencies: apache-curator, checkbox-ng, gant, gnome-clocks, hawtjni, jackrabbit, jersey1, libjsr305-java, mathjax-docs, mlpy, moap, octave-geometry, paste, pdf.js, pyinotify, pytango, python-asyncssh, python-mock, python-openid, python-repoze.who, shadow, swift, tcpwatch-httpproxy, transfig. The following packages became reproducible after getting fixed:

apparmor/2.10-2 uploaded by intrigeri, fixed upstream by Christian Boltz, with the same change suggested by Reiner Herrmann.
ardour/1:4.2~dfsg-2 by IOhannes m zm lnig.
dcmtk/3.6.1~20150629-1 uploaded by Andreas Tille, original patch by akira.
deap/1.0.1-4 by Daniel Stender.
firebird2.5/2.5.4.26856.ds4-2 by Damyan Ivanov.
gamera/3.4.2+svn1437-1 by Daniel Stender.
genometools/1.5.7-1 by Sascha Steinbiss.
golang-github-go-xorm-core/0.4.4-1 by Alexandre Viau.
klibc/2.0.4-4 by Ben Hutchings.
libgtk2-perl/2:1.2496-3 by intrigeri.
lsof/4.89+dfsg-0.1 uploaded by Laurent Bigonville, original patch by Lunar.
monotone/1.1-6 by Markus Wanner.
ndisc6/1.0.1-4 by Santiago Vila.
privoxy/3.0.23-4 by Roland Rosenfeld.
ruby-flexmock/2.0.0~rc1-1 by Antonio Terceiro.
ruby-html2haml/2.0.0-1 by Lunar.
tunnelx/20140102-3 uploaded by Wookey, original patch by Chris Lamb.
wtforms/2.0.2-1 by Orestis Ioannou, original patch by Juan Picca.

Some uploads fixed some reproducibility issues but not all of them:

maxima/5.37-1 by Camm Maguire, report by akira.

Patches submitted which have not made their way to the archive yet:

#783152 on kmod by Lunar: export SOURCE_DATE_EPOCH in debian/rules.
#799010 on 389-ds-base by Chris Lamb: use SOURCE_DATE_EPOCH value as the build date.
#799206 on python-sqlalchemy-utils by Chris Lamb: sort the list of extra requirement.
#799330 on cappuccino by Chris Lamb: pass a fixed seed to polygen.
#799410 on segment by Chris Lamb: use date of the latest debian/changelog entry as build date.

reproducible.debian.net Tests for Coreboot, OpenWrt, NetBSD, and FreeBSD now runs weekly (instead of monthly). diffoscope development Python 3 offers new features (namely yield from and concurrent.futures) that could help implement parallel processing. The clear separation of bytes and unicode strings is also likely to reduce encoding related issues. Mattia Rizolo thus kicked the effort of porting diffoscope to Python 3. tlsh was the only dependency missing a Python 3 module. This got quickly fixed by a new upload. The rest of the code has been moved to the point where only incompatibilities between Python 2.7 and Pyhon 3.4 had to be changed. The commit stream still require some cleanups but all tests are now passing under Python 3. Documentation update The documentation on how to assemble the weekly reports has been updated. (Lunar) The example on how to use SOURCE_DATE_EPOCH with CMake has been improved. (Ben Beockel, Daniel Kahn Gillmor) The solution for timestamps in man pages generated by Sphinx now uses SOURCE_DATE_EPOCH. (Mattia Rizzolo) Package reviews 45 reviews have been removed, 141 added and 62 updated this week. 67 new FTBFS reports have been filled by Chris Lamb, Niko Tyni, and Lisandro Dami n Nicanor P rez Meyer. New issues added this week: randomness_in_r_rdb_rds_databases, python-ply_compiled_parse_tables. Misc. The prebuilder script is now properly testing umask variations again. Santiago Villa started a discussion on debian-devel on how binNMUs would work for reproducible builds.

Search Results: "cas"

13 October 2015

Bastian Venthur: Thanks!

Julien Danjou: Benchmarking Gnocchi for fun & profit

12 October 2015

Joachim Breitner: Incredible Proof Machine put to the test

Jonathan Dowland: Johnny Marr

Steve Kemp: So about that idea of using ssh-keygen on untrusted input?

9 October 2015

Miriam Ruiz: Thick Skin (within Free/Open Source communities)

8 October 2015

Jonathan McDowell: The sad state of home entertainment software platforms

6 October 2015

John Goerzen: Objects On Earth Are Closer Than They Appear

Gergely Nagy: What dh-exec is, and what it isn't for

5 October 2015

Julien Danjou: Gnocchi talk at OpenStack Paris Meetup #16

4 October 2015

Philipp Kern: Root on LVM on Debian s390x, new Hercules

Lunar: Reproducible builds: week 23 in Stretch cycle

2 October 2015

Daniel Pocock: Want to be selected for Google Summer of Code 2016?

1 October 2015

Junichi Uekawa: Playing with FUSE and git.

29 September 2015

Dariusz Dwornikowski: Delete until signature in vim

Norbert Preining: Multi-boot stick update: TAILS 1.6, SysresCD 4.6.0, GParted 0.23, Debian 8.2

23 September 2015

Daniel Pocock: The only way to ensure the VW scandal never happens again

22 September 2015

Clint Adams: 404 No forwarding address for Ryan Hockert-Lotz in Fall River, MA

Russ Allbery: Review: Half Life

21 September 2015

Lunar: Reproducible builds: week 21 in Stretch cycle