Search Results: "xam"

3 December 2013

Iustin Pop: A not so dark sky

I'm aware of how not-dark our modern city night sky is, but sometimes it still heavily surprises me how full of light it actually is. Yesterday I went about five kilometres out of Z rich; I thought at that distance, on a reasonably dark hill, it would be good enough for some night-sky shots. So I setup my tripod, only to realise I can't expose over 30 seconds because (at low ISO and fast aperture, shooting not straight up), everything is bleached out: I couldn't believe my eyes. Yes, I heard that one needs to go 100Km out of big cities, but but So I went and found that since about 2004, all of Switzerland is light polluted - even going on top of Jungfrau, for example, wouldn't completely eliminate it. I also learned about Merle Walker's equation - yeah, 200Km or more away from light sources would be good. Not here, in this quite small, quite populated country in the middle of Europe :/ I realised that as soon as my eyes got acclimated to the location I was at, I could easily see everything around me, due to the light pollution. I miss a really dark sky - I remember a couple of years back, in a different country, stopping at night on the side of the road, and being shocked at how the sky was filled with starts. I didn't have any camera with me at that time Heck, I remember seeing the Milky Way way back as a child, but nowadays when I get out of work, I can barely see 4-5 points of light in the sky. Anyway, enough with ranting Thanks to modern technology, one can recover lots of detail, even in a washed out picture. So in the end, shooting straight up, I could get some resemblance of structure (and this was only at 28mm, which is not that wide): Or alternatively, one can use the glowing light for a bit of play/contrast: I also played with stacking images via Deep Sky Stacker, but stacking - I learned to my surprise - only works to reduce noise in the final image, and not to make it "brighter" or more detailed. Live and learn The result was a bit better than not stacked, but not by much: This was a 25 5s, ISO 800, f/4, same 28mm lens. What I found surprising is the "non-star object" (to call it so) near the centre of the image - I have no idea what it is, it definitely doesn't look like a star, it could be an elliptical galaxy or so. I tried navigating back in time via Stellarium, but I don't remember the orientation of the lens, so it will remain a mystery to me. Anyway, two more pictures and higher resolutions on a Smugmug album. Feel free to leave a comment or drop me an email if you have suggestions where to take nice night sky photos in Switzerland

Russell Coker: Preferring Not To

I ve just read Bartleby the Scrivener which is a short story about a scrivener who refused to work saying I d prefer not to . It reminded me of some situations in the computer industry. I ve never seen a single case where someone preferred not to work when everyone around them (colleagues and management) wanted them to work. But then the incidence of having an entire team and management wanting to work efficiently isn t nearly as common as one might imagine. In some cases it s desired that someone not work, such as a former colleague who was hired as a sysadmin but did nothing but change backup tapes (a few hours work per week). Not having him login as root improved the general reliability of the servers but it was fortunate that we never needed to restore from backups One time I had a colleague who preferred to spend most of his time in the office searching the Internet for videos of street fights. I have often told colleagues that I would prefer them to work, but in the case of a guy who s only hobby is street-fighting I decided to let it go. Managing people can be difficult, particularly for someone who doesn t like disagreements. Some managers that I ve reported to seemed to prefer not to manage in an apparent attempt to avoid disputes. One time when I complained about a colleague not even having a suitable computer to permit doing any work a manager responded with the rhetorical question what do you expect me to do? . That manager didn t do any annual reviews of staff for over a year, he only eventually did some reviews because he was told that his scheduled promotion was on hold until he got them done. I got the impression that at least two levels of management preferred not to work at that company. Sometimes it just gets weird though, such as the occasion when I was the only member of a team and the manager who was supposedly managing no-one but me never seemed to have time to have a meeting with me. But he didn t want me to bypass him and talk directly to other people in the company, so he preferred not to work and not to have anyone else do his job. Most of the companies that I ve worked for in a full-time capacity didn t seem to have any effective technical interviews (note that I ve mostly worked for financial companies and ISPs not free software companies). So it seems that anyone with minimal computer skills who wants a well paying job could just send out a CV to a bunch of recruiting agencies, get interviewed by enough companies to eventually hit one without a technical interview process and then find a job that doesn t require work. Depression The Wikipedia page about Bartleby the Scrivener [2] suggests that Bartleby was depressed. I wonder how much of the lack of performance I ve witnessed has been due to depression. There appears to be a strong correlation between work environments that cause depression and people preferring not to work. Maybe managers should be considering how to make work less depressing to try and get more effective employees (in terms of quality and quantity of work). One example of this is the sysadmin team death spiral I ve witnessed where no-one can automate solving problems (EG by cron jobs to manage resource usage and analysis tools to find minor problems before they become major problems) because everyone is dedicated to fixing things that break needlessly (EG systems crashing due to lack of disk space). When people start getting control over recurring problems and automating things then the work becomes increasingly about solving problems and less about implementing the same manual processes every day/week and it s more fun and effective for everyone. At the BoF on depression at LCA 2013 one delegate stated that many companies have people in HR who can arrange support for depressed employees. Apparently if you are depressed and you work for a company that s large enough to have a HR department then it can be beneficial to talk to HR about it. That probably works well in the case where an employee is depressed but the company is working well. But in the case where the company isn t working well it seems unlikely to help. David Graeber wrote an interesting article about Bullshit Jobs [3]. He goes a bit far, I don t think that late night pizza delivery is a bullshit job and actuaries are useful to society. But his points about the existence of useless jobs are reasonable. Management Levels I sometimes wonder whether there is some benefit in establishing social norms about working and then having management take little interest in how it happens. If a team works well together then management could just set deadlines (which would be negotiated with employees who know what s possible) and let the team work out how to do it. Then instead of having one manager for each team of ~10 people who theoretically tracks what everyone is doing you could have one manager for a dozen teams who just tracks overall team performance essentially remove a layer of management. Valve is famous for having no formal management structure and for getting things done, unfortunately that apparently allows school-style cliques to block actions [4]. But I think that the Valve experiment is useful and provides some ideas that can be used by other companies. Maybe if instead of requiring consensus of the entire company for hiring decisions they only required consensus of the team things would have worked better. Of course another down-side to such things is that hierarchical management can be good for avoiding discrimination and bullying. The article I cited about Valve compares it to high-school. It could be that Valve employees were all nice people who only hired other nice people. But if similar systems were implemented in many companies then some would surely end up being like a typical high school with all the bullying and mistreatment of minority groups that entails. Michael O. Church wrote an interesting article in which he divides employees into four categories, loser , clueless , psychopaths , and technocrats (note that he didn t invent the first three names) [5]. In his model the clueless category includes most middle-management. I think that there are some problems with Michael s model and I m not arguing for a technocracy (which is how this post might be interpreted in terms of his ideas). But I think he demonstrates some of the real problems in the way companies are managed and in his model the losers prefer not to work as long as they can get paid. Conclusion I don t have any good solutions to these problems to offer. It seems that the best we can hope for is incremental change to make work less depressing, to have the minimal amount of management, and to avoid bullshit jobs .
Related posts:
  1. IT Jobs and Working Conditions Mark Glossop has written about the best designs for offices...
  2. Preferring Low Quality and Microsoft Software Is Low Quality in Italian Academia related to the choice...
  3. university degrees Recently someone asked me for advice on what they can...

2 December 2013

Tim Retout: Questhub.io

At the London Perl Workshop last Saturday, one of the lightning talks was about Questhub.io, formerly known as "play-perl.org". It's social gamification for your task list, or something like that. Buzzword-tastic! But most importantly, there seems to be a nice community of programming types to procrastinate with you on your quests. This means I can finally get to work refuting lamby's prediction about gamification of Debian development! Tasks are referred to as "Quests", and are pursued in themed "Realms", for that World of Warcraft feeling. For example, there's a "Perl" realm, and a "Lisp" realm, and a "Haskell" realm, but also non-programming realms like "Fitness" and "Japanese". Of course, part of me now wants to construct a federated version which can be self-hosted. :) Another downside of questhub currently is the lack of SSL support - your session cookies are sent in plain text. I hope this changes soon.

30 November 2013

Gunnar Wolf: On errors in exams - Short rant

Blogging from a phone... first time ever. I don't want to forget some specifics for this :) I have just completed an exam to try to enter a postgraduate program (I'll talk more about it once it becomes real ). The exam is administered by CENEVAL, the same evaluation agency Where I presented my graduation equivalency exam some years ago - Only this exam is for all of the postgraduate studies on many national universities and is thus basically just a psychometric test. The exam had 162 questions, all to be filled in a optical reader sheet, on five subjects: mathematical reasoning, Spanish grammar and comprehension, Project management, Computers and technology, And English reading and understanding. It was all in all a fun exam to take, mostly due to the math reasoning part. But... on the Subject I Know I am an expert, I have to complain (and intend to find a easy to do so formally). First, I spotted two absolute mistakes (and answered based on What I knew others would, but knowing the answer is wrong technically). One was a subtlety, on how and why have hard drives should be defragmented (and part of my quip is that it's an obsolete habit, but besides, the answers were all erroneous), but a second one was... just wrong. It asked on what should not be part of an "Internet link" (can only guess they meant An URL). The 4 options were valid parts of a URL - including one very seldom used by most people, but very often by many of us: the @ sign. Anyway, answered it, but my other main gripe is that most of the section was in specific use of Office software. Not only In Office-like, which would be bad enough to begin with, but on specific ways of using Mainly Excel And PowerPoint. Syntax issues, or the name of the menu under which to look for specific functions.
Anyway, I will wait the stipulated 10 days for the exam to be rated, but will anyway look die a way to contact the very opaque and secretive CENEVAL. Not to demand to be better treated, but to try to correct those known mistakes and errors.

26 November 2013

Daniel Pocock: Real-time streaming market data with free software

Here is an overview of the real-time market data architecture based on OpenMAMA: All these components may be running on a single host or they may be distributed across different servers and workstations in a LAN. The OpenMAMA market data bus ties them together. External data sources These are external sources of data. These could be currency trading firms, stock brokers, bullion dealers or even Bitcoin exchanges. Some exchanges don't provide data services directly to the general public and they distribute their price data through third-party data vendors such as Reuters and Bloomberg. Feed handlers The feed handlers manage the connections to external data sources. These are typically daemon processes that implement the wire protocols used by the external vendors for transmitting data over the public internet or a leased line. They typically use a point-to-point topology such as TCP connections. The feed handlers publish the data into the market data bus. In practice, the feed handler may be a proprietary application provided by the data vendor or it may simply be a Python script that fetches exchange rates from a URL every five minutes. OpenMAMA The market data bus is a distributed framework that is accessible to all of the local servers and workstations in the LAN. At the lowest level a messaging middleware solution is used to transport the data. The Avis Event Router is a free middleware. There is upcoming support for Apache QPID as well. Some commcercial middleware is supported too - here is the full list OpenMAMA does not provide a daemon or server process of its own. It is a set of libraries that operate on top of the middleware transport. In very general terms, the OpenMAMA libraries let applications publish messages (such as price ticks) or subscribe to receive the messages published by other applications. Spreadsheet application (for example, LibreOffice) LibreOffice is introducing a new data import feature for streaming time series data. This appears to provide a useful integration point for OpenMAMA and discussion is taking place in the development community. The spreadsheet can operate in various ways. A simple application may simply poll the currency prices when the spreadsheet is opened or refreshed. A more demanding application may see the spreadsheet come alive, recalculating all cells on every tick from the data vendor (several times per second). This latter scenario is more common in dealing rooms and hedge funds. Accounting/billing software There are many free accounting applications today, including PostBooks and GnuCash and some more heavyweight solutions like Adempiere and OpenERP. In a world where credit cards and the world wide web have made international trade an everyday activity, many people are using this software to track expenses and accounts in more than one currency. For example, a British business may be paying some suppliers in Euros and a Canadian may be charging some customers in US dollars. A Swiss person may be keeping some of their savings in gold bullion in the vault of one of their world reknowned Swiss banks. A common requirement for all of these users is the balance sheet. Using real-time market data sources, the balance sheet can be refreshed at any time from the live market prices. A home user may just find it convenient that they can open the balance sheet and always see their net worth immediately without having to manually cut and paste the currency prices from a web page. A business user operating in a competitive industry with low margins may be checking the balance sheet several times per day to ensure they remain solvent and profitable. Web shops Many web shops now offer the customers the convenience of viewing prices in their own currency. It is important to make sure these prices are accurate, especially when dealing with volatile currencies or when the products have a low profit margin. The web application server can receive the market data price ticks just like any other application in the network. Using this data, it can insert live prices into the pages as they are generated. If the currencies are particularly volatile, the web application could stream the prices to the browser using WebSockets and a JavaScript client embedded in the page would update the price field in real time. In fact, this web-based streaming price update mechanism is exactly how many online financial trading services offer live market prices to their customers. Real-time valuation server In a large organisation there may be many users looking at the same values. For example, in a trading desk, many users may need to see the desk's overall position in each market. In a busy web shop, many concurrent users may need to see the prices of common products. Rather than recalculating these values for each user in parallel, a common solution involves setting up a server to receive raw values (such as currency prices) from the data feed, calculate values needed by local users and then broadcast those values over the market data bus. How applications can integrate with real-time market data Here are some steps for getting started:

Daniel Pocock: Real-time streaming market data with free software

Here is an overview of the real-time market data architecture based on OpenMAMA: All these components may be running on a single host or they may be distributed across different servers and workstations in a LAN. The OpenMAMA market data bus ties them together. External data sources These are external sources of data. These could be currency trading firms, stock brokers, bullion dealers or even Bitcoin exchanges. Some exchanges don't provide data services directly to the general public and they distribute their price data through third-party data vendors such as Reuters and Bloomberg. Feed handlers The feed handlers manage the connections to external data sources. These are typically daemon processes that implement the wire protocols used by the external vendors for transmitting data over the public internet or a leased line. They typically use a point-to-point topology such as TCP connections. The feed handlers publish the data into the market data bus. In practice, the feed handler may be a proprietary application provided by the data vendor or it may simply be a Python script that fetches exchange rates from a URL every five minutes. OpenMAMA The market data bus is a distributed framework that is accessible to all of the local servers and workstations in the LAN. At the lowest level a messaging middleware solution is used to transport the data. The Avis Event Router is a free middleware. There is upcoming support for Apache QPID as well. Some commcercial middleware is supported too - here is the full list OpenMAMA does not provide a daemon or server process of its own. It is a set of libraries that operate on top of the middleware transport. In very general terms, the OpenMAMA libraries let applications publish messages (such as price ticks) or subscribe to receive the messages published by other applications. Spreadsheet application (for example, LibreOffice) LibreOffice is introducing a new data import feature for streaming time series data. This appears to provide a useful integration point for OpenMAMA and discussion is taking place in the development community. The spreadsheet can operate in various ways. A simple application may simply poll the currency prices when the spreadsheet is opened or refreshed. A more demanding application may see the spreadsheet come alive, recalculating all cells on every tick from the data vendor (several times per second). This latter scenario is more common in dealing rooms and hedge funds. Accounting/billing software There are many free accounting applications today, including PostBooks and GnuCash and some more heavyweight solutions like Adempiere and OpenERP. In a world where credit cards and the world wide web have made international trade an everyday activity, many people are using this software to track expenses and accounts in more than one currency. For example, a British business may be paying some suppliers in Euros and a Canadian may be charging some customers in US dollars. A Swiss person may be keeping some of their savings in gold bullion in the vault of one of their world reknowned Swiss banks. A common requirement for all of these users is the balance sheet. Using real-time market data sources, the balance sheet can be refreshed at any time from the live market prices. A home user may just find it convenient that they can open the balance sheet and always see their net worth immediately without having to manually cut and paste the currency prices from a web page. A business user operating in a competitive industry with low margins may be checking the balance sheet several times per day to ensure they remain solvent and profitable. Web shops Many web shops now offer the customers the convenience of viewing prices in their own currency. It is important to make sure these prices are accurate, especially when dealing with volatile currencies or when the products have a low profit margin. The web application server can receive the market data price ticks just like any other application in the network. Using this data, it can insert live prices into the pages as they are generated. If the currencies are particularly volatile, the web application could stream the prices to the browser using WebSockets and a JavaScript client embedded in the page would update the price field in real time. In fact, this web-based streaming price update mechanism is exactly how many online financial trading services offer live market prices to their customers. Real-time valuation server In a large organisation there may be many users looking at the same values. For example, in a trading desk, many users may need to see the desk's overall position in each market. In a busy web shop, many concurrent users may need to see the prices of common products. Rather than recalculating these values for each user in parallel, a common solution involves setting up a server to receive raw values (such as currency prices) from the data feed, calculate values needed by local users and then broadcast those values over the market data bus. How applications can integrate with real-time market data Here are some steps for getting started:

Daniel Pocock: The free software country?

Loch Ness Monster, Scotland There is big news in Europe right now, especially the UK, about the Scottish nation's manifesto for independence. I thought this was really amazing, a country that is going to move to free software and protect its citizens from horrors like DRM. I couldn't wait to find out more, for example, will they use Linux or *BSD? After all, how many nations today can really consider themselves independent when they are trapped in the use of complex and opaque systems that require ongoing royalty payments to a foreign corporation, much like ancient villages paying their tithes to Rome? Today it is actually worse: these independent nations are not just sacrificing their cash - the new Rome is also using those same secret, proprietary systems to raid their subjects' privacy. So I went to the manifesto's web site to get the details. I couldn't even find the words "software" or "technology" listed in the policy index on the front page - and software doesn't even appear in the search results. A section on Culture, Communications and Digital only really looks at the issues for the media (BBC after independence). They don't even make any comment about the British Telecom (BT) pension crisis and whether Scotland will abandon BT's services, leaving only those customers south of the border to continue subsidising the deficit through higher phone bills. There is some hope, with the dot scot (.scot) top level domain coming in 2015 regardless of the referendum outcome. Does this mean an independent Scotland is going to be run without technology at all then? That could be an interesting way to avoid the type of infiltration of communications technology that has been taking place in Brussels. Or do the people calling the shots simply fail to realise the impact of software and communications technology on a modern day concept of independence? Party date set For those who just care about gatecrashing the post-independence party, start booking your leave for March 2016.

23 November 2013

Steve Kemp: Things have settled down nicely

I've now completed all my KVM migrations. Moving my personal virtual machines from one host to another. There were a few niggles, for example I didn't have a working IPv6 allocation at the time I moved things so I had to set that up post-migration. I've also joined each of the hosts into a VPN which makes cross-guest communication secure and simple. Finally I've overhauled my firewalls and service lists. I installed a couple of extra guests, using libvirt and booting from the Debian ISO. The Debian installer continues to impress, though it did make me think I should overhaul my PXE setup at home. It wouldn't be hard to have a Raspberry PI running as a TFTP + DHCP server. You could plug it into a network, reboot your desktop, and then have it boot into the imager. At the moment I run DHCP + TFTPD + etc on my main desktop, and that allows me to reimage any of the hosts in the flat easily, except itself obviously. The last time I reinstalled this system I had to reconfigure DHCP + PXE + TFTP on another host. I think the next time I need to reinstall any system I'll "waste" an SD-card on an image-server host. Finally I've recently read the Rick Cook Wizardy Series: Fun idea. Horrible puns. Some of the books were too long, or left plot elements dangling, but on average they were more good than bad. Albeit a little predictable and "simple".

22 November 2013

Iustin Pop: Z rich at night

Last weekend I realised that one of the advantages of winter is that it gets dark quite early - so I took a short walk through Z rich at night to take a set of pictures. I usually only shoot handheld at night, which has its limitations, and this time I took a tripod with me. It's a very cheap tripod which I have for many years now and I don't really use - and the test was to see if it's actually usable for long exposures. The results can be seen on my Smugmug gallery. Some of the pictures I really like, but that's not the main point - I learned a lot in just an hour or so. First, the tripod worked. Well, it's cheap and plasticky, so "mirror delay 3" was absolutely needed, but with that the pictures got out reasonably sharp, even at 30s exposure. Second, I forgot my cable release at home, even though I explicitly wanted to take it, so "mirror delay 3" became my remote release. Logistics aside, the most interesting thing was that in the dark, the enemy is light, and not darkness. In the city, the contrast between darkness and strong lights results in a large dynamic range requirement, and - to me - that was a surprising realisation. As an example, in this picture, I had to use f/22 to get the desired exposure time, as the light was too strong. Another surprising realisation was that, if you're shooting with 30s exposure time, a person passing in front of the camera (but not staying in the same place) will leave no trace. At shorter exposures, or when the person has a low transversal speed, then it will show, but otherwise, you don't have to care too much . Something that I knew but it was not nice to be reminded of is how "broken" auto white balance is in presence of artificial lighting (basically low temperature). Add the fact that in a non-trivial scene you have multiple lights with different temperatures makes any night scene interesting - I had to manually fix white balance for the entire album, and I don't think I got it right. And finally - light pollution. The sky above Z rich was cloudy, and the clouds were glowing orange/reddish. I can't wait to get somewhere a bit far from civilisation with a nice, dark sky

Vincent Sanders: Error analysis is the sweet spot for improvement

Although Don Norman was discussing designers attitude to user errors I assert the same is true for programmers when we use static program analysis tools.

The errors, or rather defects in the jargon, that a static analysis tools produce can be considered low cost well formed bug reports available very early in the development process.

When I say low cost it is because they can be found by a machine without a user or fellow developer wasting their time finding them. Well formed comes because the machine can describe exactly how it came to the logical deduction leading to the defect.
IntroductionStatic analysis is in general terms using the computer to examine a program for logic errors beyond those of pure syntax before it is executed. Examining a running program for defects is known as dynamic program analysis and while a powerful tool in its own right is not the topic of discussion.

This analysis has historically been confined to compiled languages as their compilers already had the Abstract Syntax Tree (AST) of the code available for analysis. As an example the C language (released in 1972) had the lint tool (released in 1979) based on the PCC compiler.

Practical early compilers (I am generalising here as the 19070s were a time of white hot innovation in computing and examples of just about any innovation in the field could probably be found) were pretty primitive and produced executables which were less good than hand written assembler output. Due to practical constraints the progress of optimising compilers was not as rapid as might be desired so static analysis was largely used as an external process.

Before progressing I ought to explain why I just mixed the concept of an optimising compiler and static analysis. The act of optimisation within those compilers requires program analysis, from which they can generate defect reports which we all know and love as compiler warnings, also explaining why many warnings only appear at higher optimisation levels where deeper analysis is required.

The attentive reader may now enquire as to why we would need external analysis tools when our compilers already perform the task. The answer stems from the issue that a compiler is trying to reconcile many desirable traits including:
The slow progress in creating optimising compilers initially centred around the problem of getting the compiled output in a reasonable time to allow for a practical edit-compile-run-debug cycle although the issues more recently have moved more towards the compiler implementation costs.

Because the output generation time is still a significant factor compilers limit the level of static analysis performed to that strictly required to produce good output. In standard operation optimising compilers do not do the extended analysis necessary to find all the defects that might be detectable.

An example: compiling one 200,000 line C program with the clang (v3.3) compiler producing x86 instruction binaries at optimisation level 2 takes 70 seconds but using the clang based scan-build static analysis tool took 517 seconds or more than seven times as long.
Using static analysis
As already described warnings are a by-product of an optimising compilers analysis and most good programmers will endeavour to remove all warnings from a project. Thus almost all programmers are already using static analysis to some degree.

The external analysis tools available can produce many more defect reports than the compiler alone as long as the developer is prepared to wait for the output. Because of this delay static analysis is often done outside the usual developers cycle and often integrated into a projects Continuous Integration (CI) system.

The resulting defects are usually presented as annotated source code with a numbered list of logical steps which shows how the defect can present. For example the steps might highlight where a line of code allocates memory from the heap and then an exit path where no reference to the allocated memory is kept resulting in a resource leak.

Once the analysis has been performed and a list of defects generated the main problem with this technology rears its ugly head, that of so called "false positives". The analysis is fundamentally an undecidable problem (it is a variation of the halting problem) and relies on algorithms to generate approximate solutions. Because of this some of the identified defects are erroneous.

The level of erroneous defect reports varies depending on the codebase being analysed and how good the analysis tool being used is. It is not uncommon to see false positive rates, even with the best tools, in excess of 10%

Good tools allow for this and provide ways to supply additional context through model files or hints in the source code to suppress the incorrect defect reports. This is analogous to using asserts to explicitly constrain variable values or a type cast to suppress a type warning.

Even once the false positives have been dealt with there comes the problem of defects which while they may be theoretically possible take so many steps to achieve that their probability is remote at best. These defects are often better categorized as a missing constraint and the better analysis tools generate fewer than the more naive implementations.

An issue with some defect reports is that often defects will appear in a small number of modules within programs, generally where the developers already know the code is of poor quality, thus not adding useful knowledge about a project.

As with all code quality tools static analysis can be helpful but is not a panacea code may be completely defect free but still fail to function correctly.
Defect DensityA term that is often used as a metric for code quality is the defect density. This is nothing more than the ratio of defect to thousands of lines of code e.g. a defect density of 0.9 means that there is approximately one defect found in every 1100 lines of code.

The often quoted industry average defect density value is 1, as with all software metrics this can be a useful indicator but should not be used without understanding.

The value will be affected by improvements in the tool as well as how lines of code are counted so is exceptionally susceptible to gaming and long term trends must be treated with scepticism.
Practical examplesI have integrated two distinct static analysis tools into the development workflow for the NetSurf project which I shall present as case studies. These examples show a good open source solution and a commercial offering highlighting the issues with each.

Several other solutions, both open source and commercial, exist many of which have been examined and discarded as either impractical or proving less useful than those selected. However the investigation was not comprehensive and only considered what was practical for the project at the time.
clangThe clang project is a frontend to the LLVM project providing an optimising compiler for the C, C++ and objective C languages. As part of this project the compiler has been enhanced to run a collection of "checkers" which implement various methods of analysis on the code being compiled.

The "scan-build" tool is provided to make the using these features straightforward. This tool generates defect reports as a series of html files which show the analysis results.


NetSurf CI system scan-build overview
Because the scan-build takes in excess of eight minutes on powerful hardware the NetSurf developers are not going to run this tool themselves as a matter of course. To get the useful output without the downsides it was decided to integrate the scan into the CI system code quality checks.

NetSurf CI system scan-build result list
Whenever a git commit happens to the mainline branch and the standard check build completes successfully on all target architectures the scan is performed and the results are published as a list of defects.

The list is accessible directly through the CI interface and also incorporates a trend graph showing how many defects were detected in each build.

A scan-build report showing an extremely unlikely path to a defect
Each defect listed has a detail link which reveals the full analysis and logic necessary to cause the defect to occur.

Unfortunately even NetSurf which is a relatively small piece of software (around 200,000 lines of code at time of writing) causes 107 defects to be emitted by scan-build.

All but 25 of the defects are however "Dead Store" where the code has a value assigned but is never checked. These errors are simply not interesting to the developers and are occurring in code generated by a tool.

Of the remaining defects identified the majority are false positives and several (like the example in the image above) are simply improbable requiring a large number of steps to reach.

This shows up the main problem with the scan-build tool in that there is no way to suppress certain checks, mark defects as erroneous or avoid false positives using a model file. This reduces the usefulness of these builds because the developers all need to remember that this list of defects is not relevant.

Most of the NetSurf developers know that the project currently has 107 outstanding issues and if a code change or tool improvement were to change that value we have to manually work through the defect list one by one to check what had changed.
CoverityThe coverity SAVE tool is a commercial offering from a company founded in the Computer Systems Laboratory at Stanford University in Palo Alto, California. The results of the original novel research has produced a good solution which improved on analysis tools previously available.

Coverity Interface showing summary of NetSurf analysis. Layout issues are a NetSurf bug
The company hosts a gratis service for open source projects, they even provide scans for the Linux kernel so project size does not appear to be an issue.

The challenges faced integrating the coverity tool into the build process differed from clang however the issue of execution time remained and the CI service was used.

The coverity scanning tool is a binary executable which collects data on the build which is then submitted to the coverity service to be analysed. This tool obviously relies upon the developer running the executable to trust coverity to some degree.

A basic examination of the binary was performed and determined the executable was not establishing network connections or performing and observably undesirable behaviour. From this investigation the decision was made that running the tool inside a sandbox environment on a CI build slave was safe. The CI system also submits the collected results in a compressed form directly to the coverity scan service.

Care must be taken to only submit builds according to the services Acceptable Use Policy which limits the submission frequency of NetSurf scans to every other day. To ensure the project stays within the rules the build performed by the CI system is manually controlled and confined to a subset of NetSurf developers.

Coverity connect defect management console for NetSurfThe results are presented using the coverity connect web technology based defect management tool. Access to the coverity connect interface is controlled by a user management system which precludes publicly publishing the results within the CI system.

Unfortunately NetSurf itself does not currently have good enough JavaScript DOM bindings to support this interface so another browser must be used to view it.

Despite the drawbacks the quality of the analysis results is greatly superior to the clang solution. The false positive rate is very low while finding many real issues which had not been previously detected.

The analysis can be enhanced by use of collection configuration and modelling files which remove intended constructions from consideration reducing the false positive rate to very low levels. The ability to easily and persistently suppress false positives through the web interface is also available.

The false positive management capabilities coupled with a user interface that makes understanding the defect path simple make this solution very practical and indeed the NetSurf developers have removed over 50 actual issues within a relatively short period since the introduction of the tool.

Not all of those defects could be considered serious but they had the effect of encouraging deeper inspection of some very dubious smelling source.
ConclusionsThe principle conclusions of implementing and using static analysis have been:

When I started looking at this technology I was somewhat dubious about its usefulness but I have definitely changed my mind. It is a useful addition to any non-trivial project and the return on time and effort should be repaid handsomely in all but already perfect code (if you believe you have such code I have a bridge to sell you).

19 November 2013

Carl Chenet: Yarn: a scenario testing tool

Follow me also on Twitter Those who follow me on Twitter may have read my recent tweet about Yarn, a really nice software by Lars Wirzenius I discovered the last week-end while attending the Mini-Debconf UK in Cambridge. To make a long story short, Yarn allows the user to write scenarios of the executions of softwares. This scenario will be automatically played by Yarn which also returns the overview of the execution, with the goal to test the behaviour of these softwares. 1. Small example of how to use Yarn Let s have a look of how to use Yarn with a real example. The scenario will test if the software Brebis, a fully automatized backup checker, executes in an expected way. We can proceed as follow :
  1. set up the test environment
  2. launch a command with the software under test in the test environment
  3. verify the execution of the test
This is an example of a Yarn scenario file, the file is formatted using the Markdown syntax: 
    SCENARIO basic brebis execution
    GIVEN setting up brebis
    AND generating backup configuration with brebis
    WHEN brebis verifies a backup
    THEN verify brebis output
    IMPLEMENTS GIVEN setting up brebis
    hg clone http://hg.brebisproject.org $DATADIR/brebis
    mkdir -p $DATADIR/brebis/yarn-test
    cp $DATADIR/brebis/functional-tests/expected-generated-list-for-tar-archive/expected-generated-list-for-tar-archive.tar.gz $DATADIR/brebis/yarn-test
    IMPLEMENTS GIVEN generating backup configuration with brebis
    $DATADIR/brebis/brebis.py -G $DATADIR/brebis/yarn-test/expected-generated-list-for-tar-archive.tar.gz
    IMPLEMENTS WHEN brebis verifies a backup
    $DATADIR/brebis/brebis.py -c $DATADIR/brebis/yarn-test -l $DATADIR/brebis/yarn-test/brebis.log
    IMPLEMENTS THEN verify brebis output
    if [ -s $DATADIR/brebis/yarn-test/brebis.log ]; then return 1; else return 0; fi
2. Result of the execution of the Yarn scenario Yarn provides at the end of the scenario an overview of the execution, as show below: $ yarn brebis-scenario
Scenario test suite PASS, with 1 scenarios (4 total steps), in 16.4 seconds 3. Details of the Yarn scenarios The five first lines are basically your scenario. Each step takes a keyword and a sentence to identify a step. You define a name with the SCENARIO keyword. The next step, GIVEN, allows generally the setup of the test environment. It is possible of course to define different steps with the keywork AND. Once the environment setup has been completed, we launch the test with the keyword WHEN. At last we check the result of the execution with THEN. It is worth noting that if you need further steps after checking the result of the execution, you can use FINALLY. Very helpful while setting up the test environment is the variable $DATADIR, automatically initialized by Yarn. It provides the path to a temporary directory where you can store files during the execution of your scenario. This directory is by default removed at the end of the scenario. Now the real trick: at each step I described above is associated shell commands to execute thanks to the IMPLEMENTS keyword.It s in my opinion a great way to increase the understanding of what the scenario really does. In a short glance of the first lines, if the names of the steps are carefully chosen, you will understand what the scenario does without needing to read the actual shell code. Simple and efficient. 4. More about Yarn Yarn is written in Python and now available in Debian in the package cmdtest. The sources are available here. In my opinion, if you were looking for this kind of tool, Yarn is in a really good direction. This project is still young but the core features do the job and the upstream is eager to receive feedbacks (and patches) :) Some links: Interested in Yarn? What do you think about it? Let us know in the comments of this post.

18 November 2013

Daniel Pocock: $75,000 to explore free software during maternity leave?

A number of countries offer women various social benefits during maternity leave. A new scheme in Australia promises to give virtually all women one hundred percent of their salary (capped at a whopping $12,500 per month) and extend maternity leave to 6 months. Some believe it has been funded by abolishing fibre-to-the-home, but that is another topic... This is a dramatic upgrade - in both cash and time - from the existing scheme where women have 18 weeks leave at the minimum wage. In Australia, the "minimum wage" figure is a fantasy, well below the real cost of living. In practice, the existing scheme and economic pressure force many women back into their job in less than 18 weeks. The new scheme with a one hundred percent salary replacement eliminates much of that pressure. Many European countries offer women some payment that pays the real cost of living, for example, EUR 2,400 per month in France. That is more than the minimum wage in Australia, enough to cover food, clothing and rent in most French cities. Australia's new system, in comparison, will give women on high salaries enough cash to immediately hire a nanny and other discretionary expenses that will maintain their lifestyle as much as they can within reason while dealing with a new baby. The cap is 300% higher than in France and it is 100% wage replacement, whereas many European schemes only give women a percentage of their salary. For this reason, many people have described it as the "Rolls Royce" of maternity pay. Just who are the women in these higher salary brackets? It is hardly any surprise that many of them are in IT and engineering professions. The upper salary range of the new maternity leave scheme appears to squarely match with the cashflows of female software developers, web designers, project managers and potentially many other professionals with a background in science or engineering. Many women worry about being out of the workforce for too long and the potential impact on their skills and professional networks. In computing jobs, technology can change a lot in 6 months. There are many men and women who enjoy computing in their free time as much as in their job. For female developers with this time out of the office, the world's most comfortable maternity leave scheme could be an opportunity to spend a little time exploring the free software world. Just to be clear, I'm not suggesting that women should be reaching for the laptop when they first go into labor and getting plugged in to the web before the umbilical cord has been severed. Some people find technology to be a rewarding pursuit throughout their whole life, while others simply see it as a job. For those who find free software interesting, some may simply pursue something creative that has no relation to the IT projects in their job, some may bring some new experiences back to their job. Others may choose to reconsider their career, performing free software development work on a freelance basis rather than going back to an office environment. The "Rolls Royce" maternity pay scheme could provide a comfortable buffer for them as they evaluate their next steps and make such decisions.

16 November 2013

Lars Wirzenius: Debian: developing it wrong

This is an essay form of a talk I have given today at the Cambridge Mini-debconf. The talk was videoed, so will presumably show up in the Debconf video archive eventually. Abstract Debian has a long and illustrious history. However, some of the things we do perhaps no longer make as much sense as they used to do. It's a new millennium, and we might find better ways of doing things. Things that used to be difficult might now be easy, if we dare look at things from a fresh perspective. I have been doing that at work, for the Baserock project, and this talk is a compilation of observations based on what I've learnt, concentrating on things that affect the development workflow of package maintainers. Introduction and background I have been a Debian developer since August, 1996 (modulo a couple of retirements), and have used it a little bit longer. I have done a variety of things for Debian, from maintaining PGP 2 packages to writing piuparts to blathering excessively on the mailing lists. My day job is to develop the Baserock system at Codethink. Baserock is a set of tools and workflows for developing embedded and appliance Linux systems. If you squint, it looks a bit like a source-based Linux distribution. I have worked on Baserock since September, 2011. Some of Baserock's design has been influenced by my experience with Debian. With Baserock I have the chance to fix all the things that are wrong in Debian, and this talk is me giving back to Debian by pointing out some the things I feel should be fixed. I don't have solutions for these problems: this is a bug report, not a patch. It's also perhaps a bit of a rant. I am specifically concentrating here on technical and tooling issues that affect the development process of Debian. I excluding social issues. I am also not trying to get Debian to switch to Baserock. Baserock is targeting embedded and appliance systems, and makes simplifying assumptions based on those targets, which Debian does not get to do. I am pointing out problems, and I am outlining solutions as implemented in Baserock, when I think the concept carries over well to Debian. Build tools should be intelligent, packaging should be dumb In Debian, the fundamental tool for compiling the upstream code and assembling a binary package is dpkg-buildpackage. It uses debian/rules, which is a Makefile with targets with specific names and semantics. By executing the right targets in the right order, dpkg-buildpackage tells the source package to build the binary package. On the one hand, this is a nice design, because it abstracts away the large variety of upstream build systems into one API for dpkg-buildpackage to use. On the other hand, it puts all the intelligence for how packages are built into the packages. Making packaging intelligent, rather than the build tool, means packagers need to do more work, and there's more work to do when, say, the Debian policy changes, or when there are other changes that affect a large number of packages. If packaging is intelligent, then every package needs changes. If the build tool is intelligent, then you change the build tool and re-build everything. In Baserock we put as much intelligence as we can into the Morph tool, which drives the build process. It turns out that today, unlike in 1995, most upstream projects use one of a handful well-known build systems: autotools, cpan, cmake, Python distutils, etc. With just a little extra logic in Morph we avoid having any delta against upstream in Baserock. This doesn't work for quite all upstream projects, of course, but we've spent the 2% (two, not twenty) of effort that solves 80% of the problem. In recent years, the dh approach to packaging has made a lot of packages be able to have only a minimal, 3-line debian/rules file. This is excellent. Wouldn't it be nice if even that wasn't needed? It'd save thousands of files in source packages across the archive, at least. It would be easy to: if the file is missing, have dpkg-buildpackage assume the 3-line version by default. Getting rid of a single file is, of course, not a particularly big win. The big win is the change in mindset: rather than dealing with all new issues in development by adding yet more fields to debian/control and more optional, competing tooling outside the core toolset, if you improve the tools everyone uses, then everyone's packages gets better. The goal should, in my opinion, be that for the large number of packages where upstream uses a well-known, well-behaved build system, and uses it in a reasonably sensible way, the Debian source package should not require anything added to make the package build. There will still be a need to add some stuff, such as the debian/copyright file, to make a good Debian package, but just getting the package to build should require nothing extra. (Side note: wouldn't it be nice if there was a well-known, widely used way to mark up copyright information so that debian/copyright could be constructed and updated automatically?) Configuration file handling on upgrades: add ucf to dpkg already In the 1990s, dpkg had excellent handling of configuration files and merging local changes with changes from the new package version, but it was excellent only because it tried to do that at all, and mostly nothing else did. It hasn't changed much, since, and it's not excellent on any absolute scale. We have the ucf tool, which can do a better job, but requires to be added to each package wanting to use it. Why don't we make dpkg smarter, instead? If ucf is not not good enough to be merged wholesale into dpkg, let's write something better. Making every package maintainer use ucf manually is just wasteful. This is not the kind of thing that should be done for each package separately: the package manager should be smart so that packaging can be stupid. The goal should be that dpkg is smart enough in its configuration file handling that having the package do it is a very rare special case. Clean building shouldn't be hard The basic tool for building a package is dpkg-buildpackage. It is ever so slightly cumbersome to use, so there are some wrappers, most importantly debuild. However, if you're making a build intended to be uploaded to the Debian archive, you should be doing a clean build. This means having to learn, configure, and use yet more tools. A clean build is important: security updates, development, debugging, quality control, porting, and user support become more difficult if we don't know how a package was built, and can't reproduce the build. It gets harder to keep build dependencies correct, making it harder for everyone to build things. Luckily, Debian has solved the clean build problem. Unluckily, it has solved it multiple times, creating the problem of having to choose the clean building approach you want to use. The default way of building is not clean, so then you have to remember to use the non-standard way. You also get to spend time maintaining the clean build environments yourself, since that doesn't seem to be fully automated. None of this is hard, as such, but it's extra friction in the development workflow. The primary approaches for clean building in Debian seem to be pbuilder, cowbuilder, and sbuild. I happen to use pbuilder myself, because it's what's been around the longest, but I make no claim of having made an informed choice. That is part of the problem here: why should I have to spend the effort to become informed to make a choice well? Why is the default way of building not clean? Don't say performance: Morph sets up a clean staging area in seconds, and does not offer you a choice of not doing so. It's a chroot, with everything but the build tree hardlinked from cached, unpacked build dependencies, and protected using read-only bind mounts. What's more, this approach avoids having to maintain the pbuilder base tarballs or sbuild chroots manually. It's all automatic, and up to date, for every build. Furthermore, the staging area contains only the specified build dependencies, and not anything else, meaning a build fails if a build dependency is missing, rather than succeeding because it happens to be in the default set of packages the build tool installs. Mass building shouldn't be hard Suppose you want to try out a large-scale change in Debian. It might be trying out a new version of GCC, or using llvm's clang as the default C compiler, or updating glibc, or doing a large library transition such as a new version of GTK+, or trying a new dpkg version, or even something more exploratory such as changing the default optimisation flags for all packages to see what breaks. All of these will require you to at least rebuild all affected packages. Ideally you'd test the built packages as well, but let's concentrate on the building for now. Here's what you do. You make the changes you want to try out, and build those packages. You create a dedicated APT repository, and upload your packages there. You configure your build environment to favour that APT repository. You make a list of all reverse build dependencies of the packages you changed. You write a script to build all of those, preferably so that if you change A, and rebuild B, then you also rebuild C which build depends on B. Each rebuilt package you also upload to your APT repository. You keep track of the build log, and success or failure, of each build. The people in Debian who do this kind of stuff regularly presumably have tools for doing it. It shouldn't be a rare, special thing, though. If my package has reverse build dependencies, I should at least consider testing building when I'm making changes. Otherwise, it might take years until the reverse build dependencies are rebuilt, and the problem is only found then, making it harder to fix. To be fair, building a lot of packages takes a lot of resources. It's not feasible to rebuild everything in Debian every time there's any change to, say, eglibc. However, it's feasible to do it, for large subsets of the archive, without huge hardware investments. One VCS to rule them all In 1995 there was really only one relevant version control system: CVS. It was not a great tool. In 2000, another contender existed: Subversion. It fixed some problems in CVS, but still wasn't a great tool. In 2005, there was a great upheaval and distributed version control systems started to become mainstream. There were a large handful of them. In 2010, it was becoming pretty clear that git had won. It's ugly, but it's powerful. I'm not going to debate the relative merits of different version control systems. Until recently, I was a Bazaar boy, and all of my personal projects were kept in Bazaar. (I have recently switched everything to git.) There are, however, strong benefits from everyone using the same system. Developers don't need to learn a dozen version control systems. Tools that operate on many repositories are easier to write and maintain. Workflows become simpler if one system can be assumed. Debian has a strong historical tendency to choose every option. This is sometimes a good thing, and sometimes a bad thing. For keeping source packages in version control I believe it is to be a bad thing. The status quo is that a Debian source package may not be in version control at all, or it might be in any version control system. This is acceptable when everyone only ever needs to maintain their own packages. However, in a distribution the size of Debian, that is not the case. NMUs, security support, archive wide transitions, and other situations arise when I might need to change yours, or you might need to change mine. We try to work around this by having a complicated source package format, using quilt to maintain patches to upstream semi-manually in a debian/patches directory. This is an awkward workflow. It's a workflow that trips up those that are not used to it. (I know quilt is a patch management system, not a version control system. I think git does it much better anyway.) It would be oh so much easier if everyone kept their source packages in the same, real version control system. I don't even care what it is, as long as it is powerful enough to handle the use cases we have. Imagine a world where every Debian source package is kept, for argument's sake, git, and everyone also uses the same layout and roughly the same workflow to maintain it. What would this mean? It would mean that if you want to inspect the history of your package, you know how to do that. If you want to merge in some bugfix from upstream code, you know how to do that, without having to figure out which of the several source package formats are in use. It would make feasible the development of more powerful, more higher level tooling. For example, it would allow Debian to have what we call system branches in Baserock. In Debian we have stable, testing, unstable, and experimental. We may get something like Ubuntu's PPAs, or perhaps an improved version of those. These are very poor versions of system branches, just like quilt is a poor way to manage patches and deltas against upstream. For example, you can upload an experimental version of gccto experimental, but then nobody else can upload another experimental version. You can set up your own PPA for this, but you'll still be affected by all the uploads to unstable while you're working. A Baserock system branch is a branch of the entire system, or the entire distribution in the case of Debian. It is isolated from other system branches. In branch in an individual repository is a well-known concept. A system branch is conceptually like branching every repository in the distribution at once. The actual implementation is more efficient, of course. This would be possible to implement without standardising on one version control system, but it would be much harder to implement, and would have to live with the lowest common denominator for features. CVS and Subversion, for example, don't really to merges, where Bazaar, Mercurial, and git do. Possible does not mean feasible. Any work you do in a system branch is isolated. Your work doesn't affect others, and theirs doesn't affect yours, until a merge happens. This is a simple, but very powerful tool. Cheap system branches, and powerful merging, makes it possible to do experiments safely, with little fuss. Combine that with being able to build everything cleanly and quickly, and get into a situation where there's no need to have make technical decisions based on arguments on mailing lists, and instead they can be done by choosing by looking at working code. I don't know how this could be implemented in Debian, but think about it. If Debian could have this, it might many archive-scale changes easier. debian/rules clean: really? One of the silliest things we require of packages is that they have a debian/rules clean rule that cleans up after a build perfectly so that we can do repeated builds in the same source tree. Let's just use git clean -fdx instead. This is a problem that is superbly well suited for automation. There is no point whatsover making packagers do any manual work for this. Large scale semi-mechanical changes require too much effort About a decade ago, we decided to follow a new version of the Filesystem Hierarchy Standard and transition from /usr/doc to /usr/share/doc. This was an almost entirely mechanical change: in many cases, a mere rebuild would fix it, and in almost every other case it was just a minor tweak to the packaging. A one-line change. It took us seven years to do this. Seven years. Think about it. In a recent discussion about building R binary data files from source at package build time it was suggested that we take 2-3 release cycles to get this done. That's four to six years. Think about it. These are not isolated cases. Every time we need to make a change that affects more than a small handful of packages, it becomes a major undertaking. Most of the time all the people involved in this are agreeable to the change, and welcome it. The change takes long because it requires co-ordinating a large number of people. Sometimes people are just busy. Sometimes they've left the project, but haven't properly orphaned their packages. Waiting for a timeout for answers about packages drags the process longer. Mechanical changes, or semi-mechanical ones, which are very easy and very quick to do should not take years. They should take an evening, no more. There's no end of changes we might want to do like this. In 2005 and 2006 I filed a few hundred bugs from failed piuparts runs. They're still being fixed, even when the fix is simple, such as adding a postrm script to remove, when the package is purged, a configuration file created by postinst, or starting the service with invoke-rc.d rather than running an init.d script directly. Mostly one-line changes. There's more mechanical changes that might happen. For example, changing the value of the VCS-Browser field when the Debian version control server changes domain names. It's not just about simple, mechanical changes, either. Transitions of important library packages, for example, which require changes to reverse dependencies due to a changed API are another source of pain. What should be a few evenings of build fixes can drag out to months of co-ordination. This is caused partly by technical issues, and partly due to social issues. The main social issue is that we have quite a strong ownership of packages, and NMUs are to be done carefully, and only in some cases. This is both good and bad, and I won't discuss that aspect now. The technical issues are that our tools are primarily intended for maintaining individual packages, rather than all of them together, making it harder to make even simple changes in large number of packages. In addition to easy mass building and system branches, as outlined above, large-scale changes would require testing tools so that you don't build the new package versions, but also test things automatically. Essentially, CI at the distribution level. Conclusion I've listed above a small variety of problems I see in the Debian development processes and tools. They're not the important part of this talk. The important part is that it we, the Debian developers, should look at our tools, and workflows, critically, and improve them when we can. Even small improvements are very useful, when they affect each of our twenty thousand source packages. The important change I argue for here is one in mindset, rather than any specific problem in any tool. We need a mindset of constant, incremental improvement for our tools.

13 November 2013

Andrea Veri: Configuring DNSSEC on your personal domain

Today I ll be working out how to properly configure DNSSEC on a BIND9 installation, I ll also make sure to give you all the needed instructions to properly verify if a specific domain is being correctly covered by DNSSEC itself. In addition to that a few more details will be provided about adding the relevant SSHFP s entries on your DNS zone files to be able to automatically verify the authenticity of your domain when connecting to it with SSH avoiding any possible MITM attack. First of all, let s create the Zone Signing Key (ZSK) which is the key that will be responsible to sign any other record on the zone file which is not a DNSKEY record itself: 
dnssec-keygen -a RSASHA1 -b 1024 -n ZONE gnome.org


Note: the dnssec-keygen binary file should be part of bind97 (RHEL 5) or bind (RHEL6) package according to yum whatprovides: RHEL 5: 
32:bind97-9.7.0-17.P2.el5_9.2.x86_64 : The Berkeley Internet Name Domain (BIND) DNS (Domain Name System) server
Repo : rhel-x86_64-server-5
Matched from:
Filename : /usr/sbin/dnssec-keygen


RHEL 6: 
32:bind-9.8.2-0.17.rc1.el6.3.x86_64 : The Berkeley Internet Name Domain (BIND) DNS (Domain Name System) server
Repo : rhel-x86_64-server-6
Matched from:
Filename : /usr/sbin/dnssec-keygen


Then, create the Key Signing Key (KSK), which will be used to sign all the DNSKEY records: 
dnssec-keygen -a RSASHA1 -b 2048 -n ZONE -f KSK gnome.org


Creating the above keys can take several minutes, when done copy the public keys to the zone file this way: 
cat Kgnome.org*.key >> gnome.org


When done you can clean out the useless bits from the zone file and just leave the DNSKEY records (which are not commented out as you will notice) An additional and cleaner way of accomplishing the above is to use the INCLUDE rule on the zone file itself as follows: 
$INCLUDE /srv/dnssec-keys/Kgnome.org+005+12345.key
$INCLUDE /srv/dnssec-keys/Kgnome.org+005+67890.key


Choosing which method to use is really up to you. Once that is done you can go ahead and sign the zone file. As of myself I m making use of the do-domain script taken from the Fedora Infrastructure Team s repositories. If you are going to use it yourself, make sure to adjust all the relevant variables to match your setup, especially keyspath, region_zones, template_zone, signed_zones and AREA. The do-domain script also checks your zone file through named-checkzone before signing it.
/me while editing the do-domains script with the preview of gnome-code-assistance!

/me while editing the do-domains script with the preview of gnome-code-assistance!

If instead you don t want to use the script above, you can sign the zone file manually in the following way: 
dnssec-signzone -K /path/to/your/dnssec/keys -e +3024000 -N INCREMENT gnome.org


By default, the above command will append .signed to the file name, you can modify that behaviour by appending the -f flag to the dnssec-signzone call. The -N INCREMENT will increment the serial number automatically making use of the RFC 1982 arithmetics while the -e flag will extend the zone signature end date from the default 30 days to 35. (this way we can safely run a monthly cron job that will sign the zone file automatically) You can make use of the following script to achieve the above: 
#!/bin/sh
SIGNZONE="/usr/sbin/dnssec-signzone"
DNSSEC_KEYS="/srv/dnssec-keys"
NAMEDCHROOT="/var/named/chroot"
ZONEFILES="gnome.org"
cd $NAMEDCHROOT
for ZONE in $ZONEFILES; do
$SIGNZONE -K $DNSSEC_KEYS -e +3024000 -f $ZONE.signed -N INCREMENT $ZONE
done
/sbin/service named reload


Once the zone file has been signed just make sure to include it on named.conf and restart named: 
zone "gnome.org"  
file "gnome.org.signed";
 ;


When you re done with that we should be moving ahead adding a DS record for our domain at our domain registrar. My example is taken from the known gandi.net registrar.
gandi

Gandi s DNSSEC interface

Select KSK (257) and (RSA/SHA-1) on the dropdown list and paste your public key on the box. You will find the public key you need on one of the Kgnome.org*.key files, you should look for the DNSKEY 257 entry as dig DNSKEY gnome.org shows: 
;; ANSWER SECTION:
gnome.org. 888 IN DNSKEY 257 3 5 AwEAAbRD7AymDFuKc2iXta7HXZMleMkUMwjOZTsn4f75ZUp0of8TJdlU DtFtqifEBnFcGJU5r+ZVvkBKQ0qDTTjayL54Nz56XGGoIBj6XxbG8Es+ VbZCg0RsetDk5EsxLst0egrvOXga27jbsJ+7Me3D5Xp1bkBnQMrXEXQ9 C43QfO2KUWJVljo1Bii3fTfnHSLRUsbRn8Puz+orK71qxs3G9mgGR6rm n91brkpfmHKr3S9Rbxq8iDRWDPiCaWkI7qfASdFk4TLV0gSVlA3OxyW9 TCkPZStZ5r/WRW2jhUY/kjHERQd4qX5dHAuYrjJSV99P6FfCFXoJ3ty5 s3fl1RZaTo8=


Once that is done you should have a fully covered DNSSEC domain, you can verify that this way: 
dig . DNSKEY   grep -Ev '^($ ;)' > root.keys
dig +sigchase +trusted-key=./root.keys gnome.org. A   cat -n


The result: 
105 ;; WE HAVE MATERIAL, WE NOW DO VALIDATION
106 ;; VERIFYING DS RRset for org. with DNSKEY:59085: success
107 ;; OK We found DNSKEY (or more) to validate the RRset
108 ;; Ok, find a Trusted Key in the DNSKEY RRset: 19036
109 ;; VERIFYING DNSKEY RRset for . with DNSKEY:19036: success
110
111 ;; Ok this DNSKEY is a Trusted Key, DNSSEC validation is ok: SUCCESS

Bonus content: Adding SSHFP entries for your domain and verifying them You can retrieve the SSHFP entries for a specific host with the following command: 
ssh-keygen -r $(hostname --fqdn) -f /etc/ssh/ssh_host_rsa_key.pub


When retrieved just add the SSHFP entry on the zone file for your domain and verify it: 
ssh -oVerifyHostKeyDNS=yes -v subdomain.gnome.org


Or directly add the above parameter into your /etc/ssh/ssh_config file this way: 
VerifyHostKeyDNS=yes


And run ssh -v subdomain.gnome.org , the result you should receive: 
debug1: Server host key: RSA 00:39:fd:1a:a4:2c:6b:28:b8:2e:95:31:c2:90:72:03
debug1: found 1 secure fingerprints in DNS
debug1: matching host key fingerprint found in DNS
debug1: ssh_rsa_verify: signature correct

That s it! Enjoy!

Joachim Breitner: Constructing a list in a Monad

In Haskell, lists are ubiquitous. In real-world Haskell, monads are ubiquitous. So it should be an easy task to fill a list with values obtained in a monad, i.e. a list of n random numbers (in the Rand monad), or getting some input from the user, and storing it in a list, until the user chooses to finish the input. The task For now, assume a function getInput :: IO Int; we want to run it 10000 and then have the list of the values returned. Sounds simple and straight forward, but it is actually really hard to get the run-time behaviour that we naively and optimistically expect. The expectation In an imperative setting, we would have a simple counting loop. In the loop, we run getInput, allocate a new cell for the linked list, and append it to the list. Clearly, this
  • consumes no stack space and
  • at the end of the loop, the list is already fully evaluated in memory, ready to be used.
Can we achieve that with Haskell? Let s look at a few alternatives (complete code available)
Attempt 1: The accumulator I ll start with the probably worst attempt. Unfortunately, from what I have seen from correcting Haskell exams, this is what beginners would write, especially if they have the above imperative algorithm in mind, and know about the convert loops to recursion with accumulators approach to solving Haskell problems: 
accumAppend :: Int -> IO [Int]
accumAppend n = go n []
  where
    go 0 l = return l
    go n l = do  x <- getInput; go (n-1) (l++[x])
To check the space usage of this, I pass -Kn to the GHC runtime and see what limit is just enough to let this program calculate accumAppend 10000 and fully evaluate the resulting list. It turns out that we need 328336 bytes of stack space. So what about the second requirement; that the list is fully evaluated? Experienced Haskellers see it immediatelly: The accumulator is not used in the loop, so each iteration adds a thunk around the previous value, that would add the next entry to the end. Only when the list is actually used, these are run, each traversing the whole list, causing quadratic cost. We can verify this using ghc-vis:
Clearly (and not surprisingly) this is not the best solution. Attempt 2: The accumulator (forced) Lets try to fix the second issue, by making sure that the list is always fully evaluated. We import Control.DeepSeq and write 
accumAppendSeq :: Int -> IO [Int]
accumAppendSeq n = go n []
  where
    go 0 l = return l
    go n l = do  x <- getInput ; go (n-1) $!! (l++[x])
As the accumulator gets fully evaluated before being passed to the next invocation of go, the result will not contain thunks. But note that this has also fixed the stack consumption: The code now runs the smallest setting for -K, 8 bytes. This works as both go and deepseq are tail-recursive. But this is still not the solution we are looking for, as the quadradic cost caused by using ++ is still there. Attempt 3: The accumulator (reversed) Since we know that l++[x] is expensive, but x:l is cheap, we can simply use this to update the accumulator. This has the slightly annoying consequence that the entries of the list are in the reverse order, but we can fix that in the base case: 
accumReverse :: Int -> IO [Int]
accumReverse n = go n []
  where
    go 0 l = return (reverse l)
    go n l = do  x <- getInput ; go (n-1) (x:l)
And indeed, we get noticably faster code that still runs in 8 bytes of stack. Unfortunately, we still don t construct the final list directly. Attempt 4: Naive recursion Maybe we can do without an accumulator. The most naive such code would be 
listRecurse :: Int -> IO [Int]
listRecurse n = go n
  where
    go 0 = return []
    go n = do  x <- getInput; r <- go (n-1); return (x:r)
and I d expect that most experienced Haskellers would write that code (if they are asked not to use a library function). And indeed, this code does create the final list directly. But it requires a large stack or 164616 bytes. The reason is that this is no longer tail recursive: After calling go again we need to take the result and prepend x Attempt 5: library functions Maybe we should simply not worry about the implementation, and use library functions? Clearly, the authors of such libraries have found the best solution. So how does 
listReplicateM :: Int -> IO [Int]
listReplicateM n = Control.Monad.replicateM n getInput
fare? Unfortunately, not better than the naive recursion. In fact, the stack space requirement is the same 164616 bytes. The same holds when using special libraries for generating and consuming list, e.g. the Streams module from the vector package: 
listStreams :: Int -> IO [Int]
listStreams n = MStream.toList $ MStream.replicateM n getInput
Attempt 6: Difference lists It is time to dig deeper into our box of tricks. What if we have lists with constant time Snoc (a.k.a. appending a new element)? Then Attempt 3 would not require reversing the list. Such lists are provided by difference lists; there are libraries for that, but we can simply implement them using lambdas: 
accumDList :: Int -> IO [Int]
accumDList n = go n id
  where
    go 0 l = return (l [])
    go n l = do  x <- getInput; go (n-1) (l . (x:))
Indeed, no stack is required (8 bytes are enough). But we are cheating here: We are still not constructing the final list, but rather we combine functions that append elements to lists. So when accumDList returns, what we have in memory, is a thunk and sequence of partial function applications in the wrong order, and evaluating the thunk does basically the same thing as reverse in Attempt 3. It might be a bit faster, is still not satisfying enough:
Attempt 7: Unsafe functions Even deepter in the box of tricks, somewhere in the grit and dust where one usually should not reach, there is a function that can help us out here: unsafeInterleaveIO. This allow us to modify the naive recursion in a way that first creates the Cons-cell, and then continues the loop: 
listUnsafeInterleave :: Int -> IO [Int]
listUnsafeInterleave n = do  l <- go n; return $!! l 
  where
    go 0 = return []
    go n = do x <- getInput
              r <- unsafeInterleaveIO (go (n-1))
              return (x:r)
The stack consumption is 8 bytes. It is worth noting that the all to deepseq (in $!!) will actually drive the evaluation of go. Also, it guarantees that the unsafe effects of unsafeInterleaveIO are confined to the execution of listUnsafeInterleave, i.e. are actually safe. Run time statistics I ran criterion over the functions (generating lists of length 1000), and found these results:
  • accumAppend: 6.5 ms
  • accumAppendSeq: 8.2 ms
  • accumReverse: 14.0 us
  • listRecurse: 8.0 us (same as listReplicateM and listStreams)
  • accumDList: 12.0 us
  • listUnsafeInterleave: 174.0 us
Conclusion There are many ways to write a list-constructing list in Haskell, none are perfect they either fill the heap, or do not construct the list directly (usually requiring a second pass through it), or rely on IO-specific unsafe functions. The naive recursion performs the best, but may cause a stack overflow (note, though that from GHC 7.8, the stack will be unlimited). The next best option seems to be the difference lists
I m still wondering: What would be required from Haskell, GHC or the monads in question to have a fully satisfactory solution here, i.e. one that is as fast as the naive recursion, as stack efficient as the difference list solution, and allocates no thunks, only list cells?

12 November 2013

Russell Coker: Bad Advertising of Protoplast

Carpal Tunnel Syndrome In early 2010 I wrote about Carpal Tunnel Syndrome, I gave a picture of the wrist brace and a description of the process of creating it [1]. A few months later I wrote a follow-up about how the wrist brace had helped me, at the time I was anticipating wearing it at night for the rest of my life [2]. But my wrist has continued to improve. Nowadays I never wear the wrist brace and hardly even think of it. I am a little more careful about posture than I was before, but mostly it has no impact on my life. I think this counts as a full recovery. At the time I was mildly interested in buying some thermo-plastic of that type to make things, but the only source I found was Jaycar which sold 100 grams for $14.95, Jaycar prices have now dropped to $11.50 for 100 grams or $89.50 for a kilo but it s still expensive [3]. Protoplast When looking at the sites that link to my blog I saw a site advertising Protoplast which is a type of thermo-plastic that can be used for making things [4]. Protoplast have more reasonable prices, E30 ($AU42.75) for a kilo and their web site offers better prices for anyone who purchases 10kg or more (good for hackerspaces etc). As they post no further than Belgium they aren t an option for me but they do seem to make a good offer, better than Jaycar offers in Australia. Protoplast is probably a good product, but I think that their advert is misleading. While it is probably possible to make a wrist-brace similar to mine with Protoplast it s a fact that my wrist brace was not made with it. My wrist brace started out as a flat sheet of plastic which could be formed to the final product without being melted. Converting a packet of plastic granules (as sold by Protoplast or Jaycar) to a sheet to even start work on such a wrist brace would take some work. To make a sheet of plastic out of beads sold by Jaycar or Protoplast you would ideally completely melt the plastic, if you just softened the plastic then it wouldn t be as strong as using a pre-made sheet. During the time that I wore the wrist brace I probably repaired it more than 30 times, this wasn t particularly difficult to do, I poured boiling water over it to soften the part that cracked until it became tacky and then pushed it together again. But the result was significantly weaker than the original due to the difficulty in welding the broken sections together. I ended up cutting some strips from less important parts of the wrist brace (the part nearest to my elbow) to reinforce the weakest part which was between my thumb and fore-finger. That section ended up being about three times thicker than it was originally while also not being as strong as it was originally. I also had difficulty in making it flat so it wasn t nearly as comfortable as the original version. It seems rather misleading to include the picture of my wrist brace on a site advertising plastic which wasn t used to make it and which probably couldn t be used to make something that looks as good with the same ratio of strength to mass. But with a suitable disclaimer indicating that the user probably can t get the same result it would be OK to give it as an example of what could be done. I can imagine someone using a product like Protoplast to make a wrist brace that can hold a mobile phone, that can be worn with a watch, or has other features that aren t offered by medical staff. License Issues The license for my blog doesn t permit people to just copy pictures, it s a Creative Commons Attribution-Noncommercial-Share Alike based license [5]. I am interested in supporting companies that make interesting products and anything that can help people with medical issues is also of interest. So I wouldn t rule out the possibility of a free license for a picture from my blog to be used in the way that they used it (with a link back to the original blog post). But I would want a description that informs readers that it s not an item made with their product, merely something similar that can be used for inspiration.
Related posts:
  1. Carpal Tunnel Getting Better Three months ago I wrote about getting Carpal Tunnel Syndrome...
  2. Carpal Tunnel Syndrome Diagnosis A few weeks ago I was referred to...
  3. Desks Lindsay Holmwood has written about the benefits of a standing...

11 November 2013

Vincent Bernat: Snimpy: SNMP & Python

While quite old fashioned, SNMP is still an ubiquitous protocol supported by most network equipments. It comes handy to expose various metrics, like network interface counters, to be gathered for the purpose of monitoring. It can also be used to retrieve and modify equipments configuration. Variables exposed by SNMP agents (servers) are organized inside a Management Information Base (MIB) which is a hierarchical database1. Each entry is identified by an OID. Querying a specific OID allows a manager (client) to get the value of an associated variable. For example, one common MIB-module is IF-MIB defined in RFC 2863. It contains objects used to manage network interfaces. One of them is ifTable whose rows are representing agent s logical network interfaces. Each row will expose the interface name, characteristics and various associated counters.
ifIndex ifDescr ifPhysAddress ifOperStatus ifOutOctets
1 lo up 545721741
2 eth0 0:18:f3:3:4e:4 up 78875421
3 eth1 0:18:f3:3:4e:5 down 0
ifTable is indexed by its first column ifIndex. If you want to get the operational status of the second interface, you need to query IF-MIB::ifOperStatus.2 which is translated to OID .1.3.6.1.2.1.2.2.1.8.2 using information provided by the MIB definition.

Scripting SNMP An SNMP agent can deliver a lot of interesting information: You can gather those information manually with tools like snmpget and snmpwalk: 
$ snmpwalk -v 2c -c public localhost IF-MIB::ifDescr      
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
However, building robust scripts with them is quite challenging. For example, if you wanted to get the descriptions of all active interfaces as well as the total number of octets transmitted, you could do something like that: 
#!/bin/sh
set -e
host="$ 1:-localhost "
community="$ 2:-public "
args="-v2c -c $community $host"
for idx in $(snmpwalk -Ov -OQ $args IF-MIB::ifIndex); do
    descr=$(snmpget -Ov -OQ $args IF-MIB::ifDescr.$idx)
    oper=$(snmpget -Ov -OQ $args IF-MIB::ifOperStatus.$idx)
    in=$(snmpget -Ov -OQ $args IF-MIB::ifInOctets.$idx)
    out=$(snmpget -Ov -OQ $args IF-MIB::ifOutOctets.$idx)
    [ x"$descr" != x"lo" ]   continue
    [ x"$oper" = x"up" ]   continue
    echo $descr $in $out
done
Hopefully, SNMP bindings in various languages are pretty common. For example, Net-SNMP ships with a Python binding: 
import argparse
import netsnmp
parser = argparse.ArgumentParser()
parser.add_argument("host", default="localhost", nargs="?",
                    help="Agent to retrieve variables from")
parser.add_argument("community", default="public", nargs="?",
                    help="Community to query the agent")
options = parser.parse_args()
args =  
    "Version": 2,
    "DestHost": options.host,
    "Community": options.community
 
for idx in netsnmp.snmpwalk(netsnmp.Varbind("IF-MIB::ifIndex"),
                            **args):
    descr, oper, cin, cout = netsnmp.snmpget(
        netsnmp.Varbind("IF-MIB::ifDescr", idx),
        netsnmp.Varbind("IF-MIB::ifOperStatus", idx),
        netsnmp.Varbind("IF-MIB::ifInOctets", idx),
        netsnmp.Varbind("IF-MIB::ifOutOctets", idx),
        **args)
    assert(descr is not None and
           cin is not None and
           cout is not None) #  
    if descr == "lo":
        continue
    if oper != "1": #  
        continue
    print("     ".format(descr, cin, cout))
This binding is quite primitive and has several drawbacks:
  1. It exports everything as strings. See .
  2. Error handling is just deficient. If you mispell something, like a variable name, you ll get snmp_build: unknown failure on the standard error. No exception. If a variable does not exist, you ll get None instead. See .
This inability to sanely handle failures makes this binding quite dangerous to use in scripts. Imagine making important modifications on the basis of the returned values. If you forget to check against None, your script may cause havoc!

Snimpy Because I didn t find any reliable Python binding for SNMP, I decided to write Snimpy with two goals in mind:
  1. Leverage information contained in MIBs to provide a pythonic interface.
  2. Any error condition should raise an exception.
Here is how the previous script could be written: 
#!/usr/bin/env snimpy
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("host", default="localhost", nargs="?",
                    help="Agent to retrieve variables from")
parser.add_argument("community", default="public", nargs="?",
                    help="Community to query the agent")
options = parser.parse_args()
m = M(options.host, options.community, 2)
load("IF-MIB")
for idx in m.ifDescr:
    if m.ifDescr[idx] == "lo":
        continue
    if m.ifOperStatus[idx] != "up":
        continue
    print("     ".format(m.ifDescr[idx],
                            m.ifInOctets[idx],
                            m.ifOutOctets[idx]))
You can also use a list comprehension: 
load("IF-MIB")
print("\n".join([ "     ".format(m.ifDescr[idx],
                                    m.ifInOctets[idx],
                                    m.ifOutOctets[idx])
                  for idx in m.ifDescr
                  if m.ifDescr[idx] != "lo"
                  and m.ifOperStatus[idx] == "up" ]))
Here is another simple example to get the routing database from the agent: 
load("IP-FORWARD-MIB")
m=M("localhost", "public", 2)
routes = m.ipCidrRouteNextHop
for x in routes:
    net, netmask, tos, src = x
    print(" :>15s / :<15s  via  :<15s  src  :<15s ".format(
        net, netmask, routes[x], src))
IP-FORWARD-MIB::ipCidrRouteNextHop is a more complex table with a compound index. Despite this, querying the table still seems natural. Have a look at Snimpy s documentation for more information. Behind the hood, SNMP requests are handled by PySNMP and MIB parsing is done with libsmi2. Snimpy supports both Python 2, Python 3 and Pypy.
  1. A MIB is defined using a subset of ASN.1 called SMI. However, it is not uncommon to refer to the definition as a MIB too.
  2. Unfortunately, there is currently no robust SMI parser written in pure Python. For example, PySNMP relies on smidump which comes with libsmi. Snimpy uses a custom CFFI wrapper around libsmi.

10 November 2013

Sylvain Le Gall: opam2debian, a tool to create Debian binary package out of OPAM

One week ago, a thread in opam-devel mailing list started about the possibility to create binary snapshot to distribute OPAM packages. Distributing binary packages with everything already compiled is pretty useful when you want to make sure that everyone has the same version of packages installed and you don't want to spend time configuring all the computer of your colleague. As a matter of fact I was also interested to see that happening. I have several computers where I want to install a set of packages and I want to snapshot OPAM archives when I am ready for an upgrade. I have tested for a long time another source distribution: GODI. I even wrote a puppet module to drive it. This was a fun experience, but as with any source only distribution, there are some drawbacks. Especially even if it is fully automated, you get a lot of errors and timeout when trying to build automatically from source. This is how opam2debian has started. I wanted a replacement for puppet-godi that takes advantage of OPAM and the possibility to distribute my set of packages everywhere quickly. The goals of this project are:
  1. generate a Debian binary package with all the dependencies set on external packages (like libpcre)
  2. use OPAM to build everything
  3. use standard Debian process to build package
  4. create snapshot of OPAM repository that allow to rebuild exactly the same thing on different arches
The real challenge behind opam2debian was to be able to use standard Debian process to create package to get all the power of dependencies computation provided by the Debian maintainer scripts. My great achievement on this topic was to find a tool called proot that allows you to bind mount a directory as a user. This is an achievement because it allows to create a package using a directory where you are not allowed to write (i.e. /opt/opam2debian directory which is owned by root). This is a way to do it on any Jenkins builder without root access and a standard way for Debian to build packages. Usage example I need to compile a set of package to install on all my Jenkins builder. Here is the list of packages I need and I also want to use the latest OCaml version. I want to build a Debian package for Debian Wheezy which has only OCaml 3.12.1. Building the package: 
$ opam2debian create --build --name opam2debian-test2 --compiler 4.01.0 \
 ocamlfind fileutils ocaml-data-notation ocaml-expect ounit \
 ocamlmod ocamlify oasis yojson sexplib extlib pcre-ocaml \
 calendar ocaml-inifiles ocamlnet ocurl gettext inotify ocaml-sqlexpr \
 ocamlrss ocaml-xdg-basedir
Getting the package list right can be tricky because the process will stop at the first error -- and if it is because the last package fails to build it can be long. I have implemented a --keep-build-dir in case you want to tune the package list live. The program will build everything and it can takes quite a while. At the end you get a file opam2debian-test2_20131105_amd64.deb which has a reasonable size of 232MB. That is big, but it looks like the biggest directory is $OPAMROOT/4.01.0/build, I am not sure if I can remove it but we may save some space here (the build directory represents 50% of the package size). Then you can just do a standard Debian installation: 
$ sudo dpkg -i opam2debian-test2_20131105_amd64.deb
And you can use it: 
$ eval $(opam config env --root /opt/opam2debian/opam2debian-test2/)
$ which ocamlfind
 /opt/opam2debian/opam2debian-test2/4.01.0/bin/ocamlfind
$ ocamlfind list
 [...]
 stdlib              (version: [distributed with Ocaml])
 str                 (version: [distributed with Ocaml])
 threads             (version: [distributed with Ocaml])
 threads.posix       (version: [internal])
 threads.vm          (version: [internal])
 type_conv           (version: 109.41.00)
 unidiff             (version: 0.0.2)
 unix                (version: [distributed with Ocaml])
 userconf            (version: 0.3.1)
 xdg-basedir         (version: 0.0.3)
 xmlm                (version: 1.1.1)
 yojson              (version: 1.1.5)
A nice thing to note, is that it is a standard OPAM install. You can install, update and upgrade OPAM from there, as root. However, I would recommend not to do it, and just rebuild a newer Debian package to upgrade. Install You will need the opam and proot Debian package available in Debian jessie and sid. You will also need various OCaml libraries (cmdliner, calendar, fileutils and jingoo). Download the opam2debian tarball on the forge, build and install it. The project is hosted on github. Open issues On the initial list of goals of the project. Not everything is completed. I have still several open issues. In particular 4. (create snapshot of OPAM repository) was blocked by a bug in opam-mk-repo, that prevents snapshots (see my pull-request to solve it). Another issues is about licenses of the included files. I should list them all and I need to figure out a way to extract every licences. Submit bugs directly to github.

Thomas Goirand: Symplifying sysv-rc init.d scripts

Peter Reinholdtsen posted an article on planet.debian.org about symplifying init scripts. I don t think that s a good idea. I tried to do that, and then scrapped all my code, because I ve found OpenRC that was doing what I wrote, in a much better way. We shouldn t reivent the wheel, OpenRC is there already!

Jelmer Vernooij: Quantified Self

Dear lazyweb, I've been reading about what the rest of the world seems to be calling "quantified self". In essence, it is tracking of personal data like activity, usually with the goal of data-driven decision making. Or to take a less abstract common example: counting the number of steps you take each day to motivate yourself to take more. I wish it'd been given a less annoying woolly name but this one seems to have stuck. There are a couple of interesting devices available that track sleep, activity and overall health. Probably best known are the FitBit and the jazzed-up armband pedometers like the Jawbone UP and the Nike Fuelband. Unfortunately all existing devices seem to integrate with cloud services somehow, rather than giving the user direct access to their data. Apart from the usual privacy concerns, this means that it is hard to do your own data crunching or create a dashboard that contains data from multiple sources. Has anybody found any devices that don't integrate with the cloud and just provide raw data access?

Next.

Previous.