Search Results: "tar"

26 November 2013

Axel Beckert: Next Debian Meetup in Zurich

The first Debian meetup in Zurich last month was quite a success and I look forward to further Debian meetups in Zurich every first Tuesday of the month. The next meetup will be

on Tuesday, 2013-Dec-03 starting at 18:30 CET
at St. Gallerhof, Konradstrasse 2, 8005 Zurich
Please note that this is a different location than last month. Everybody who is interested in Debian is welcome to join us. Registering is not necessary. There was also some interest in Debian meetups in other Swiss cities, namely Bern and somewhere around Lac Leman. In case you want a Debian meetup elsewhere in Switzerland, too, or if you re interested in any Debian meetup in Switzerland, feel free to join the Swiss Debian Community Mailing List and help organising other Swiss Debian meetups.

Daniel Pocock: The free software country?

Loch Ness Monster, Scotland There is big news in Europe right now, especially the UK, about the Scottish nation's manifesto for independence. I thought this was really amazing, a country that is going to move to free software and protect its citizens from horrors like DRM. I couldn't wait to find out more, for example, will they use Linux or *BSD? After all, how many nations today can really consider themselves independent when they are trapped in the use of complex and opaque systems that require ongoing royalty payments to a foreign corporation, much like ancient villages paying their tithes to Rome? Today it is actually worse: these independent nations are not just sacrificing their cash - the new Rome is also using those same secret, proprietary systems to raid their subjects' privacy. So I went to the manifesto's web site to get the details. I couldn't even find the words "software" or "technology" listed in the policy index on the front page - and software doesn't even appear in the search results. A section on Culture, Communications and Digital only really looks at the issues for the media (BBC after independence). They don't even make any comment about the British Telecom (BT) pension crisis and whether Scotland will abandon BT's services, leaving only those customers south of the border to continue subsidising the deficit through higher phone bills. There is some hope, with the dot scot (.scot) top level domain coming in 2015 regardless of the referendum outcome. Does this mean an independent Scotland is going to be run without technology at all then? That could be an interesting way to avoid the type of infiltration of communications technology that has been taking place in Brussels. Or do the people calling the shots simply fail to realise the impact of software and communications technology on a modern day concept of independence? Party date set For those who just care about gatecrashing the post-independence party, start booking your leave for March 2016.

Ricardo Mones: Shared memory crazyness

The output of some commands explains it all.

These are the default values in a Wheezy system:

$ ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1


That's not enough for all the data I want to load in a single segment, so let's start with 1Gb of shared memory:

# sysctl kernel.shmmax=1073741824 kernel.shmall=1073741824
kernel.shmmax = 1073741824
kernel.shmall = 1073741824

So now, both should be equal, isn't it?

$ ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 1048576
max total shared memory (kbytes) = 4294967296
min seg size (bytes) = 1

Uh!?

A free $BEVERAGE when we met for the one which tells me what's happening here ;-)

25 November 2013

Steve McIntyre: Mini-Debconf in Cambridge, November 14-17 2013

Phew, we did it! We successfuly ran our mini-conf last weekend. We welcomed roughly 60 people to the ARM offices in Cambridge over the 4 days. We started off with two days of sprint time, set aside for focussed hacking. We had a selection of ARM development boards on hand that people were playing with, plus a number of other people working on various other projects. The sprint sessions also saw some lively discussions on a couple of topics: the status of the ARM ports in Debian and automated system testing. Saturday and Sunday saw us move over to the more traditional style of conference session. Two full days of talks covered a wide range of topics: system testing to git tools, Debian ftp team work to bootstrapping the new arm64 port, improving ways of tracking and crediting Debian contributions to dealing with the new world of UEFI. Several volunteers from the DebConf video team were on hand too, so our talks were streamed live and recorded too - look out for them online soon! The mini-conf went well and feedback from attendees was universally positive, so we're already planning to run another similar event next year and maybe even apply to host a full DebConf in Cambridge soon too. :-) Many more details about the conference are in the debconf wiki at https://wiki.debconf.org/wiki/Miniconf-UK/2013 . Thanks to the sponsors who helped to make it all happen: ARM, Boston, Open@Citrix/Xen and Codethink/Baserock. 2013-11-25 UPDATE: Videos are now online at http://meetings-archive.debian.net/pub/debian-meetings/2013/mini-debconf-cambridge/ in various formats.

23 November 2013

Russell Coker: Failures of Intelligence Agencies

There s been an ongoing news issue related to the NSA that I m not going to directly comment on, Charles Stross has one of the more interesting comments about it [1]. One of his major points is that any success at government secrecy relies on the type of work environment that existed 40+ years ago. Corporatism has killed government secrecy now it s a matter of time before corporate whistle-blowers start seriously leaking documents. One article that Charles links to is an interesting and insightful blog post Adam Curtis wrote for the BBC about the many failures of MI5 [2]. I think that everyone is aware of their biggest failures but from Adam s article it seems that they have only failed and never succeeded in anything. I wonder whether ASIO is any better, it s probably not (the Wikipedia page notes the involvement of MI5 in creating ASIO). It seems that the best way of achieving the goals of ASIO and MI5 would be to disband those organisations and assign regular police to do such work, after all it s been proven that British police are better at catching spies than MI5.

22 November 2013

Enrico Zini: loncity

Airports Photo of a commercial in London City airport saying 'In the lap of luxury - Want to reach a captive audience with dwell time? Why advertise anywhere else? - London City Airport Media Sales' In the airport, we are not travellers. We are a captive audience with dwell time. In other words, suckers stuck in a room where the only pastime provided is spending money and staring at advertisements selling advertisement space in rooms full of suckers like them.

Vincent Sanders: Error analysis is the sweet spot for improvement

Although Don Norman was discussing designers attitude to user errors I assert the same is true for programmers when we use static program analysis tools.

The errors, or rather defects in the jargon, that a static analysis tools produce can be considered low cost well formed bug reports available very early in the development process.

When I say low cost it is because they can be found by a machine without a user or fellow developer wasting their time finding them. Well formed comes because the machine can describe exactly how it came to the logical deduction leading to the defect.
IntroductionStatic analysis is in general terms using the computer to examine a program for logic errors beyond those of pure syntax before it is executed. Examining a running program for defects is known as dynamic program analysis and while a powerful tool in its own right is not the topic of discussion.

This analysis has historically been confined to compiled languages as their compilers already had the Abstract Syntax Tree (AST) of the code available for analysis. As an example the C language (released in 1972) had the lint tool (released in 1979) based on the PCC compiler.

Practical early compilers (I am generalising here as the 19070s were a time of white hot innovation in computing and examples of just about any innovation in the field could probably be found) were pretty primitive and produced executables which were less good than hand written assembler output. Due to practical constraints the progress of optimising compilers was not as rapid as might be desired so static analysis was largely used as an external process.

Before progressing I ought to explain why I just mixed the concept of an optimising compiler and static analysis. The act of optimisation within those compilers requires program analysis, from which they can generate defect reports which we all know and love as compiler warnings, also explaining why many warnings only appear at higher optimisation levels where deeper analysis is required.

The attentive reader may now enquire as to why we would need external analysis tools when our compilers already perform the task. The answer stems from the issue that a compiler is trying to reconcile many desirable traits including:
The slow progress in creating optimising compilers initially centred around the problem of getting the compiled output in a reasonable time to allow for a practical edit-compile-run-debug cycle although the issues more recently have moved more towards the compiler implementation costs.

Because the output generation time is still a significant factor compilers limit the level of static analysis performed to that strictly required to produce good output. In standard operation optimising compilers do not do the extended analysis necessary to find all the defects that might be detectable.

An example: compiling one 200,000 line C program with the clang (v3.3) compiler producing x86 instruction binaries at optimisation level 2 takes 70 seconds but using the clang based scan-build static analysis tool took 517 seconds or more than seven times as long.
Using static analysis
As already described warnings are a by-product of an optimising compilers analysis and most good programmers will endeavour to remove all warnings from a project. Thus almost all programmers are already using static analysis to some degree.

The external analysis tools available can produce many more defect reports than the compiler alone as long as the developer is prepared to wait for the output. Because of this delay static analysis is often done outside the usual developers cycle and often integrated into a projects Continuous Integration (CI) system.

The resulting defects are usually presented as annotated source code with a numbered list of logical steps which shows how the defect can present. For example the steps might highlight where a line of code allocates memory from the heap and then an exit path where no reference to the allocated memory is kept resulting in a resource leak.

Once the analysis has been performed and a list of defects generated the main problem with this technology rears its ugly head, that of so called "false positives". The analysis is fundamentally an undecidable problem (it is a variation of the halting problem) and relies on algorithms to generate approximate solutions. Because of this some of the identified defects are erroneous.

The level of erroneous defect reports varies depending on the codebase being analysed and how good the analysis tool being used is. It is not uncommon to see false positive rates, even with the best tools, in excess of 10%

Good tools allow for this and provide ways to supply additional context through model files or hints in the source code to suppress the incorrect defect reports. This is analogous to using asserts to explicitly constrain variable values or a type cast to suppress a type warning.

Even once the false positives have been dealt with there comes the problem of defects which while they may be theoretically possible take so many steps to achieve that their probability is remote at best. These defects are often better categorized as a missing constraint and the better analysis tools generate fewer than the more naive implementations.

An issue with some defect reports is that often defects will appear in a small number of modules within programs, generally where the developers already know the code is of poor quality, thus not adding useful knowledge about a project.

As with all code quality tools static analysis can be helpful but is not a panacea code may be completely defect free but still fail to function correctly.
Defect DensityA term that is often used as a metric for code quality is the defect density. This is nothing more than the ratio of defect to thousands of lines of code e.g. a defect density of 0.9 means that there is approximately one defect found in every 1100 lines of code.

The often quoted industry average defect density value is 1, as with all software metrics this can be a useful indicator but should not be used without understanding.

The value will be affected by improvements in the tool as well as how lines of code are counted so is exceptionally susceptible to gaming and long term trends must be treated with scepticism.
Practical examplesI have integrated two distinct static analysis tools into the development workflow for the NetSurf project which I shall present as case studies. These examples show a good open source solution and a commercial offering highlighting the issues with each.

Several other solutions, both open source and commercial, exist many of which have been examined and discarded as either impractical or proving less useful than those selected. However the investigation was not comprehensive and only considered what was practical for the project at the time.
clangThe clang project is a frontend to the LLVM project providing an optimising compiler for the C, C++ and objective C languages. As part of this project the compiler has been enhanced to run a collection of "checkers" which implement various methods of analysis on the code being compiled.

The "scan-build" tool is provided to make the using these features straightforward. This tool generates defect reports as a series of html files which show the analysis results.


NetSurf CI system scan-build overview
Because the scan-build takes in excess of eight minutes on powerful hardware the NetSurf developers are not going to run this tool themselves as a matter of course. To get the useful output without the downsides it was decided to integrate the scan into the CI system code quality checks.

NetSurf CI system scan-build result list
Whenever a git commit happens to the mainline branch and the standard check build completes successfully on all target architectures the scan is performed and the results are published as a list of defects.

The list is accessible directly through the CI interface and also incorporates a trend graph showing how many defects were detected in each build.

A scan-build report showing an extremely unlikely path to a defect
Each defect listed has a detail link which reveals the full analysis and logic necessary to cause the defect to occur.

Unfortunately even NetSurf which is a relatively small piece of software (around 200,000 lines of code at time of writing) causes 107 defects to be emitted by scan-build.

All but 25 of the defects are however "Dead Store" where the code has a value assigned but is never checked. These errors are simply not interesting to the developers and are occurring in code generated by a tool.

Of the remaining defects identified the majority are false positives and several (like the example in the image above) are simply improbable requiring a large number of steps to reach.

This shows up the main problem with the scan-build tool in that there is no way to suppress certain checks, mark defects as erroneous or avoid false positives using a model file. This reduces the usefulness of these builds because the developers all need to remember that this list of defects is not relevant.

Most of the NetSurf developers know that the project currently has 107 outstanding issues and if a code change or tool improvement were to change that value we have to manually work through the defect list one by one to check what had changed.
CoverityThe coverity SAVE tool is a commercial offering from a company founded in the Computer Systems Laboratory at Stanford University in Palo Alto, California. The results of the original novel research has produced a good solution which improved on analysis tools previously available.

Coverity Interface showing summary of NetSurf analysis. Layout issues are a NetSurf bug
The company hosts a gratis service for open source projects, they even provide scans for the Linux kernel so project size does not appear to be an issue.

The challenges faced integrating the coverity tool into the build process differed from clang however the issue of execution time remained and the CI service was used.

The coverity scanning tool is a binary executable which collects data on the build which is then submitted to the coverity service to be analysed. This tool obviously relies upon the developer running the executable to trust coverity to some degree.

A basic examination of the binary was performed and determined the executable was not establishing network connections or performing and observably undesirable behaviour. From this investigation the decision was made that running the tool inside a sandbox environment on a CI build slave was safe. The CI system also submits the collected results in a compressed form directly to the coverity scan service.

Care must be taken to only submit builds according to the services Acceptable Use Policy which limits the submission frequency of NetSurf scans to every other day. To ensure the project stays within the rules the build performed by the CI system is manually controlled and confined to a subset of NetSurf developers.

Coverity connect defect management console for NetSurfThe results are presented using the coverity connect web technology based defect management tool. Access to the coverity connect interface is controlled by a user management system which precludes publicly publishing the results within the CI system.

Unfortunately NetSurf itself does not currently have good enough JavaScript DOM bindings to support this interface so another browser must be used to view it.

Despite the drawbacks the quality of the analysis results is greatly superior to the clang solution. The false positive rate is very low while finding many real issues which had not been previously detected.

The analysis can be enhanced by use of collection configuration and modelling files which remove intended constructions from consideration reducing the false positive rate to very low levels. The ability to easily and persistently suppress false positives through the web interface is also available.

The false positive management capabilities coupled with a user interface that makes understanding the defect path simple make this solution very practical and indeed the NetSurf developers have removed over 50 actual issues within a relatively short period since the introduction of the tool.

Not all of those defects could be considered serious but they had the effect of encouraging deeper inspection of some very dubious smelling source.
ConclusionsThe principle conclusions of implementing and using static analysis have been:

When I started looking at this technology I was somewhat dubious about its usefulness but I have definitely changed my mind. It is a useful addition to any non-trivial project and the return on time and effort should be repaid handsomely in all but already perfect code (if you believe you have such code I have a bridge to sell you).

Russ Allbery: Review: The Pathway to Awesomeness

Review: The Pathway to Awesomeness, by Mark Forster
Publisher: Hyperink
Copyright: October 2013
ASIN: B00FXGXMPQ
Format: Kindle
Pages: 97
Mark Forster is probably my favorite time management (or, if you prefer, attention management) writer. The place to start with Forster if you're not familiar with him is Do It Tomorrow, not here. Unlike the other books by him I've reviewed, this is just an edited collection of blog posts presented in a convenient ebook form, without even an ISBN number. But if you've read his books (at least the ones that aren't out of print and cost a fortune), this is a convenient way to not have to trawl back through his blog and read the articles in a more convenient fashion. For me, it was also an easy way to give him a bit of money for his blog. Unless you have a similar motive, I don't think the content to price ratio is worth it. But I think everything is still up at Get Everything Done in case anything here strikes your fancy. As with most blog compilations, The Pathway to Awesomeness (I really hate that title) is composed of a whole bunch of fairly short articles, rarely more than three pages each, vaguely collected by topic. The first few are rather general and focused on the normal self-help platitudes that try to build up your sense that you're in control of your own life. But Forster doesn't take too long to get to what he's best at: short ideas, mental tools, interesting tricks, and simple techniques. Even more so than his books, due to the nature of this collection, there's a huge variety here, which means you can probably find something that strikes a chord no matter what attention management problem you've been running into. That's one of the main reasons why I prefer his writing: he tries to provide a giant toolbox instead of a single, streamlined System For Everyone. The other thing I like about Forster is that he is much more blunt than many authors about the biggest problem I have with attention management: trying to do more than anyone could possibly do given the available time. There are some more essays here about that, and several of them are quite good. I think my favorite is the short essay in which he points out how often people decide to do some task for a particular reason, and then get so engrossed in the performance of that task that they forget the original reason, risking sinking far more energy into that task than is actually warranted. Like a lot of Forster's observations, it's obvious in retrospect, but he has a way of phrasing those observation so that they shake the right thing loose in my head and let me re-evaluate something I was doing. Regular readers of Forster's blog know that he loves playing around with different simple to-do list management systems. Unfortunately, none of those are here, so don't pick this up expecting a collection of articles about Superfocus, Autofocus, etc. It makes sense given the topic of this collection, which is more general advice rather than systems, but this collection is so short that I wish he'd added a section on system development. I would have enjoyed reading a collection of all of his miniature systems together in one place and showing how he tweaked and changed them over time, particularly if he'd added some discussion of why he made each change and what goals he was trying to meet with each system. Ah well. If you aren't already familiar with Forster, there's no point in this, but if you are, I got a few new tidbits out of it and will probably skim through it a few more times. Buying it is mostly an exercise in putting money in the tip jar, and his writing is much less polished here (and more prone to cliches), but I still enjoyed reading through it. Rating: 7 out of 10

21 November 2013

Russ Allbery: Small but lovely haul

A few things for which I'd been waiting for quite some time were released, so there was another book order. Allie Brosh Hyperbole and a Half (graphic novel)
Jacqueline Carey Dark Currents (sff)
Debra Dunbar A Demon Bound (sff)
Nicola Griffith Hild (historical)
Ursula K. Le Guin The Unreal and the Real, Volume 1 (sff)
Ursula K. Le Guin The Unreal and the Real, Volume 2 (sff)
Ken Stern With Charity for All (non-fiction) I'm trying to buy fewer books given how many I have and haven't read, and also because I've been on a gaming tear recently and there are only so many hours in the day. Unfortunately, playing a lot of video games means reading fewer books. So I'm trying for quality rather than quantity, and this particular order has one of the best densities of hopefully good books as anything I've ordered in a while. For those who aren't familiar with Allie Brosh, Hyperbole and a Half is one of the most brilliant blogs that I've ever read, and this is a collection plus new stories and new illustrations. I specifically got the hardcover, which is beautiful and very high-quality. I've been waiting for Hild for years since Griffith started talking about writing it on her blog, and now I have the lovely hardcover in my hands. That's probably going to be Christmas reading. The Le Guin is a two-volume hardcover collection of Le Guin's personal favorites from her large lifetime oeuvre of short fiction.

Craig Small: WordPress password bots

Browsing through my logs I noticed that one particular IP address was continuously trying to go to wp-login.php After a few more greps, it seems he really likes this URL. So, Mr 37.115.188.210 congratulations for testing a few things and welcome to the blocklist. I love fail2ban, but initially I didn t have it for the wordpress login. That needed to get fixed real quick, so a visit to the wordpress plugins site and we have WP fail2ban up and running. And doesn t it work well:
2013-11-21 22:54:47,742 fail2ban.actions: WARNING [wordpress] Ban 37.115.188.210
2013-11-21 22:58:29,037 fail2ban.actions: WARNING [wordpress] Unban 37.115.188.210
2013-11-21 22:58:39,450 fail2ban.actions: WARNING [wordpress] Ban 37.115.188.210
2013-11-21 23:08:40,164 fail2ban.actions: WARNING [wordpress] Unban 37.115.188.210
2013-11-21 23:09:27,241 fail2ban.actions: WARNING [wordpress] Ban 37.115.188.210
2013-11-21 23:19:27,919 fail2ban.actions: WARNING [wordpress] Unban 37.115.188.210
2013-11-21 23:20:09,991 fail2ban.actions: WARNING [wordpress] Ban 37.115.188.210
2013-11-21 23:30:10,689 fail2ban.actions: WARNING [wordpress] Unban 37.115.188.210
You get the idea! I ve sent a message off to the responsible ISP, we ll see how that goes.

Gunnar Wolf: Back aches...

This year, I had slowly taken again running. It's an activity I enjoy, even though I'm far from the condition I had when I did it every day My maximum was running about 8Km four days a week (ocassionally up to 15Km, say, on weekends)... But I slowly drifted out of it over the last three years. Yes, I have taken it up now and then, but dropped out again after a few weeks. Well, this year it felt I was getting back in the routine. Not without some gaps, but I had ran most weeks since July at least once, but usually two times. I started slowly, doing about 3Km close to home, but didn't take much to go back to my usual routes inside UNAM, doing a modest average of 4.5-5Km per run, and averaging somewhat over 8Km/h (6.5 min/Km). But... Well, I cannot relate precisely what happened on this Exactly a month ago, after a very average run (even a short one, 3.6Km), I had a lower back ache. Didn't pay too much attention to it, but it was strange since the first day, as I often have upper back aches Never lower. The pain came and went slowly several times. I kept cycling to work, although not every day. On October 27th, I even took the time to do the Ciclot n, a nice 38Km (including the ~5Km distance from home) around central Mexico City. I enjoyed the ride, even though every stop and re-start of the bike was a bit painful what hurts is to get off and on the bike: The posture change. But the pain has been almost constantly there. When I stand up, or when I sit down, it's about five minutes until the body gets used to the new position and stops hurting. Another strange data point: This last weekend (together with a national holiday) we went to a small conference in Guadalajara, and then to visit our friends in Guanajuato. So, we spent 12 hours on the bus there (12 Yes, there was an accident on the road, and we could not pass... So many hours were wasted there, and going back to a junction to take an alternative, longer road), then one night in a hotel bed, and two nights in our friends' guest mattresses which are not precisely luxury-grade. And Sunday night... I had no pain at all! Came back home, and after only one night back in our bed... I just could not move. I had the strongest pain so far. Could not even walk without some help. We went to the orthopedist a friend recommended, and I was seriously bending my posture: While that part of my posture is usually stable, my right hip was about 2cm higher than the left one, and my shoulders were almost 7cm displaced from my hips! So... Well, I'm having a cocktail of painkillers and antiinflamatories. The doctor says next week he wants me to have a tomography taken to better understand the causes for this. And, of course, tomorrow I'm leaving for Festival de Software Libre in Puerto Vallarta. I'm going there by plane, so no big hassle. But the way back, it will be 12 more hours on a bus. I'm... not precisely looking forward to this bus ride :( Anyway... I sorely miss my bike+running :(

Petter Reinholdtsen: All drones should be radio marked with what they do and who they belong to

Drones, flying robots, are getting more and more popular. The most know ones are the killer drones used by some government to murder people they do not like without giving them the chance of a fair trial, but the technology have many good uses too, from mapping and forest maintenance to photography and search and rescue. I am sure it is just a question of time before "bad drones" are in the hands of private enterprises and not only state criminals but petty criminals too. The drone technology is very useful and very dangerous. To have some control over the use of drones, I agree with Daniel Suarez in his TED talk "The kill decision shouldn't belong to a robot", where he suggested this little gem to keep the good while limiting the bad use of drones:
Each robot and drone should have a cryptographically signed I.D. burned in at the factory that can be used to track its movement through public spaces. We have license plates on cars, tail numbers on aircraft. This is no different. And every citizen should be able to download an app that shows the population of drones and autonomous vehicles moving through public spaces around them, both right now and historically. And civic leaders should deploy sensors and civic drones to detect rogue drones, and instead of sending killer drones of their own up to shoot them down, they should notify humans to their presence. And in certain very high-security areas, perhaps civic drones would snare them and drag them off to a bomb disposal facility. But notice, this is more an immune system than a weapons system. It would allow us to avail ourselves of the use of autonomous vehicles and drones while still preserving our open, civil society.
The key is that every citizen should be able to read the radio beacons sent from the drones in the area, to be able to check both the government and others use of drones. For such control to be effective, everyone must be able to do it. What should such beacon contain? At least formal owner, purpose, contact information and GPS location. Probably also the origin and target position of the current flight. And perhaps some registration number to be able to look up the drone in a central database tracking their movement. Robots should not have privacy. It is people who need privacy.

Steve McIntyre: Mini-Debconf in Cambridge, November 14-17 2013

Phew, we did it! We successfuly ran our mini-conf last weekend. We welcomed roughly 60 people to the ARM offices in Cambridge over the 4 days. We started off with two days of sprint time, set aside for focussed hacking. We had a selection of ARM development boards on hand that people were playing with, plus a number of other people working on various other projects. The sprint sessions also saw some lively discussions on a couple of topics: the status of the ARM ports in Debian and automated system testing. Saturday and Sunday saw us move over to the more traditional style of conference session. Two full days of talks covered a wide range of topics: system testing to git tools, Debian ftp team work to bootstrapping the new arm64 port, improving ways of tracking and crediting Debian contributions to dealing with the new world of UEFI. Several volunteers from the DebConf video team were on hand too, so our talks were streamed live and recorded too - look out for them online soon! The mini-conf went well and feedback from attendees was universally positive, so we're already planning to run another similar event next year and maybe even apply to host a full DebConf in Cambridge soon too. :-) Many more details about the conference are in the debconf wiki at https://wiki.debconf.org/wiki/Miniconf-UK/2013 . Thanks to the sponsors who helped to make it all happen: ARM, Boston, Open@Citrix/Xen and Codethink/Baserock.

Michael Banck: 21 Nov 2013

PostgreSQL talk at Linux-Stammtisch

Next Wednesday (November 27th), the Munich Linux-Stammtisch is happening again. This time the topic will be Open-Source databases. Mathias Brandstetter will present MySQL and I will give a talk about PostgreSQL.

So if you are from or around Munich and using/interested in PostgreSQL in particular or databases in general, come around! The Stammtisch is in the Maximiliansst berl at Hofbr ukeller, starting from 19:00. As a Brotzeit is sponsored for attendees, you need to sign up either at the Xing page or here.

Craig Small: Quieting dbus

I run a program called logcheck which regularly scans the logfiles looking for interesting things . One of these interesting things (to logcheck, not to me at first) was dbus complaining about mythtv. The logs look something like this:
Nov 17 05:08:18 elmo dbus-daemon[1621]: dbus[1621]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.236" (uid=999 pid=3669 comm="/usr/bin/mythfilldatabase --verbose general --logl") interface="org.freedesktop.NetworkManager" member="GetDevices" error name="(unset)" requested_reply="0" destination="org.freedesktop.NetworkManager" (uid=0 pid=1930 comm="/usr/sbin/NetworkManager ")
What this is is the dbus daemon rejecting a message being sent from mythfilldatabase that wants to go to NetworkManager. Looking at NetworkManager s dbus configuration, its doing all the right things. So it s a simple matter of just fixing up the dbus configuration and you re done. Then I hit the first snag; the dbus documentation is a little sad. I remember seeing a show where they MRI scanned some artifacts to get to the writing and then used a maker-bot to recreate the hidden writing. They then painstakingly translated it using a team of experts to work out the English translation. Working out the dbus configurations is not quite that difficult, but it s close. The easiest way is to copy something and see if it works, which is what I did. I created a file /etc/dbus-1/system.d/mythtv.conf which permitted the myth user access to the GetDevices method. It looks like:
<!DOCTYPE busconfig PUBLIC
 "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>
        <policy user="mythtv">
                <allow send_destination="org.freedesktop.NetworkManager"
                       send_interface="org.freedesktop.NetworkManager"
                       send_member="GetDevices"/>
        </policy>
</busconfig>
Reloaded dbus and all went quiet.

19 November 2013

Carl Chenet: Yarn: a scenario testing tool

Follow me also on Twitter Those who follow me on Twitter may have read my recent tweet about Yarn, a really nice software by Lars Wirzenius I discovered the last week-end while attending the Mini-Debconf UK in Cambridge. To make a long story short, Yarn allows the user to write scenarios of the executions of softwares. This scenario will be automatically played by Yarn which also returns the overview of the execution, with the goal to test the behaviour of these softwares. 1. Small example of how to use Yarn Let s have a look of how to use Yarn with a real example. The scenario will test if the software Brebis, a fully automatized backup checker, executes in an expected way. We can proceed as follow :
  1. set up the test environment
  2. launch a command with the software under test in the test environment
  3. verify the execution of the test
This is an example of a Yarn scenario file, the file is formatted using the Markdown syntax:
    SCENARIO basic brebis execution
    GIVEN setting up brebis
    AND generating backup configuration with brebis
    WHEN brebis verifies a backup
    THEN verify brebis output
    IMPLEMENTS GIVEN setting up brebis
    hg clone http://hg.brebisproject.org $DATADIR/brebis
    mkdir -p $DATADIR/brebis/yarn-test
    cp $DATADIR/brebis/functional-tests/expected-generated-list-for-tar-archive/expected-generated-list-for-tar-archive.tar.gz $DATADIR/brebis/yarn-test
    IMPLEMENTS GIVEN generating backup configuration with brebis
    $DATADIR/brebis/brebis.py -G $DATADIR/brebis/yarn-test/expected-generated-list-for-tar-archive.tar.gz
    IMPLEMENTS WHEN brebis verifies a backup
    $DATADIR/brebis/brebis.py -c $DATADIR/brebis/yarn-test -l $DATADIR/brebis/yarn-test/brebis.log
    IMPLEMENTS THEN verify brebis output
    if [ -s $DATADIR/brebis/yarn-test/brebis.log ]; then return 1; else return 0; fi
2. Result of the execution of the Yarn scenario Yarn provides at the end of the scenario an overview of the execution, as show below: $ yarn brebis-scenario
Scenario test suite PASS, with 1 scenarios (4 total steps), in 16.4 seconds 3. Details of the Yarn scenarios The five first lines are basically your scenario. Each step takes a keyword and a sentence to identify a step. You define a name with the SCENARIO keyword. The next step, GIVEN, allows generally the setup of the test environment. It is possible of course to define different steps with the keywork AND. Once the environment setup has been completed, we launch the test with the keyword WHEN. At last we check the result of the execution with THEN. It is worth noting that if you need further steps after checking the result of the execution, you can use FINALLY. Very helpful while setting up the test environment is the variable $DATADIR, automatically initialized by Yarn. It provides the path to a temporary directory where you can store files during the execution of your scenario. This directory is by default removed at the end of the scenario. Now the real trick: at each step I described above is associated shell commands to execute thanks to the IMPLEMENTS keyword.It s in my opinion a great way to increase the understanding of what the scenario really does. In a short glance of the first lines, if the names of the steps are carefully chosen, you will understand what the scenario does without needing to read the actual shell code. Simple and efficient. 4. More about Yarn Yarn is written in Python and now available in Debian in the package cmdtest. The sources are available here. In my opinion, if you were looking for this kind of tool, Yarn is in a really good direction. This project is still young but the core features do the job and the upstream is eager to receive feedbacks (and patches) :) Some links: Interested in Yarn? What do you think about it? Let us know in the comments of this post.

18 November 2013

Russ Allbery: Review: Anathem

Review: Anathem, by Neal Stephenson
Publisher: William Morrow
Copyright: 2008
ISBN: 0-06-147409-6
Format: Hardcover
Pages: 935
Fraa Erasmas is a little less than twenty years old and is an avout of the decade math of the Concent of Saunt Edhar. As Anathem opens, he's approaching his first apert. All of those words are left for the reader to puzzle out for themselves, unless you cheat and read the glossary (which I would recommend doing only with caution, but more on that in a moment), since Erasmas, the first-person narrator of this story, writes (mostly) as if the reader is familiar with the mathic world and its concepts. Translated into English, it means that Erasmas was brought into something akin to a scientific monastery (the "math") at around the age of nine and has been there for ten years, since his math is a decade math and therefore remains closed to all outside (s cular) ideas and contact for periods of ten years at a time. He is still a student there, not yet declaring allegiance to one of the chapters. That's something that he will be expected to do after the first apert. Every ten years, the decade math opens its doors to the outside world for a period of ten days (apert), and the fraas and suurs can leave the math and travel through the outside world as they wish. The start of Anathem is an exercise in alienation and reorientation. Stephenson has constructed a complex society with a long history and its own specific technical jargon and throws the reader into it with a minimum of orientation. Despite being familiar with SF novels that use this technique, I did read the preface and recommend that other readers do as well unless they particularly love piecing together a world from hints and clues. The timeline, in particular, was invaluable and something I referred to throughout the story. But even with the preface, you should expect to read the first part of Anathem interpolating meanings from context or just letting things slip past until you later come to understand them. A lot of people bounce off of Anathem here, and it's one of the reasons why I'd been postponing reading it (the other being that it's a substantial doorstop and I wanted to give it my undivided attention). But it's really not as bad as I feared. One gets a feel for the terms fairly quickly, and the more difficult ones Stephenson defines as he goes with excerpts from a dictionary of the mathic world. I found it slightly disorienting for a chapter or two, and then it started growing on me. The terminology is sometimes different just for the sake of being different (jeejahs are indistinguishable from smartphones for all practical purposes), but usually the invented words provide either important links to the past of the constructed world of Arbre or are technical terms that add precision and clarity once you know their meanings. It can be a little bit frustrating to remember the mapping of famous theories and theorems to the Orth names, but only mildly. (And Diax's Rake is so useful of a name for something that English doesn't name that I may start using it myself.) This is also why I recommend against reading the glossary. The definitions of terms are linked to important parts of history, which are, in turn, linked to important parts of the plot, and the glossary therefore risks spoilers. I liked having the term introduced by the frequent dictionary excerpts scattered through the chapters, and enjoyed the feeling of the terminology unfolding with the plot. The glossary is there if you really get lost, but I think the experienced SF reader (particularly if you have enough history background to pick up on the obvious parallels and start mapping bits to Greece, Rome, the Catholic Church, etc.) will be able to navigate the language without that much difficulty. I also stopped caring about the terminology because the story is so engrossing that it pulled me right into the book, closed over me, and made Erasmas's world feel real, precious, and fascinating. Whether that's going to be true of other people is a surprisingly difficult question, and one that I'll try to tackle as part of this review. But I will say up-front that I think this is the best book that Stephenson has ever written, even better than Snow Crash, despite some undeniable flab and one extremely irritating choice. Anathem is, at its heart, a novel about the scientific method. I think it's the first book by Stephenson that elegantly mingles his tendencies (and obsessions) as a writer with the interests (and obsessions) of the characters and makes the whole novel feel coherent. All of Stephenson's books are prone to discursive infodumps, but they've usually had to be shoehorned in around the characters. Even with the Baroque Cycle, a series full of natural philosophers, the digressions are jammed in around the plot like the filling of an over-packed box and frequently stick out at odd angles or dribble out on the floor. With Anathem, Stephenson has populated a book full of characters who have completely believable and engrossing reasons to digress into scientific debate and analysis, can put them into dialogue and thereby avoid some of the strain of the infodump, and (most importantly) largely restrains himself from narrative digressions via first-person narration and careful attention to when the characters themselves would follow the same digression. In short, this is a book full of very smart and very well-educated people who figure things out from first principles using agreed-upon theory, thrown into an unknown and dangerous situation that requires a great deal of deep thinking, science, engineering, and philosophy. All of Stephenson's quirks are still here, on prominent display, but rarely have I ever seen a better match of writer, characters, plot, and world background than this book. I would go so far as to say that, if you don't like this book, you're unlikely to like any long Stephenson. I haven't said a lot about the plot. Erasmas's apert is only the first part, and barely touches on any significant plot elements. It's mostly there to set some groundwork, introduce characters, and get the reader oriented. I'm not going to say much more, since the way the novel transitions from routine to unusual to emergency, and the resulting scramble to understand the emergency, is the heart and soul of this book and should not be spoiled. (I spoiled myself slightly by reading some analysis that I shouldn't have and regretted it, although it wasn't too much of a problem.) This is one of Stephenson's best-plotted novels. It develops slowly enough for the reader (and the characters) to think hard about it and make some good and bad guesses, but fast enough to stay engrossing. Most of it is very well-paced provided that you find the background of science and philosophy of science interesting. (If you don't, this is probably not a book you're going to enjoy.) I say "most of it" because, as mentioned earlier, this book is a bit flabby. There are a couple of sections where the theory and analysis are pushed aside by exigent circumstances and more action-oriented plot elements, both of which reminded me that Stephenson is very bad at writing action sequences and both of which I wished were shorter. And there are a couple of sections where Erasmas is getting oriented to a new situation or working up the courage to do something where I wished he'd get on with it already. But for a novel of nearly 1,000 pages, the amount of tension and reader interest Stephenson maintained for me was quite impressive. But as good as the plot is, the best part of this book is its full-bore, no-apologies embrace of understanding things. This is a book about making sense of the world, and about what it means to make sense of the world, and how you go about it, and what preconditions are required to do so. It is not, contrary to so much science fiction, a book about knowing things. The people who are already experts appear in this book, but generally only in the context of figuring out new things they've not previously been exposed to. Rather, it's a book about learning things, the thrill that comes from applying theory to understand something new, and the satisfaction of working through something from first principles or previously-established theory, arriving at testable predictions, and then testing them. It's the first novel I've read that captures some of the joy and delight that I got from reading George Gamow's One Two Three... Infinity as a child. Stephenson novels have always been about showing the reader neat things he learned, but I think previous books relied on the reader to bring their own enthusiasm. Anathem, at least for me, goes a step farther and uses the passion and approach of its characters to create that enthusiasm. It's one of those books that I not only enjoyed but went away from feeling like it made me a better person in some subtle but detectable way. Sadly, this is also where the irritating part comes in. I'm going to be very careful and indirect here, since the details are all extremely significant spoilers. You may want to skip to the last paragraph of this review if you want to avoid any knowledge about the end of Anathem. Stephenson pulls several rabbits out of the hat in this book, but most of them are well-defended and nicely handled bits of misdirection. There are a few where I thought the science or engineering was dubious, but even those are usually backed with enough hand-waving about Arbre technology that I could swallow it enough for the purposes of this book. (The timeline here is very helpful in allowing one to write off a few things as unknown technology.) But he puts, at the core of the plot, one of the most scientifically dubious bits of the whole book, and there's simply no avoiding that by the end of the story. Now, this doesn't mean that he just expects the reader to swallow it. Much of Anathem is a carefully-constructed defense of the bits he uses for the plot, and the defense is not half-bad within his fictional world. There are substantial in-book justifications for believing these techniques will work on Arbre even if they wouldn't in our scientific domain. So it's not so much the plausibility that bothered me; the idea is at least as plausible as some of the FTL drive concepts that I've swallowed without complaint. I do hold Anathem to a higher standard because the book is about science in a way that most science fiction isn't, but even with that, I think Stephenson mostly managed to dance across the thin ice he built. The problem is more subtle but more serious: if taken seriously on its own terms, the approach Stephenson takes to the plot destroys science in his universe. I can't really say more than that, since all of the details are huge spoilers, but the more I thought about it, the more irritated and annoyed I got. Anathem is otherwise a brilliant defense of the scientific method, and it felt like Stephenson injected a poison pill in the center of it, leading necessarily to a world where the scientific method no longer works. This does not happen on camera, or even between the pages of the book; you have to think about the implications for a while before you realize that's what happened. But once you do, it's a bitter pill to swallow, and it's hard to escape the feeling that it fundamentally undermines Stephenson's whole project here. And, to make it worse (although also oddly better in a way), I think it was unnecessary. He would have had to reconstruct the ending, but I can't shake the feeling that there were ways to write the ending, even maintaining the same character strengths, without having to pull that particular rabbit out of the hat. Because I think it's, in the end, unnecessary, I was still able to enjoy the whole book, including the ending, despite this, but while the rest of Anathem is a wholehearted 10, that choice knocked a full point off my impression of the book. It's profoundly irritating precisely because Stephenson did such an excellent job with the rest of it. But, that said, this book is still exceptional. I think it's the book that Stephenson was meant to write. Not only did he get the overall construction nearly perfect, he gets so many of the details right, from the history (which for a while I thought was merely clever, once I started seeing the correspondences that moved it out of the realm of pure invention, but which transitioned back to profound with later revelations in the book) to the language to the way that he excerpted some theoretic digressions into very well-done appendices. If you've ever thought of a theory as beautiful, or like reading about smart people debating the nature of understanding and knowledge, I cannot recommend this book too highly. Despite the highly irritating flaw at its center, and despite a few sections that show Stephenson is not a well-rounded writer, I think it's brilliant. Rating: 9 out of 10

17 November 2013

Dmitrijs Ledkovs: libnih (upstart dependency) ported to kFreeBSD/eglibc


Back in September at the Linux Plumbers Conference in New Orleans, James Hunt and I presented Upstart roadmap, which included porting Upstart to kFreeBSD/glibc. The first milestone is now complete.

Libnih is ported to kFreeBSD/eglibc. During the course to get there the following has happened:

  • Later libnih test-suite gets a test case failure, which is then reported to FreeBSD.org. Turns out some syscalls on FreeBSD returned bogus siginfo_t.si_status information in some cases. Luckily there is now a full test-case and kernel patch posted by Jilles Tjoelker. Once that's committed upstream, I'll push for uploads to the affected kernels in Debian.


  • A few small patches applied to libnih, mostly adding POSIX compliant header includes and the like.
So with a patched libc and patched kernel one can compile libnih & run its test-suite. There are however some caveats feature miss-partiy:

  • nih_watch / inotify support is disabled. This is TODO on my list to port nih_watch to use kevent/kqueue EVFILT_VNODE.
  • FreeBSD kernel doesn't have abstract namespace domain sockets, so at the moment I have disabled some abstract namespace test-cases & changed dbus-connection tests to use pathname sockets instead. This is still a TODO because unlink before bind is not done, and the socket is not cleaned up after servers close it at the moment.




With all caveats above, here are the test-suite results:

================================================
libnih 1.0.4: nih-dbus-tool/test-suite.log
================================================

# TOTAL: 2492
# PASS: 2492
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

===========================================
libnih 1.0.4: nih-dbus/test-suite.log
===========================================

# TOTAL: 110
# PASS: 110
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

======================================
libnih 1.0.4: nih/test-suite.log
======================================

# TOTAL: 788
# PASS: 787
# SKIP: 0
# XFAIL: 1
# FAIL: 0
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

PASS: test_watch
================

not ok 1 - nih_watch not supprted yet # TODO
XFAIL: test_watch 1 - nih_watch not supprted yet # TODO

at tests/test_watch.c:1594 (main).
1..1

Joachim Breitner: Breaking News: Debian TC leaked decision on init system

According to the German news page heisse news , the Debian Technical Committee has made a decision on the question about the default init system in Debian: There will be none! Instead, according to Ian Vorlon Bart, Debian will start to distribute a suspend-to-disk image to their users, with all daemons already started, completely eradicating the need for an init system. If you are able to read German, read all about it at Debian Technical Committee entscheidet sich gegen ein Init-System

16 November 2013

Russ Allbery: Review: Wanderlust

Review: Wanderlust, by Ann Aguirre
Series: Sirantha Jax #2
Publisher: Ace
Copyright: September 2008
ISBN: 0-441-01627-8
Format: Mass market
Pages: 312
Wanderlust is the second book in the Sirantha Jax series and a direct sequel to Grimspace. There's no point in reading them out of order. The new shape of Sirantha's universe is still coming together after the climactic events of Grimspace, but they seem to involve a new job. In politics, even, but one for which she's uniquely qualified. Before she can even accept it, she finds out that multiple parties are interested in her taking this position, for widely varying reasons. But she starts her new position by taking a detour to visit some old friends (and the planet I liked least in Grimspace's settings), which turns into a war and a comprehensive distraction that lasts the rest of the book. The biggest problem with Wanderlust (and unfortunately there are several) is that it doesn't feel like a complete book. The first few chapters set up what appears to be the next step of Sirantha's adventures, as well as establishing a new villain, and then the rest of the book is an extended detour. She does learn more about the new villain, but almost everything that's set up at the start of this book is still unresolved at the end of it. The detours are moderately entertaining and (at least after an initial decision based on unfinished business from the first book) are at least related to the plot, but this feels like the setup and middle-book distractions of a book twice as long. I came away wondering why all of that was really necessary and why I had to wait to the next book to read the actual plot. The second problem with this book is the whining. And the angst. And the self-esteem issues. And did I mention the whining? Okay, I get that one of the distinguishing elements of this series is that it's space opera with an urban fantasy first-person perspective, and the latter usually comes with extensive angst and insecurity about one's love life. But, really, I am not interested in Sirantha complaining about her appearance yet again, and I am particularly not interested in the obnoxiously tedious "I love you and therefore I must push you away" dance that both she and the male romance lead do through half this book, and only stop doing because they can't any more. I realize that self-destructiveness is a thing, and that people do react this way, but I don't want to read about it, and I particularly don't want to listen to the narrator whine about how hard it all is for much of the book. Some of the appeal of Sirantha is that she's fragile, paranoid, and all sorts of messed up, but she muddles through and manages to do the right thing anyway. However, for me, the key word there is "do." When she's twitchy and angry and sarcastic and scared but doing something (which does also happen in this book), I really enjoy this series. When she's living in her own head and building elaborate plans for how she's going to destroy her life this time, it starts getting tiresome. It doesn't help that I don't see what she sees in her romantic partner (at least most of the time; he has his moments), and therefore I have even less sympathy for her grand self-sacrificing gestures. Thankfully, there are good parts too. Aguirre introduces a few new characters, all of whom I liked. Sirantha is assembling quite the ragtag band of misfits, and they're misfits in all the right sorts of ways. They play off each other well and they're full of plot hooks. We also get an amusing twist on Sirantha's relationship with her personal assistant AI, which, while bizarrely tangential to the plot, looks like it will hold a lot of promise for the future. And I love Vel, who made the end of Grimspace and who continues to be a highlight of these books, partly because he doesn't throw emotional temper tantrums or indulge in bizarre silent treatments. I always love the quietly competent characters in stories like this. This is not up to Grimspace, but hopefully it's an aberration. It's not bad, exactly, but it went off the rails. Thankfully, the next book promises quite a lot of Vel, which will hopefully be an improvement. Followed by Doubleblind. Rating: 6 out of 10

Next.

Previous.