Last Friday, after 5 long years, I have finally submitted my PhD thesis. It was quite a relief, more or less as if an elephant was taken off my back. An English title for my thesis would be Structural Complexity Characterization in Software Systems. Here is an abstract:

This thesis proposes a theory to characterize structural complexity in software systems. This theory aims to identify (i) the contribution of several factors to the structural complexity variation and (ii) the effects of structural complexity in software projects. Possible factors in the structural complexity variation include: human factors, such as general experience of the developers and their familiarity with the different parts of the system; factors related to the changes performed on the system, such as size variation and change diffusion; and organizational factors, such as the maturity of the software development process and the communication structure of the project. Effects of structural complexity include higher effort, and consequently higher cost, in software comprehension and maintenance activities. To test the validity of the proposed theory, four empirical studies were performed, mining data from free software project repositories. We analyzed historical data from changes performed in 13 systems from different application domains and written in different programming languages. The results of these studies indicated that all the factors studied influenced the structural complexity variation significantly in at least one of the projects, but different projects were influenced by different sets of factors. The models obtained were capable of describing up to 93% of the structural complexity variation in the projects analyzed. Keywords: Structural Complexity, Software Maintainance, Human factors in Software Engineering, Mining Software Repositories, Theories in Software Engineering, Empirical Software Engineering, Free/Open Source Software Projects.

Those who read Portuguese can check out the actual thesis text as a PDF file. Most of the studies discussed in the thesis are presented in English in papers I have published during the last years. My defense is going to be on March 23rd. If you happen to be at Salvador at that day, please feel cordially invited.

In the last few days, people on the debian-devel mailing list have been debating again (and again, and again...) how much the ports and architectures used only by an handful of people should influence the direction of the project. With this post I want to briefly summarize my position. The reason we support niche architectures and toy ports is that that some people are interested in doing the work, and that these do not hinder development of the architectures that people actually use. When m68k started inconveniencing too much the real world development, the port was killed and the sky did not fall, notwithstanding the complaints of the few retrocomputing enthusiasts who still used and developed it. I do not mind if some people like to play with kFreeBSD and Hurd as long as they do not inconvenience me too much, both as a user and as a developer. While it is true that targeting multiple architectures helps finding bugs, after we have covered all the useful combinations of endianity, size of variables and char signedness the incremental benefits of adopting a new architecture just for its sake are minimal. I am a Red Hat customer (also because of Debian shortcomings), I do not like it much and I do not want to be one forever: please do not kill my hopes.

For some time, we use Pootle as online translation tool for phpMyAdmin. It is indeed great tool, but somehow we seem to be hitting more and more issues with it and I feel it really won't fit nicely once we will want to support translations on two divergent branches. Also it's code base is not something easy to understand so that I would be able to fix bugs myself without major effort. So this is when I started to look for alternatives. Unfortunately there is nothing I could find what would work in way I want. I've looked at Transifex, Translatewiki, Crowdin and others whose names I've forgotten and I still haven't found what I'm looking for. The key feature I want from such system is full integration with git. This means that whatever is changed should be committed to local git branch and merging translations back should be as easy as merging remote branch. I've hacked such thing into Pootle (in quite dirty way), so this is basically workflow we have right now and I'm pretty much happy with that. Another thing (which seems to be existing in some tools) is support for simultaneous translation of more branches - if same string exists in more branches, it should be translated in all of them. And last but not least, it should support import of po files allowing users offline translation. Thinking about this I'm starting to thing that it should be pretty easy to implement something what I want using existing pieces of software without major effort. I'll probably start to play with Django, translate-tookit and GitPython to see if I can come up with something working reasonably well in few days... PS: Anybody working on updated GitPython packages for Debian, looks like I will need them soon :-).

Filed under: English Phpmyadmin Weblate 10 comments Flattr this!

The Debian/Ubuntu games team recently organised a half-day screenshots party for creating screenshots for all the games that are available in Debian/Ubuntu. Unfortunately, only 5 people participated in the screenshots party, I had hoped that we would get more folks turn up. Only one of those was not already involved in the games team and that one person found out via our spam on the #debian-mentors IRC channel. This means that most of our promotion of the event was ineffective. Hopefully for future games parties we can improve this, please let us know if you have any ideas about that. We also had 7 other people on the channel during the party lurking, discussing and or working on other things, in particular Ben Armstrong was working on a games live CD. Over the 7 hours of the party, we uploaded 100 screenshots for 40 different packages. Of the packages, 2 are available only in Debian (auralquiz, cutechess), one is available only in Ubuntu (plasma-widget-tictactoe) and the rest are available in both. I think we got quite a bit done for such a small group. Some of us could not resist filing some bugs on packages that had issues preventing or delaying us from taking screenshots. I also suggested to one maintainer that his package (acm4) be removed from Debian, since it is obsoleted by acm and unusable. If you want to play some games, check out goplay, which is a tool for browsing available games using debtags that displays screenshots from the games-thumbnails package, which has just been updated with the screenshots created during the party. goplay needs some more development, if you would like to help out with that, you would be most welcome. If you are interested to see our screenshots, they are all on the screenshots.debian.net website and the packages are auralquiz, cardstories, londonlaw, cube2font, ardentryst, cutechess, pianobooster, emacs-chess, xye, flare, tmw, garden-of-coloured-lights, flight-of-the-amazon-queen, geekcode, glpeces, fofix, unknown-horizons, klickety, polygen, tictactoe, xscreensaver-screensaver-dizzy, goban-original-games, xjokes, mudlet, xabacus, xtux-client, connectagram, crystalspace, pescetti, purity-ng, qstat, slashem, sudoku, minetest, xscavenger, plasma-widget-tictactoe, xbomb, xfrisk, xpilot-ng and xsok.

This weekend we wrapped up and published the second release candidate for Debian Edu / Skolelinux based on Squeeze. The full announcement did for some reason not make it the project announcement list, but is available from the Debian development announcement list. Check it out if you need a software solution for your school.

With first RC for phpMyAdmin 3.5, we're really close to final release, so let's again look at translations status. Here comes fourth round of translation status update. Since last update we have three new translations at 100%:

• Czech
• English (Great Britain)
• Estonian
• French
• Japanese
• Russian
• Slovenian
• Spanish
• Swedish
• Turkish

There are few which are really close to 100% and I hope they will get there soon:

• Danish
• Dutch
• Italian
• Polish

As you can see, there is still lot of languages missing, this might be your opportunity to contribute to phpMyAdmin.

Filed under: English Phpmyadmin 2 comments Flattr this!

The 13th Debian Conference is just around the corner. DebConf12 will be held in Managua, Nicaragua, July 8-14, 2012, and we need your help! Why should I care? DebConf is the annual conference for Debian contributors and interested users. Previous Debian Conferences have featured speakers and attendees from around the world (DebConf11 had attendees from 46 different countries). DebConf helps share the knowledge and experiences of individual developers with the entire community. It also helps promote enthusiasm for specialized areas of development and encourages discussions about Debian s future. The benefits that DebConf brings to the Debian project as a whole, and the larger Free Software community, will improve Debian and by extension will help improve GNU/Linux for everyone, whether they use Debian (or a Debian derivative), or any other distribution. How can I help? DebConf is a lot to organize and pay for. We invite you to help that this is going to be a successfull event.

• Since DebConf is an event organised by volunteers, its quality strongly depends on effort from these volunteers and as such every help is appreciated. Please check the Volunteering section to obtain more information
• Without money we can not organize a Debconf. Donating is a great way to help us.
• Do you want to be one of our generous sponsors? You can get nice publicity and contribute valuably to this DebConf.

Nagios or Icinga users probably know the problem that neither the default plugin packages (nagios-plugins-basic and -standard) nor the few other plugin packages ship all the plugins you need to monitor your hardware and software properly. And unless you have puppet, cfengine or some other automation software, you probably just start searching plugins on Nagios Exchange, MonitoringExchange or some other machines you are monitoring already. Your problem shall be solved! nagios-plugins-contrib passed NEW today. It is a collection of various useful plugins, maintained within the Debian Nagios Maintainer Group. So far it contains only 6 plugins (check_email_delivery, check_ipmi_sensor, check_lm_sensors, check_memcached, check_raid, check_rbl), but more are going to come. To have your favorite plugins added to the package we would like to encourage you to send pull requests, either using github, alioth or your personal repository. Please keep in mind that you'll be added to the Uploaders of the package automatically to ensure that you keep your plugin updated and in a good shape. See debian/README.source for some instructions on adding new plugins. Unmaintained plugins will be removed! Of course you can also submit bug reports to have a new plugin added, but you have to convince somebody to maintain it for you (or that he wants to use the plugin and needs to maintain it therefore ). We might also also have a similar package in contrib as a lot of commonly used plugins require non-free software, so don't hesitate to prepare plugins for inclusion in such a package and let us know! Suggestions and ideas for improvement are always welcome. And so is help to maintain the packages!

phpMyAdmin is quite popular software (to give some numbers let's mention 10000 downloads daily on SourceForge.net or 122685 reports in Debian's popcon) and as such is quite attractive target for various scripted attacks. If you run phpMyAdmin installation somewhere you should really make sure it is enough secured, so that these script kiddies don't get through. In past month I've looked at what kind of attacks are these guys trying and in all cases these are pretty old vulnerabilities, some of them fixed years ago. So the first thing you should do is to update. It is always good to run latest stable version, but in case you can not for whatever reason, try at least taking the most important fixes and using them. In ideal world your distribution would do this job for you, but in case it did not, you can for example take patches from Debian, which is pretty good at taking our patches (surprisingly it is not much related to my involvement there). To check which patches they have applied you can use excellent patch-tracker tool, which exposes patches from all released packages. To give you overview of which issues are mostly being attempted to exploit by script kiddies right now, here is the list:

• PMASA-2010-3 - yes, more than two years old, but still unpatched in some places
• PMASA-2011-5 - "only" half year old
• PMASA-2011-6 - only useful together with wrongly configured PHP

If you have fixed these, you should be pretty safe for now, but follow our security announcements for possible future issues (you can use RSS feed or subscribe to news mailing list, where all security issues are announced as well). However there are more things you can do to keep you safer:

• remove setup directory from phpMyAdmin, you will probably not use it after initial setup
• prevent access to libraries directory from browser, as it is not needed, supplied .htaccess file does this
• properly choose authentication method - cookie is probably the best choice for shared hosting
• in case you don't want all MySQL users to be able to access phpMyAdmin, you can use AllowDeny rules to limit them
• consider hiding phpMyAdmin behind authentication proxy, so that MySQL credenticals are not all users need to login

So these are the basic steps which will help you against possible compromise, I might return to some of these in more details in future posts.

Filed under: Debian English Phpmyadmin Suse 0 comments Flattr this!

Intro RAID sucks, and so do all other Free alternatives in the Linux world. Having been in exactly the same situation several times in the past, I have been following Russell Coker's posts regarding data integrity with interest. FWIW, the correct answer to "BTRFS (sic) and Xen" is: subvolumes. Now that Fedora has, once again, decided to postpone their switch to Btrfs as default file system, I decided to write up my own take on this topic. Any and all my considerations are made under the assumption that important data is backed up while semi-important data is at least living on two different machines. Beware, this is a long post, but I do like to think it's well worth reading. RAID Disk failures Let's recap the current situation for "traditional" file systems like xfs, ext 2..4 , etc. In case you need to freshen up on the technical details go here.

• Single disks: Fine for many use cases like Laptops, Desktops, etc, they are what most of us are using for most storage needs.
• RAID 0:If you are using that, you are most likely doing it wrong. Barring increasing caches for SSD-based volumes on machines which you simply take out of your cluster if there is any kind of problem, I don't know of not a sigle valid use case. Other than to get rid of data, that is.
• RAID 1: The default for small servers and important machines which don't need a lot of disk space.
• RAID 5: OK for personal storage servers, avoid once disks become larger than 500GB-1TB, depending on personal preference. To write data, you need to calculate new checksums and thus read all data from all corresponding slices, impacting write performance significantly.
• RAID 6: The RAID 10 people will disagree, but I like this RAID level best. While your write performance will definitely take a hit, you can mitigate this by decreasing your volume sizes. The extra cost in ways of power, disks, controllers, and rack space are a price we gladly pay for the ability to lose two disks and still retain all data.
• RAID 10: Great write performance until the day when two disks in the same RAID position die right after each other and your company's main mail storage dies. Yes, this has happenend to me and it sucks.
• RAID 2,3,4,50,60,foo: Mostly irrelevant in the real world; disregard.

Silent corruption All the RAID levels with redundancy are fine and dandy if you lose a whole disk and put in a new one. Of course, that does not help you the tiniest bit once you get read errors. Controllers which manage RAID 1 or 10 may or may not compare the data while it's being read and may or may not tell you about discrepancies. They can then toss a coin and give you either result. There is no way to determine which is correct. RAID 2, 3, 4, 5, and 6 could, in theory, verify the data as it's being read, but that would limit your read rate to that of a single disk so no one does that. We actually had a massive web presence fail over night, once. No one knew what the cause was as there hadn't been any kind of access logged. As that particular deployment is done directly from a VCS, we simply ran a diff and found one change. We traced the failure down to a syntax error; a one character change made everything fail. Looking at the ASCII table, it was clear that one single bit had flipped. This sucked. A lot. Mitigation So you end up scrubbing your RAID sets on a weekly basis, recovering from silent corruption with the help of your RAID 6's two parity stripes. And even though you schedule the scrubbings with the least priority, you still take a performance hit when randomly seeking a lot. ZFS Some smart people at Sun set out to fix these problems and more, and fix it they did. The "Z" in ZFS is meant to imply that this is the last file system you will ever need and if not for its license, this might have worked. Of course, "last" and "forever" mean "not now" and "10 years" respectively, in computer terms. The limits within ZFS have been chosen so that the entropy needed to create a disk able to max out a ZFS volume would, literally, require our oceans to boil. Let's just say that you won't encounter these limits any time soon. While that is a nice thing to know, it does not say anything about data integrity. There are several mechanisms ensuring data integrity built in ZFS, the most fundamental being extensive checksumming. Checksums are vastly superior to live comparisions which RAID would have to perform as it does not need to sacrifice any significant i/o reading checksum data. With HDD access, you are i/o-bound, and not CPU-bound, anyway, so performing the checksum calculations is laughably cheap when factoring in the added safety. Everything, be data, metadata, inodes, you name it, is checksummed. Every read operation must go through the checksumming functions, ensuring that all data which reaches your user space is correct. If ZFS can't verify the checksum, it will not deliver any data at all, ensuring noticeable, as opposed to silent, corruption. So how does ZFS recover when the checksum does not match the data? copies= 1..3 The easy way is to store several copies of your data within your volume. Mount a ZFS volume or subvolume with copies=2 or 3 and all data that is being written from then on will be copied twice or three times. Trivial in principle, but powerful when built into a file system. RAIDz 1..3 This is where things start to get interesting. RAIDz1, 2, and 3 will allow you to lose 1, 2, or 3 disks respectively while still retaining your data; at the obvious expense of the storage capacity of as many disks. Basically, RAIDz2 is ZFS' variant of RAID 6, but it's so much better than mere RAID 6. Moo! RAIDz will not assign fixed slices and fill them up like RAID does. It will look at the current write speeds and use appropriate slice sizes dynamically. As a direct result, data which has been written at roughly the same time will always be near other data written at the same time. This is very nice for functionality like snapshots, writeable subvolumes and other things and is called COW, ZFS' variant of super cow powers. This also enables ZFS to write data, read it back and verify the checksum and only then point to the new data. Atomic commits done right. Keep rollin' As ZFS has to keep track of what data is still in use in which volumes and subvolumes anyway, it knows which regions are free to be reclaimed. Instead of filling up random places, ZFS will roll over your disks, overwriting unused data on the fly. This bears repeating: Contrary to RAID 5 and 6, ZFS will never need to read old data in order to write new data. Write performance galore and anyone still stuck on RAID 10 can finally enjoy the increased data security. Btrfs To be completely honest, I do not know Btrfs as well as I know ZFS. If I get anything wrong, correct me. If I offend anyone with my outsider's interpretation, that is not my intention. Btrfs has been initiated by Oracle way before it ever even thought about buying Sun or history might have run a different course, for worse or a lot worse. The closing of Java, Solaris, ZFS, Hudson, OpenOffice (since then "gifted" aka thrown away), and others makes me think the latter. As there are few technological developments which caused me as much stress, overtime and pain as OCFS2, I am naturally wary of file systems sponsored by Oracle and their data focus integrity/availability. The fact that there's still no way to fsck a Btrfs volume could be birthing pain or yet another facet of Oracle's stance on this topic, I honestly don't know. Either way, it's a good idea to be wary for now. I can't say too much on Btrfs' technical underpinnings, but I know it uses COW and that it has its own variant of RAID. Toss in snapshots (writeable?), subvolumes and integrated block device management, and you have the building blocks of a decent file system. Still, Btrfs is not ready for prime time. Btrfs has been "one or two years in the future" for a few years now so I will not be holding my breath. And once the first distributions are starting to use Btrfs by default, people will lose data. A year or three after that, I will feel comfortable to use it myself and in production. Commercial solutions You can either hand a lot of money to proprietary vendors who will not tell you a thing about how things work internally (and no way to fix things directly) or you can buy solutions that employ ZFS in the back-end. I prefer my storage hardware to be relatively dumb while keeping the intelligent bits where I can see and poke them so this is not really part of these considerations. So what can you do? Test your disks before deploying them Easy, but vitally important:

disk=/dev/foo
smartctl -a $disk
smartctl -t long $disk
badblocks -swo $vendor_$model_$serial_$timestamp.badblocks.swo $disk
smartctl -a $disk

If you are lazy, get a copy of disktest, a tool I wrote to do exactly this. It should wait for the long self-test to finish, can not read out all vendor names, and does not have a way to document who ran the test, but it's a start. Patches and feedback are, obviously, welcome. And yes, I will package this soonish. Using your disks This list is surprisingly short. You can use

• distinct disks
• RAID as per above
• ZFS-FUSE
• Debian/kFreeBSD
• Nexenta

And that's it. At work, I prefer small RAID 6 volume sets. It sucks, but there is nothing better. For personal use, I have a machine with a FUSE-based RAIDz2 mounted with copies=2. Three disks can fail and my data is still secure. While this setup is slow as molasses due to FUSE, speed is not a consideration for me here; data safety is. A migration to Debian/kFreeBSD in the medium term is still likely. If you have things to add or disagree with me, I would love to hear from you. Update: Jan Christian Kaessens pointed out that, contrary to my experience, zfsonlinux did not bring down his sytem in flames. As zfsonlinux is a kernel module, potential performance is a lot better. Also, it supports ZFS Pool Version 28 as opposed to FUSE-ZFS' 23. While the really nice features are in v 29 and 30 (which are closed thanks to Oracle) v 28 still has some nice changes.

<style type="text/css"> @import "/pub/special-css/venk.css"; </style> My friend Venkatesh, pictured above, is going to bike four thousand miles, all the way across the continental US, from Baltimore to Portland. He's doing it to raise money for the Ulman Cancer Fund for Young Adults. I'm writing this because I want you to donate money to his cause. He's a DragonFly BSD developer, loves bikes, and your donation could make a big difference. I first met Venkatesh through the Johns Hopkins computer club, an ACM chapter. I was the head of the club, and he had just started his career at Hopkins. He was looking for advice on running Brickwiki, the LEGO encyclopedia. Quickly, I became his friend; in that time, I've learned the following things about him.

• He is friendly!
• He believes in science; beyond just writing sharp code, he likes to ask questions.
• He is melodramatically attracted to the power and complexity of biology. (The above quote about cancer are his words.)
• He wants to do something to make cancer less of a killer.

In the years since I graduated from Hopkins, I've been impressed by Venkatesh's ongoing curiosity and contributions to open source projects like DragonFly. I'm honored to have this chance to help him bike across the country for a good cause. Here is a quick word about the 4K for cancer effort:

Since 2002, groups of college students have undertaken a 70 day, 4000+ mile summer bike ride across the United States with the goal of offering hope, inspiration and support to cancer communities along the way. This past summer was our 10th year of cycling across the country as 76 volunteers rode along three different routes: Baltimore to San Francisco, Baltimore to Portland, and Baltimore to Seattle. Our riders raised a combined $476,000 to support organizations and individuals in the fight against cancer.

His fundraising goal is $5,000. Anything from $5 to $500 is a donation to an organization that helps young adult cancer surviers and their families get access to information and support resources. Can you help?

The new RcppArmadillo release 0.2.35 now supports the Rcpp::Rcout output stream device. Based on a contributed Rcpp patch by Jelper Ypma, the Rcpp::Rcout output stream gets redirected to R's buffered output. In other words, R's own output and that eminating from C++ code using Rcpp::Rcout are now both in sync. This avoids a stern warning from Section 5.6 in the Writing R Extensions manual:

Using C++ iostreams, as in this example, is best avoided. There is no guarantee that the output will appear in the R console, and indeed it will not on the R for Windows console. Use R code or the C entry points (*note Printing) for all I/O if at all possible.

and does in fact provide exactly what is recommended: the same entry points R itself uses. Below is a sample program, once again using the wonderful inline package to compile, load and link C++ code into R from a simple text variable submitted to the cxxfunction. What is shown in R code to load the package, the definition of the C++ code as assigned to a variable src and the creation of the dynamically-loadaded R function called fun which contains the code from we compiled, link and load via a single call to cxxfunction() given src.

library
library(inline)
src <- '
  Rcpp::Rcout << "Armadillo version: " << arma::arma_version::as_string() << std::endl;
  // directly specify the matrix size (elements are uninitialised)
  arma::mat A(2,3);
  // .n_rows = number of rows    (read only)
  // .n_cols = number of columns (read only)
  Rcpp::Rcout << "A.n_rows = " << A.n_rows << std::endl;
  Rcpp::Rcout << "A.n_cols = " << A.n_cols << std::endl;
  // directly access an element (indexing starts at 0)
  A(1,2) = 456.0;
  A.print("A:");
  // scalars are treated as a 1x1 matrix,
  // hence the code below will set A to have a size of 1x1
  A = 5.0;
  A.print("A:");
  // if you want a matrix with all elements set to a particular value
  // the .fill() member function can be used
  A.set_size(3,3);
  A.fill(5.0);
  A.print("A:");
  arma::mat B;
  // endr indicates "end of row"
  B << 0.555950 << 0.274690 << 0.540605 << 0.798938 << arma::endr
    << 0.108929 << 0.830123 << 0.891726 << 0.895283 << arma::endr
    << 0.948014 << 0.973234 << 0.216504 << 0.883152 << arma::endr
    << 0.023787 << 0.675382 << 0.231751 << 0.450332 << arma::endr;
  // print to the cout stream
  // with an optional string before the contents of the matrix
  B.print("B:");
  // the << operator can also be used to print the matrix
  // to an arbitrary stream (cout in this case)
  Rcpp::Rcout << "B:" << std::endl << B << std::endl;
  // save to disk
  B.save("B.txt", arma::raw_ascii);
  // load from disk
  arma::mat C;
  C.load("B.txt");
  C += 2.0 * B;
  C.print("C:");
  // submatrix types:
  //
  // .submat(first_row, first_column, last_row, last_column)
  // .row(row_number)
  // .col(column_number)
  // .cols(first_column, last_column)
  // .rows(first_row, last_row)
  Rcpp::Rcout << "C.submat(0,0,3,1) =" << std::endl;
  Rcpp::Rcout << C.submat(0,0,3,1) << std::endl;
  // generate the identity matrix
  arma::mat D = arma::eye<arma::mat>(4,4);
  D.submat(0,0,3,1) = C.cols(1,2);
  D.print("D:");
  // transpose
  Rcpp::Rcout << "trans(B) =" << std::endl;
  Rcpp::Rcout << trans(B) << std::endl;
  // maximum from each column (traverse along rows)
  Rcpp::Rcout << "max(B) =" << std::endl;
  Rcpp::Rcout << max(B) << std::endl;
  // maximum from each row (traverse along columns)
  Rcpp::Rcout << "max(B,1) =" << std::endl;
  Rcpp::Rcout << max(B,1) << std::endl;
  // maximum value in B
  Rcpp::Rcout << "max(max(B)) = " << max(max(B)) << std::endl;
  // sum of each column (traverse along rows)
  Rcpp::Rcout << "sum(B) =" << std::endl;
  Rcpp::Rcout << sum(B) << std::endl;
  // sum of each row (traverse along columns)
  Rcpp::Rcout << "sum(B,1) =" << std::endl;
  Rcpp::Rcout << sum(B,1) << std::endl;
  // sum of all elements
  Rcpp::Rcout << "sum(sum(B)) = " << sum(sum(B)) << std::endl;
  Rcpp::Rcout << "accu(B)     = " << accu(B) << std::endl;
  // trace = sum along diagonal
  Rcpp::Rcout << "trace(B)    = " << trace(B) << std::endl;
  Rcpp::Rcout << std::endl;
'
fun <- cxxfunction(signature(), body=src, plugin="RcppArmadillo")
setwd("/tmp")                           # adjust on other OSs
fun()                                   # output to stdout
sink("rcpparma.log.txt")                # start 'sink' to output to file
fun()                                   # no output to screen
sink()                                  # stop 'sink'

We then switch to a temporary directory (as the example code, taken from one of the two examples in Conrad's Armadillo sources, creates a temporary file) and run the new function. To demontrate how it does in fact now mesh perfectly with R, we create an output 'sink' (which catches all output) and re-run. This simple example demonstrated how we can use the new Rcout output stream from Rcpp to have dynamically-loaded C++ code cooperate more cleanly with the (buffered) R output. It also demontrated some of the nice features in Armadillo which we bring to R via RcppArmadillo.

For various reasons, having vmware running on my desktop would be kind of useful. VMware provide a Free (as in beer) version of their software called VMware Player. I downloaded the file VMware-Player-4.0.2-591240.x86_64.bundle off their website and tried to build it. It failed to build. Given my previous lack of success with VMware server, I wasn t too surprised. What was surprising was it wasn t too hard to fix it. The problem was that the vmnet module would not compile and that was due to three things:

• net device ops no longer has set_multicast_list (in netif.c)
• the linux module header needs to be included to define THIS_MODULE
• skb_frag_t has been redefined and needs an adjustment

The patch is only a few lines and means I can compile vmware on my Debian sid computer running kernel 3.2.0-1 vmnet.patch To use it, you will need to find where the modules are built, for me it is /usr/lib/vmware/modules/source

1. mv vmnet.tar vmnet2.tar
2. tar xf vmnet2.tar
3. patch -p1 < vmnet.patch
4. tar cf vmnet.tar vmnet-only

With that you can run the player which will try to build the modules and you re done!

Related articles

• imabonehead: Cross-compiling VMWare View for ARM Linux (in Debian/Ubuntu) CNXSoft Embedded Software Development (cnx-software.com)
• Virtually Run Another OS in your System (tech2copycat.wordpress.com)
• Upgrade to Oneiric breaks VMWare Player (aslakjohansen.wordpress.com)

I m in the process of getting the tests passing for the 0.03 release of NIST::NVD::Store::SQLite3 wherein our hero imports the CWE data and cross-indexes it with CVEs and CPEs. Follow along and suggest some patches. I m developing on Debian Wheezy, but I would very much like input from devs on other platforms. http://git.colliertech.org/?p=NIST-NVD-Store-SQLite3.git;a=summary

cjac@foxtrot:/tmp$ time git clone http://git.colliertech.org/git/NIST-NVD-Store-SQLite3.git
Cloning into 'NIST-NVD-Store-SQLite3'...
real	0m32.757s
user	0m0.200s
sys	0m0.088s
cjac@foxtrot:/tmp$ ls NIST-NVD-Store-SQLite3/t/data/
cwec_v2.1.xml  nvdcve-2.0-test.xml

Publish your patches and I ll fetch them, or you can submit them in udiff format and I ll review/apply. Thanks for playing along! [edit 20120216T1456 -0800]
Seems I need to update the NIST::NVD package as well.

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove  t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................read 68 entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Can't locate object method "put_idx_cwe" via package "NIST::NVD::Update" at /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve line 77.
real	0m13.072s
user	0m12.421s
sys	0m0.044s

$ time git clone http://git.colliertech.org/git/NIST-NVD.git
Cloning into 'NIST-NVD'...
real	0m2.921s
user	0m0.016s
sys	0m0.024s

[edit 20120216T16:28 -0800]

cjac@foxtrot:/usr/src/git/f5/NIST-NVD$ git log   head -5
commit 82c72a79ee810c2b5c269a15dca5151ad67059f9
Author: C.J. Adams-Collier <cjac>
Date:   Thu Feb 16 16:25:53 2012 -0800
    added put_idx_cwe to NIST::NVD::Update

[edit 20120216T1635 -0800]

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove  t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................read 68 entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Done.
real	0m13.968s
user	0m13.225s
sys	0m0.064s

Alright. before going to bed, I made those w characters above mean something. Same with some of the v characters above. [edit 20120217T10:50 -0800] Now processing the Categories from the CWE files. At this point we have parsed the Description elements. Next up are: [edit 20120217T11:35 -0800]

1. Likelihood_of_Exploit
2. Time_of_Introduction
3. Affected_Resources
4. Applicable_Platforms

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove  t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................Done.
read 68 nvd entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
read 27 cwe Views
read 1 cwe Categories
read 693 cwe Weaknesses
read 1 cwe Compound Elements
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Done.
real	0m14.306s
user	0m13.501s
sys	0m0.072s

[edit 20120217T1333 -0800] Alright, I ve got enough of the CWE processing done that I feel comfortable releasing this stuff. Let s make sure that the data got all the way down into the database. I ll write some tests in t/cwe.t to exercise the CRUD [edit 20120217T1345 -0800] Tests are in place and failing (this is good, by the way). Now to turn the tests green.

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ pushd ../NIST-NVD ; perl Makefile.PL ; make ; popd ; rm t/data/*.db *.db ; perl Makefile.PL ; make ; time prove -v -I../NIST-NVD/blib/lib -Iblib/lib t/cwe.t
/usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/svn/f5/axiom/branches/cjac/F5-Discovery
Writing Makefile for NIST::NVD
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/Base.pm (unchanged)
Skip blib/lib/NIST/NVD.pm (unchanged)
Skip blib/lib/NIST/NVD/Query.pm (unchanged)
Skip blib/lib/NIST/NVD/Update.pm (unchanged)
Skip blib/lib/NIST/NVD/Store/DB_File.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::Base.3pm
Manifying blib/man3/NIST::NVD.3pm
Manifying blib/man3/NIST::NVD::Query.3pm
Manifying blib/man3/NIST::NVD::Update.3pm
Manifying blib/man3/NIST::NVD::Store::DB_File.3pm
/usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/svn/f5/axiom/branches/cjac/F5-Discovery
rm: cannot remove  t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
t/cwe.t ..
1..10
ok 1 - use NIST::NVD::Query;
ok 2 - no error
ok 3 - constructor returned an object of correct class
not ok 4 - cve_for_cpe returned ARRAY ref
#   Failed test 'cve_for_cpe returned ARRAY ref'
#   at t/cwe.t line 36.
#          got: ''
#     expected: 'ARRAY'
not ok 5 - cwe_for_cpe returned ARRAY ref
#   Failed test 'cwe_for_cpe returned ARRAY ref'
#   at t/cwe.t line 37.
#          got: ''
#     expected: 'ARRAY'
Can't use an undefined value as an ARRAY reference at t/cwe.t line 39.
# Looks like you planned 10 tests but ran 5.
# Looks like you failed 2 tests of 5 run.
# Looks like your test exited with 2 just after 5.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 7/10 subtests 
Test Summary Report
-------------------
t/cwe.t (Wstat: 512 Tests: 5 Failed: 2)
  Failed tests:  4-5
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 10 tests but ran 5.
Files=1, Tests=5,  1 wallclock secs ( 0.02 usr  0.00 sys +  0.07 cusr  0.01 csys =  0.10 CPU)
Result: FAIL
real	0m0.442s
user	0m0.144s
sys	0m0.028s

[edit 20120220T1311 -0800] CWE load is passing. It takes 13 seconds to process all CWEs in cwec_v2.1.xml and some recent CVEs from the 13th of December of 2011. It seems I ve broken t/query.t, though. When it s green I ll push a new release. Everything is in master. Check it out if you like pain.

cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db ; pushd ../NIST-NVD/ ; perl Makefile.PL ; make ; popd ; perl -I../NIST-NVD/blib/lib Makefile.PL ; make ; PERL5LIB=../NIST-NVD/blib/lib:blib/lib prove -I../NIST-NVD/blib/lib -bv t/01-load-nvdcve.t
/usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3
Writing Makefile for NIST::NVD
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/Base.pm (unchanged)
Skip blib/lib/NIST/NVD.pm (unchanged)
Skip blib/lib/NIST/NVD/Query.pm (unchanged)
Skip blib/lib/NIST/NVD/Update.pm (unchanged)
Skip blib/lib/NIST/NVD/Store/DB_File.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::Base.3pm
Manifying blib/man3/NIST::NVD.3pm
Manifying blib/man3/NIST::NVD::Query.3pm
Manifying blib/man3/NIST::NVD::Update.3pm
Manifying blib/man3/NIST::NVD::Store::DB_File.3pm
/usr/src/git/f5/NIST-NVD-Store-SQLite3
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
t/01-load-nvdcve.t ..
1..20
ok 1 - $dist_dir is a directory
ok 2 - $test_dir is a directory
ok 3 - $data_dir is a directory
ok 4 - $convert_script is a file
ok 5 - $nvd_source_file is a file
ok 6 - $db_file does not yet exist
# running /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3 2>&1
ok 7 - took less than 15 seconds to load CWE data: 13
ok 8 - conversion script returned cleanly
ok 9 - database file exists
ok 10 - database file not empty
ok 11 - database file readable
ok 12 - database file writeable
ok 13 - database file not executable
ok 14 - file is correct type: [application/octet-stream]
ok 15 - $mtime is close
ok 16 - opened database file for reading
ok 17 - file contents indicate correct type: [application/octet-stream]
ok 18 - file contents indicate correct type: [application/octet-stream]
ok 19 - file contents indicate correct type: [SQLite 3.x database]
ok 20 - constructor returned an object of correct class
ok
All tests successful.
Files=1, Tests=20, 13 wallclock secs ( 0.02 usr  0.02 sys + 12.78 cusr  0.07 csys = 12.89 CPU)
Result: PASS

[edit 20120220T2220 -0800]
Okay, it s published. I ll need to factor the changes in to the DB_File storage engine as well.

As phpMyAdmin is approaching to 3.5 release, it has come time to share about translations status. Here comes third round of translation status update. Since last update we have one new translation at 100%, but one has dropped from the list:

• Czech
• English (Great Britain)
• Russian
• Slovenian
• Spanish
• Swedish
• Turkish

There are few which are really close to 100% and I hope they will get there soon:

• French
• Danish
• Japanese
• Polish

As you can see, traveling around FOSDEM was really useful for Czech translation, which I've managed to complete during that weekend (+ Monday when I traveled to Nuremberg). There was great amount of work done on Polish translation, so hopefully it will be on 100% in next report. Also Rouslan is progressing quite well on Italian, but there is still fair amount of work to be done. Was your language not mentioned? It's now perfect time to contribute to it.

Filed under: English Phpmyadmin 0 comments Flattr this!

As many would know, Debian GNU/Linux is one of the oldest, and the largest Linux distributions that is available for free. Since it was first released in 1993, several people have analysed the size and produced cost estimates for the project. In 2001, Jes s M. Gonz lez-Barahona et al produced an article entitled Counting Potatoes , an analysis of Debian 2.2 (code named Potato). When Potato was released in June 2003, it contained 2,800 source packages of software, totalling around 55 million lines of source code. When using David A. Wheeler s sloccount tool to apply the COCOMO model of development, and an average developer salary of US$56,000, the projected development cost that Gonz lez-Barahona calculated to start-from-scratch and build Debian 2.2 in 2003 was US$1.9 billion. In 2007 an analysis entitled Macro-level software evolution: a case study of a large software compilation by Jes s M. Gonz lez-Barahona, Gregorio Robles, Martin Michlmayr, Juan Jos Amor and Daniel M. German was released. It found that Debian 4.0 (codename Etch released April 2007) had just over 10,000 source packages of software and 288 million lines of source code. This analysis also delved into the dependencies of software packages, and the update flow between Debian release (not all packages are updated with each release). Today (February 2012) the current development version of Debian, codenamed Wheezy, contains some 17,141 source packages of software, but as it s still in development this number may change over the coming months. I analysied the source code in Wheezy, looking at the content from the original software that Debian distributes from its upstream authors without including the additional patches that Debian Developers apply to this software, or the package management scripts (used to install, configure and de-install packages). One might argue that these patches and configuration scripts are the added value of Debian, however the in my analysis I only examined the pristine upstream source code. By using David A Wheeler s sloccount tool and average wage of a developer of US$72,533 (using median estimates from Salary.com and PayScale.com for 2011) I summed the individual results to find a total of 419,776,604 source lines of code for the pristine upstream sources, in 31 programming languages including 429 lines of Cobol and 1933 lines of Modula3! In my analysis the projected cost of producing Debian Wheezy in February 2012 is US$19,070,177,727 (AU$17.7B, EUR 14.4B, GBP 12.11B), making each package s upstream source code wrth an average of US$1,112,547.56 (AU$837K) to produce. Impressively, this is all free (of cost). Zooming in on the Linux Kernel In 2004 David A. Wheeler did a cost analysis of the Linux Kernel project by itself. He found 4,000,000 source lines of code (SLOC), and a projected cost between US$175M and US$611M depending on the complexity rating of the software. Within my analysis above, I used the standard (default) complexity with the adjusted salary for 2011 (US$72K), and deducted that Kernel version 3.1.8 with almost 10,000,000 lines of source code would be worth US$540M at standard complexity, or US$1,877M when rated as complex . Another Kernel Costing in 2011 put this figure at US$3 billion, so perhaps there s some more variance in here to play with. Individual Projects Other highlights by project included:

Project	Version	Thousands of SLOC	Projected cost at US$72,533/developer/year
Samba	3.6.1	2,000	US$101 (AU$93M)
Apache	2.2.9	693	US$33.5M (AU$31M)
MySQL	5.5.17	1,200	US$64.2M (AU$59.7M)
Perl	5.14.2	669	US$32.3M (AU$30M)
PHP	5.3.9	693	US$33.5M (AU$31.1M)
Bind	9.7.3	319	US$14.8M (AU$13.8M)
Moodle	1.9.9	396	US$18.6M (AU$17.3M)
Dasher	4.11	109	US$4.8M (AU$4.4M)
DVSwitch	0.8.3.6	6	US$250K (AU$232K)

Debian Wheezy by Programming Language The upstream code that Debian distributes is written in many different languages. ANSI C with 168,536,758 is the dominant language (40% of all lines), followed by C++ at 83,187,329 (20%) and Java with 34,698,990 (8%). If you are intersted in finding the line count and cost projections for any of the 17,000+ projects, you will find them in the raw data CSV. Other Tools and Comparisons Ohcount is another source code cost analysis tool. In March 2011 Ohcount was run across Debian Sid: its results are here. In comparison, its results appear much lower than the sloccount tool. There s also the Ohloh.net Debian Estimate which only finds 55 Million source lines of code and a projected cost of US$1B. However Ohloh uses Ohcount for its estimates, and seems to be to be around 370 million SLOC missing compared to my recent analysis. Summary Over the last 10 years the cost to develop Debian has increased ten-fold. It s intersting to know that US$19 billion of software is available to use, review, extend, and share, for the bargain price of $0. If we were to add in Debian patches and install scripts then this projected figure would increase. If only more organisations would realise the potential they have before them. Need help with Linux (including Debian), Perl, or AWS? See www.jamesbromberger.com.

Late last year I wrote about the way that modern laptops suck [1]. One of the problems that inspired that post was the excessive heat generated by my Thinkpad T61. There is a partial solution to this, Fool Control explains how the kernel option pcie_aspm=force can be used on kernels from 2.6.38 onwards to solve a heat regression problem [2]. I applied this to my Thinkpad T61 and the result was that on a cool evening (ambient temperature about 24C) the temperature changed from 85C to 66C on the NVidia video card, and for the virtual devices it changed from 80C and 78C to 60C and 61C. I m not sure exactly what each of those measurements refers to, but it seems that the change was somewhere between 17C and 20C. This changes the system from being almost unbearable to use to being merely annoyingly warm. I m not going to make my laptop be my primary computing device again though, the combination of a desktop system with a 27 monitor and an Android phone is working quite well for me [3]. But I haven t yet got version control systems working for all my software. Also Wouter suggested using NBD which is something I haven t got working yet and probably won t until I can swap on it and therefore have a diskless workstation. Finally I still haven t got the Chrome to Phone browser extension working such that a page I m viewing at home can be loaded on my phone.

• [1] http://etbe.coker.com.au/2011/09/09/modern-laptops-suck/
• [2] http://foolcontrol.org/?p=1511
• [3] http://etbe.coker.com.au/2011/09/08/laptop-to-cloud-lifestyle/

Related posts:

1. Taking my Thinkpad Apart and Cooling Problems I ve been having some cooling problems with my Thinkpad recently....
2. thinkpad back from repair On Tuesday my Thinkpad was taken for service to fix...
3. I Just Bought a new Thinkpad and the Lenovo Web Site Sucks I ve just bought a Thinkpad T61 at auction for $AU796....

Talking about Secure Boot again, I'm afraid. One of the things that's made discussion of this difficult is that, while the specification isn't overly complicated, some of the outcomes aren't obvious at all until you spend a long time thinking about it. So here's some clarification on a few points.
Secure Boot provides no additional securityUntrue. Attacks against the pre-boot environment are real and increasingly common - this is a recent proof of concept, and this has been seen in the wild. Once something has got into the MBR, all bets are off. It can modify your bootloader or kernel, inserting its own code to return valid results whenever any kind of malware checker scans for it. The only way to reliably identify it is to either move the disk to another system or to cold boot off verified media. By restricting pre-OS code to something that's been signed, Secure Boot does provide enhanced security.
Secure Boot is just another name for Trusted BootUntrue. Trusted Boot requires the ability to measure the entire boot process, which gives the OS the ability to figure out everything that's been run before OS startup. The root of trust is in the hardware and a TPM is required. Secure Boot is simply a way to limit the applications that are run before the OS. Once booted, there is no way for the OS to identify what was previously booted, or even if the system was booted securely.
Microsoft are just requiring that vendors implement the specificationUntrue. Quoting from the Windows 8 Hardware Certification Requirements:

MANDATORY: No in-line mechanism is provided whereby a user can bypass Secure Boot failures and boot anyway Signature verification override during boot when Secure Boot is enabled is not allowed. A physically present user override is not permitted for UEFI images that fail signature verification during boot. If a user wants to boot an image that does not pass signature verification, they must explicitly disable Secure Boot on the target system.

Section 27.7.3.3 of version 2.3.1A of the UEFI spec explicitly permits implementations to provide a physically present user override. Whether this is a good thing or not is obviously open to argument, but the fact remains that Microsoft forbid behaviour that the specification permits.
Secure Boot can be used to implement DRMUntrue. The argument here is that Secure Boot can be used to restrict the software that a machine can run, and so can limit a system to running code that implements effective copy protection mechanisms. This isn't the case. For that to be true, there would need to be a mechanism for the OS to identify which code had been run in the pre-OS environment. There isn't. The only communication channel between the firmware and the OS is via a single UEFI variable called "SecureBoot", which may be either "1" or "0". Additionally, the firmware may provide a table to the OS containing a list of UEFI executables that failed to authenticate. It is not required to provide any information about the executables that authenticated correctly.

In both these cases, the OS is required to trust the firmware. If the firmware has been compromised in any way (such as the user turning off Secure Boot), the data provided by the firmware can be trivially modified and the OS told that everything is fine. As long as machines exist where users are permitted to disable Secure Boot, Secure Boot is not any kind of DRM scheme.
Secure Boot provides physical securityUntrue. Secure Boot does not in any way claim to improve security against attackers who have physical access, for the same reasons as the DRM case. The OS has no way to determine that the firmware's behaviour has been modified. A physically-present attacker can simply disable Secure Boot and install a piece of malware that lies to the OS about platform security. The "Evil Maid" attack still exists.
Secure Boot only defines the interaction between the firmware and the bootloader. It doesn't specify any higher policyMisleading. It's true that Secure Boot only specifies the authentication of pre-OS code. However, if you implement Secure Boot it's because you want to ensure that only authenticated code has run before your OS. If there is any way for unauthenticated code to touch the hardware before your OS starts, you can't ensure that. If an authenticated Linux kernel is booted and then loads an unsigned driver, that unsigned driver can fabricate a fake UEFI environment and then launch Windows from it. Windows would falsely believe that it has booted securely. If that authenticated Linux kernel were widely distributed, attackers could use it as an attack vector for Windows. The logical response from Microsoft would be to blacklist that kernel.

The inevitable outcome of implementing Secure Boot is that every component that can touch hardware must be signed. Anyone who implements Secure Boot without requiring that is adding no additional security whatsoever.
Only machines that want to boot Windows need to carry Microsoft's keysAgain, misleading. Microsoft only require one signing key to be installed, and the Windows bootloader will be signed with a key that chains back to this one. However, the bootloader is not the only component that must be signed. Any drivers that are carried on ROMs on plug-in cards must also be signed. One approach here would have been for all hardware vendors to have their own keys. This would have been unscalable - any shipped machine would have to carry keys for every vendor who produces PCI cards. If a machine carried an nvidia key but not an AMD one, swapping a geforce for a radeon would have resulted in the firmware graphics driver failing to load. Instead, Microsoft are providing a signing service. Vendors will be able to sign up for WHQL membership and have their UEFI drivers signed by Microsoft.

This leads to the problem. The Authenticode format used for signing UEFI objects only allows for a single signature. If a driver is signed by Microsoft, it can't be signed by anybody else. Therefore, if a system vendor wants to support off-the-shelf PCI devices with Microsoft-signed drivers, the system must carry Microsoft's key. If the same key is used as the root of trust for the driver signing and for the bootloader signing, that also means that the system will boot Windows.
This is only a problem for clients, not serversNot strictly true. While Microsoft's current requirements don't mandate the presence of Secure Boot on server hardware, if present it must be enabled and locked down as it is for clients. It's not valid for servers to ship with disabled Secure Boot support, or for it to be shipped in a mode allowing users to make automated policy modifications at OS install time.
Secure Boot is required by NISTThis is one that I've heard from multiple people working on Secure Boot. It's amazingly untrue. The document they're referring to is NIST SP800-147, which is a document that describes guidelines for firmware security - that is, what has to be done to ensure that the firmware itself is secure. This includes making sure that the OS can't overwrite the firmware and that firmware updates must be signed. It says absolutely nothing about the firmware only running signed software. Secure Boot depends upon the firmware being trusted, so these guidelines are effectively a required part of Secure Boot. But Secure Boot is not within the scope of SP800-147 at all.
It's easy for Linux to implement Secure BootMisleading. From a technical perspective, sure. From a practical perspective, not at all. I wrote about the details here.
It's only a problem for hobbyist Linux, not the real Linux marketUntrue. It's unclear whether even the significant Linux vendors can implement Secure Boot in a way that meets the needs of their customers and still allows them to boot on commodity hardware. A naive implementation removes many of the benefits of Linux for enterprise customers, such as the ability to use local modifications to micro-optimise systems for specific workloads. One of the key selling points of Linux is the ability to make use of local expertise when adapting the product for your needs. Secure Boot makes that more difficult.

ConclusionMuch reporting on the issues surrounding Secure Boot so far has been inaccurate, leading to misunderstandings about the (genuine) benefits and the (genuine) drawbacks. Any useful discussion must be based around an accurate understanding of the specification rather than statements from analysts with no real understanding of the Linux market or security.

comments

I was at FOSDEM over the last weekend, so here's my report for two weeks now:

• #445842 rmagic: "rmagic: fails with current libgd"
  add patch from Andreas Beckmann, upload to DELAYED/2
• #610332 aiccu: "/etc/pm/sleep.d/60aiccu hook may cause unacceptable resume delays"
  improve pm-utils hook script, upload to DELAYED/2, then moved to 0-day on maintainer's request
• #628305 src:gpa: "gpa: FTBFS: checking LIBASSUAN API version... does not match. want=1 got=2."
  apply upstream patch, upload to DELAYED/2
• #629693 src:spim: "spim: FTBFS: ../CPU/spim-utils.c:117:7: error: invalid use of void expression"
  add patch from Ubuntu / Matthias Klose, upload to DELAYED/2
• #629747 src:fso-abyss: "fso-abyss: FTBFS: build-dependency not installable: libvala-dev (>= 0.8.1)"
  switch to vala-0.10, inspired by the patch from Ubuntu / Barry Warsaw, upload to DELAYED/2
• #629782 src:fso-misc-vapi: "fso-misc-vapi: FTBFS: build-dependency not installable: libvala-dev"
  switch to vala-0.10, inspired by the patch from Ubuntu / Barry Warsaw, upload to DELAYED/2
• #635550 pmw: "pmw: piuparts: fails to install (update-gsfontmap again)"
  send patch to the BTS, fixed by maintainer later
• #642375 cardstories: "cardstories: ftbfs on i386 due to test value > sys.maxint"
  add patch from Ubuntu / Barry Warsaw, upload to DELAYED/3
• #642566 python-saga: "python-saga: doesn't depend on python"
  apply missing part of Jakub Wilk's patch, upload to DELAYED/2, later integrated in maintainer upload
• #650800 src:hiredis: "hiredis: FTBFS on mipsel: rm: cannot remove /tmp/redis.sock': No such file or directory"
  sponsor maintainer upload
• #652182 src:cardstories: "cardstories: FTBFS: TypeError: 'NoneType' object has no attribute '__getitem__'"
  drop "X-Python-Version: all", upload to DELAYED/3
• #652231 src:brickos: "brickos: FTBFS: semop(2): encountered an error: Invalid argument"
  close as unreproducible
• #656153 jspwiki: "jspwiki: postinst failure: chown: invalid user: tomcat6'"
  add thoughts in the BTS
• #656178 mandos: "mandos FTBFS on buildds"
  add build dependency and set locale for manpage building, upload to DELAYED/2
• #657478 biomaj-watcher: "biomaj-watcher: doesn't use invoke-rc.d"
  use invoke-rc.d in postinst (and quote variable in config), upload to DELAYED/2, later cancelled on maintainer's request
• #657746 prolix: "prolix: under- and wrongly-documented, not working interactively at all"
  upload new upstream release (pkg-perl)
• #658394 src:libio-socket-socks-perl: "libio-socket-socks-perl: FTBFS - test failures without network (was: [Bug 924169] [NEW] libio-socket-socks-perl fails to build without a network connection)"
  upload package prepared by Jonathan Yu (pkg-perl)