Search Results: "ed"

18 February 2012

Dirk Eddelbuettel: Using Rcout with Rcpp / RcppArmadillo to coordinate output with R

The new RcppArmadillo release 0.2.35 now supports the Rcpp::Rcout output stream device. Based on a contributed Rcpp patch by Jelper Ypma, the Rcpp::Rcout output stream gets redirected to R's buffered output. In other words, R's own output and that eminating from C++ code using Rcpp::Rcout are now both in sync. This avoids a stern warning from Section 5.6 in the Writing R Extensions manual:
Using C++ iostreams, as in this example, is best avoided. There is no guarantee that the output will appear in the R console, and indeed it will not on the R for Windows console. Use R code or the C entry points (*note Printing) for all I/O if at all possible.
and does in fact provide exactly what is recommended: the same entry points R itself uses. Below is a sample program, once again using the wonderful inline package to compile, load and link C++ code into R from a simple text variable submitted to the cxxfunction. What is shown in R code to load the package, the definition of the C++ code as assigned to a variable src and the creation of the dynamically-loadaded R function called fun which contains the code from we compiled, link and load via a single call to cxxfunction() given src. 
library
library(inline)
src <- '
  Rcpp::Rcout << "Armadillo version: " << arma::arma_version::as_string() << std::endl;
  // directly specify the matrix size (elements are uninitialised)
  arma::mat A(2,3);
  // .n_rows = number of rows    (read only)
  // .n_cols = number of columns (read only)
  Rcpp::Rcout << "A.n_rows = " << A.n_rows << std::endl;
  Rcpp::Rcout << "A.n_cols = " << A.n_cols << std::endl;
  // directly access an element (indexing starts at 0)
  A(1,2) = 456.0;
  A.print("A:");
  // scalars are treated as a 1x1 matrix,
  // hence the code below will set A to have a size of 1x1
  A = 5.0;
  A.print("A:");
  // if you want a matrix with all elements set to a particular value
  // the .fill() member function can be used
  A.set_size(3,3);
  A.fill(5.0);
  A.print("A:");
  arma::mat B;
  // endr indicates "end of row"
  B << 0.555950 << 0.274690 << 0.540605 << 0.798938 << arma::endr
    << 0.108929 << 0.830123 << 0.891726 << 0.895283 << arma::endr
    << 0.948014 << 0.973234 << 0.216504 << 0.883152 << arma::endr
    << 0.023787 << 0.675382 << 0.231751 << 0.450332 << arma::endr;
  // print to the cout stream
  // with an optional string before the contents of the matrix
  B.print("B:");
  // the << operator can also be used to print the matrix
  // to an arbitrary stream (cout in this case)
  Rcpp::Rcout << "B:" << std::endl << B << std::endl;
  // save to disk
  B.save("B.txt", arma::raw_ascii);
  // load from disk
  arma::mat C;
  C.load("B.txt");
  C += 2.0 * B;
  C.print("C:");
  // submatrix types:
  //
  // .submat(first_row, first_column, last_row, last_column)
  // .row(row_number)
  // .col(column_number)
  // .cols(first_column, last_column)
  // .rows(first_row, last_row)
  Rcpp::Rcout << "C.submat(0,0,3,1) =" << std::endl;
  Rcpp::Rcout << C.submat(0,0,3,1) << std::endl;
  // generate the identity matrix
  arma::mat D = arma::eye<arma::mat>(4,4);
  D.submat(0,0,3,1) = C.cols(1,2);
  D.print("D:");
  // transpose
  Rcpp::Rcout << "trans(B) =" << std::endl;
  Rcpp::Rcout << trans(B) << std::endl;
  // maximum from each column (traverse along rows)
  Rcpp::Rcout << "max(B) =" << std::endl;
  Rcpp::Rcout << max(B) << std::endl;
  // maximum from each row (traverse along columns)
  Rcpp::Rcout << "max(B,1) =" << std::endl;
  Rcpp::Rcout << max(B,1) << std::endl;
  // maximum value in B
  Rcpp::Rcout << "max(max(B)) = " << max(max(B)) << std::endl;
  // sum of each column (traverse along rows)
  Rcpp::Rcout << "sum(B) =" << std::endl;
  Rcpp::Rcout << sum(B) << std::endl;
  // sum of each row (traverse along columns)
  Rcpp::Rcout << "sum(B,1) =" << std::endl;
  Rcpp::Rcout << sum(B,1) << std::endl;
  // sum of all elements
  Rcpp::Rcout << "sum(sum(B)) = " << sum(sum(B)) << std::endl;
  Rcpp::Rcout << "accu(B)     = " << accu(B) << std::endl;
  // trace = sum along diagonal
  Rcpp::Rcout << "trace(B)    = " << trace(B) << std::endl;
  Rcpp::Rcout << std::endl;
'

fun <- cxxfunction(signature(), body=src, plugin="RcppArmadillo")
setwd("/tmp")                           # adjust on other OSs
fun()                                   # output to stdout
sink("rcpparma.log.txt")                # start 'sink' to output to file
fun()                                   # no output to screen
sink()                                  # stop 'sink'
We then switch to a temporary directory (as the example code, taken from one of the two examples in Conrad's Armadillo sources, creates a temporary file) and run the new function. To demontrate how it does in fact now mesh perfectly with R, we create an output 'sink' (which catches all output) and re-run. This simple example demonstrated how we can use the new Rcout output stream from Rcpp to have dynamically-loaded C++ code cooperate more cleanly with the (buffered) R output. It also demontrated some of the nice features in Armadillo which we bring to R via RcppArmadillo.

Andrea Veri: Nagios XMPP Notifications for GTalk

While improving GNOME s servers Nagios Notifications, I ended up working on a nice way to notify the relevant folks through GTalk in case something could go wrong on any of the hosted services. Looking around on the web, I found Seth Vidal s script, modified it to suit my needs and made it working with GTalk, here s the result: 
#!/usr/bin/python -tt
import warnings
warnings.simplefilter("ignore")
import xmpp
from xmpp.protocol import Message
from optparse import OptionParser
import ConfigParser
import sys
import os
parser = OptionParser()
opts, args = parser.parse_args()
if len(args) < 1:
    print "xmppsend message [to whom, multiple args]"
    sys.exit(1)
msg = args[0]
msg = msg.replace('\\n', '\n')
# Connect to the server
c  =  xmpp.Client('gmail.com')
c.connect( ( 'talk.google.com', 5223 ) )
# Authenticate to the server
jid  =  xmpp.protocol.JID( 'example@gmail.com' )
c.auth( jid.getNode( ), 'yourgmailpassword' )
if len(args) < 2:
    r = c.getRoster()
    for user in r.keys():
        if user == username:
            continue
        c.send(Message(user, '%s' % msg))
else:
    for user in args[1:]:
        c.send(Message(user, '%s' % msg))
I, then, added the command definitions on the relevant Nagios configuration file: 
define command 
        command_name    host-notify-by-xmpp
        command_line    /home/user/bin/xmppsend "Host '$HOSTALIAS$' is $HOSTSTATE$ - Info : $HOSTOUTPUT$" $CONTACTPAGER$
         
define command 
        command_name    notify-by-xmpp
        command_line    /home/user/bin/xmppsend "$NOTIFICATIONTYPE$ $HOSTNAME$ $SERVICEDESC$ $SERVICESTATE$ $SERVICEOUTPUT$ $LONGDATETIME$" $CONTACTPAGER$
And in the end on contacts.cfg: 
define contact  
        contact_name    admin
        use             generic-contact
        alias           Full Name
        email           example@gmail.com
        pager           example@gmail.com
        service_notification_commands   notify-by-xmpp
        host_notification_commands      host-notify-by-xmpp
When done just reload the configuration files with: 
sudo /etc/init.d/nagios3 reload
Enjoy your new XMPP Nagios notifications! Update: if you don t want the script to store your username or password, you can use the following modified script together with a nice config file like this one: 
[xmpp_nagios]
username=example@gmail.com
password=yourgmailpassword
Then you can invoke xmppsend this way: 
xmppsend -a config.ini

Neil Williams: Introducing ladder

We needed a new package at work and once I'd written it, I realised that it could well be useful for others, so I did the ITP and the package is now in Debian unstable.

Ladder - Stepwise repository upgrade tool


Ladder creates a local repository of packages required

to upgrade a tarball of a chroot to a new milestone or

software release.

.

The repository can be signed and includes all specified

target packages and dependencies. The repository can then

be distributed and used to upgrade multiple devices in

sequence without needing network access.


The only dependencies are some simple perl modules, reprepro for the repository, apt and gnupg for the signing. None of those need to be particularly recent versions, so once ladder has had a little time in testing, I will be doing a backport to Squeeze. Lenny is a little less likely but eminently possible if someone specifically asks for it. Not that it particularly needs a backport mind, if you want to run ladder on some other Debian-based system, it's simply a case of installing the version in unstable. reprepro was in Lenny at a suitable version for what ladder requires, so any system remotely recent should be able to run ladder without changes.

Normally, whenever you need to upgrade a device, you need to get that device onto a network, get access over the serial port or some form of pre-configured network connection, run the commands and clean up afterwards. OK, now do that again... oh and here are another 40 or 100 devices which all need precisely the same thing done and they all have to be shipped to paying customers today. What does any geek do? Script it.

Ladder is part of that process. There would need to be some scripting / programming support on the devices concerned but ladder makes it easier to put that support in place at the design stage and then automate the actual update without the device needing to get onto a network and, potentially, allowing you to decide how to offer the upgrade (automatic, user involvement, engineers only etc.) Once that is implemented, the device is simply pointed at a locally mounted filesystem which contains the ladder step for the migration. Each step is a SecureApt signed repository which is available at a deterministic location which can be easily scripted and which contains only the packages necessary (with dependencies) to upgrade any device installed at Milestone A to Milestone B. Nice and small, only the stuff you need, whatever architecture you like and if you need to migrate from Milestone A to Milestone D, then you create a few steps on the ladder and automate the migration from A to B to C to D. All you need is enough space on whatever local media you need to use to plug into the devices. Then copy the media enough times, start the interface on each device and do something useful whilst the upgrade happens. Then, when the next lot need to be upgraded, you already have the media... Oh and ladder does not need to run on the same architecture as the devices - all package handling is entirely architecture-neutral. So create the ladder steps and media on your fast x86 machines and then let the slower SSD devices take their own sweet time via automation. Simple - maybe, hopefully.

I'm expressly talking about devices here because this is where the original requirement arose - embedded devices which don't have direct / automatic / accessible network ports or even a working network configuration preset. These aren't new images to flash onto the device either, these are apt repositories, so only the stuff which needs to be changed is changed which saves a lot of time writing to slow SSD storage.

If the mechanism sounds familiar, yes, it's using the same principles as xapt, multistrap and emdebian-grip but I thought it could be useful, so I generalised the interface (a bit) and uploaded.

Let me know via the BTS if you need tweaks to the support. Ladder isn't particularly about getting desktop machines from Debian Potato to Debian Wheezy because you need to identify a target package to be the top point of the dependency chain which gets into the step repository and all devices should start with the same packages at the same versions. Normally, the kind of devices I use at work have a single top level application which is the sole user interface and which directly or indirectly depends on all the other software required for that device. Whilst the target package value could take a few carefully selected packages instead of just one and there is support for an extra packages field, it's not intended / expected to be useful for desktops. This is for use with multiple devices which all have the same package selection, are at the same milestone at the start and which all need to migrate smoothly to whatever milestone is intended - one step at a time. Ladder could be used to migrate between Debian releases and I've included example config files to support that, but the principle usage is with internal / proprietary repositories based on a common Debian stable platform where the user interface software is managed via identifiable milestones and where users have no direct control over installing packages. It's a production / manufacturing support tool more than a user / admin support tool. If you want to manage servers and desktops which allow users to install (or request to be installed) arbitrary packages, there are plenty of tools which already support this and are used by DSA in Debian for exactly these tasks. Ladder is not puppet but Linp didn't sound like a good name...

Ladder step repositories only include the binary packages, not sources - so if the migration is not going to be done in-house and you're expecting to distribute these steps to users somehow, ensure that the original source repository is available online for the Debian / free software packages concerned. If you can't do that (because the source is proprietary), you possibly shouldn't be distributing the binaries as a user update mechanism in the first place because each step will include packages from the core Debian system which need to have the sources available when distributed.

Inevitably, the ladder source package already contains two POT files for translations of the runtime messages and the POD based manpage. I expect to need to expand / clarify the manpage in due course, so don't rush to translate it yet as it's likely to change.

Mike Hommey: Debian Mozilla news

Here are the few noteworthy news about Mozilla packages in Debian:

Mike Hommey: How to waste a lot of space without knowing

const char *foo = "foo";
This was recently mentioned on bugzilla, and the problem is usually underestimated, so I thought I would give some details about what is wrong with the code above. The first common mistake here is to believe foo is a constant. It is a pointer to a constant. In practical ELF terms, this means the pointer lives in the .data section, and the string constant in .rodata. The following code defines a constant pointer to a constant:
const char * const foo = "foo";
The above code will put both the pointer and the string constant in .rodata. But keeping a constant pointer to a constant string is pointless. In the above examples, the string itself is 4 bytes (3 characters and a zero termination). On 32-bits architectures, a pointer is 4 bytes, so storing the pointer and the string takes 8 bytes. A 100% overhead. On 64-bits architectures, a pointer is 8 bytes, putting the total weight at 12 bytes, a 200% overhead. The overhead is always the same size, though, so the longer the string, the smaller the overhead, relatively to the string size. But there is another, not well known, hidden overhead: relocations. When loading a library in memory, its base address varies depending on how many other libraries were loaded beforehand, or depending on the use of address space layout randomization (ASLR). This also applies to programs built as position independent executables (PIE). For pointers embedded in the library or program image to point to the appropriate place, they need to be adjusted to the base address where the program or library is loaded. This process is called relocation. The relocation process requires information which is stored in .rel.* or .rela.* ELF sections. Each pointer needs one relocation. The relocation overhead varies depending on the relocation type and the architecture. REL relocations use 2 words, and RELA relocations use 3 words, where a word is 4 bytes on 32-bits architectures and 8 bytes on 64-bits architectures. On x86 and ARM, to mention the most popular 32-bits architectures nowadays, REL relocations are used, which makes a relocation weigh 8 bytes. This puts the pointer overhead for our example string to 12 bytes, or 300% of the string size. On x86-64, RELA relocations are used, making a relocation weigh 24 bytes! This puts the pointer overhead for our example string to 32 bytes, or 800% of the string size! Another hidden cost of using a pointer to a constant is that every time it is used in the code, there will be pointer dereference. A function as simple as
int bar() return foo;
weighs one instruction more when foo is defined const char *. On x86, that instruction weighs 2 bytes. On x86-64, 3 bytes. On ARM, 4 bytes (or 2 in Thumb). That weight can vary depending on the additional instructions required, but you get the idea: using a pointer to a constant also adds overhead to the code, both in time and space. Also, if the string is defined as a constant instead of being used as a literal in the code, chances are it s used several times, multiplying the number of such instructions. Update: Note that in the case of const char * const, the compiler will optimize these instruction and avoid reading the pointer, since it s never going to change. The symbol for foo is also exported, making it available from other libraries or programs, which might not be required, but also adds its own overhead: an entry in the symbols table (5 words), an entry in the string table for the symbol name (strlen("foo") + 1) and an entry in the symbols hash chain table (4 bytes if only one type of hash table (sysv or GNU) is present, 8 if both are present), and possibly an entry in the symbols hash bucket table, depending on the other exported symbols (4 or 8 bytes, as chain table). It can also affect the size of the bloom filter table in the GNU symbol hash table. So here we are, with a seemingly tiny 3 character string possibly taking 64 bytes or more! Now imagine what happens when you have an array of such tiny strings. This also doesn t only apply to strings, it applies to any kind of global pointer to constants. In conclusion, using a definition like
const char *foo = "foo";
is almost never what you want. Instead, you want to use one of the following forms:

Craig Small: VMware Player on Debian

For various reasons, having vmware running on my desktop would be kind of useful. VMware provide a Free (as in beer) version of their software called VMware Player. I downloaded the file VMware-Player-4.0.2-591240.x86_64.bundle off their website and tried to build it. It failed to build. Given my previous lack of success with VMware server, I wasn t too surprised. What was surprising was it wasn t too hard to fix it. The problem was that the vmnet module would not compile and that was due to three things: The patch is only a few lines and means I can compile vmware on my Debian sid computer running kernel 3.2.0-1 vmnet.patch To use it, you will need to find where the modules are built, for me it is /usr/lib/vmware/modules/source
  1. mv vmnet.tar vmnet2.tar
  2. tar xf vmnet2.tar
  3. patch -p1 < vmnet.patch
  4. tar cf vmnet.tar vmnet-only
With that you can run the player which will try to build the modules and you re done!
Related articles

Christian Perrier: 2012 update 10 for Debian Installer localization

Status for D-I level 1 (core D-I files): Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.): Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader) Full 100% completeness (hall of fame) for 0 languages

Benjamin Mako Hill: Advice for Prospective Doctoral Students

There is tons of advice on the Internet (e.g., on the academic blogs I read) for prospective doctoral students. I am very happy with my own graduate school choices but I feel that I basically got lucky. Few people are saying the two things I really wish someone had told me before I made the decision to get a PhD: Most People Getting Doctorates Probably Shouldn't In most fields, the only thing you need a PhD for is to become a professor -- and even this requirement can be flexible. You can have almost any job in any company or non-profit without a PhD. You can teach without a PhD. You can write books without a PhD. You can do research and work in thinktanks without a PhD. You don't even always need a PhD to grant PhDs to other people: two of my advisors at the Media Lab supervised PhD work but did not have doctorates themselves! Becoming a tenured professor is more difficult without a doctorate, but it is not impossible. There are grants and jobs outside of universities that require doctorates, but not nearly as many as most people applying for PhDs programs think. Getting a doctorate can even hurt: If you want to work in a company or non-profit, you are usually better off with 4-6 years of experience doing the kind of work you want to do than with the doctorate and the less relevant experience of getting one. Starting salaries for people with doctorates are often higher than for people with masters degrees. But salaries for people with masters degrees and 5 years of experience are even higher -- and that's before you take into account the opportunity costs of working for relatively low graduate student wages for half a decade. PhD take an enormous amount of time and, in most programs, you spend a huge amount of this time doing academic busy work, teaching, applying for grants or fellowships, and writing academic papers that very few people read. These are skills you'll need to be a successful professor. They are useful skills for other jobs too, but not as useful as the experience of actually doing those other jobs for the time it takes to get the degree. Evaluating Graduate Programs If you are still convinced you need a doctorate, or any graduate degree for that matter, you will need to pick a program. Plenty of people will offer advice on how to pick the right program and trying to balance all the complicated and contradictory advice can be difficult. Although I love my program and advisors, I've known many less happy students. Toward that end, there are two pieces of meta-advice that I wish everybody was told before they applied:

  1. Find recent graduates of the program you are considering, and the faculty advisor(s) you are planning on working with, and look at where they are now. Are these ex-students doing the kind of work that you want to do? Are they at great programs at great universities? Chances are good that a PhD program and its faculty will prepare future students to be like, and do work like, the students they have trained in the past. Programs that consistently make good placements are preparing their students well, supporting them, making sure they have the resources necessary to do good work, and helping their students when they are on the job market. A program whose students do poorly, or just end doing work that isn't like the kind you want to do, will probably fail you too.

  2. If recent graduates seem to be generally successful and doing the kind of work you want to do, find one who looks most like the kind of academic you want to become and talk to them about their experience. Chances are, your faculty advisors will overlap with theirs and your experience will be similar. Ex-students can tell you the strengths of weaknesses of the program you are considering and what to watch out for. If they had a horrible experience, there's a decent chance you will too, and they will tell you so.

Doing these two things means you don't have to worry about trying to think of all the axes on which you want to evaluate a program or pour through admissions material which is only tangentially connected to the reality you'll live for a long time. What matters most is the outcomes, of course, because you're be living the rest of your life for a lot longer than you'll be in the PhD program.

17 February 2012

Dirk Eddelbuettel: RcppArmadillo 0.2.35

Now that Rcpp 0.9.10 is released and on CRAN, other packages can take advantage of a small change needed to make use of the quasi-output stream Rcpp::Rcout. So the new release 0.2.35 of RcppArmadillo does just that---and input/output from Armadillo, the wonderful linear algebra / math library for C++ written chiefly by Conrad Sanderson now comes through in a coordinated buffered fashion. As as std::cout is no longer used by default, the R CMD check no longer flags this. There are two more bugfixes in the releases for issues noticed by attentive RcppArmadillo users. Teo Guo Ci spotted a missing semicolon when C++0x code was activated, and Gershon Bialer suggested a fix for an issue plagueing users of the jurassic g++ 4.2.1 compiler which Apple forces on users of OS X. We thank Martin Renner for testing these fixes. The NEWS entry below summarises the changes. 
0.2.35  2012-02-17
    o   Upgraded to Armadillo release 2.4.3
          * Support for ARMA_DEFAULT_OSTREAM using Rcpp::Rcout added
    o   Minor bug fix release improving corner cases affecting builds:
          * Missing semicolon added in Mat_meat (when in C++0x mode), with
            thanks to Teo Guo Ci 
          * Armadillo version vars now instantiated in RcppArmadillo.cpp
            which helps older g++ versions, with thanks to Gershon Bialer
          * Thanks also to Martin Renner for testing these changes
          * Unit tests output fallback directory changed per Brian Ripley's
            request to not ever use /tmp
          * Minor update to version numbers in RcppArmadillo-package.Rd
Courtesy of CRANberries, there is also a diffstat reports for 0.2.35 relative to 0.2.34 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

Joey Hess: addicted to $

One of the weird historical accidents of programming languages is that so many of them use $ for important things. The reason is just that out of the available punctuation, nearly all of it has a mathmatical or other predefined use that makes sense to retain in a programming language context, while $ (and also @ and #) do not. Still, $ annoys me, it's so asymetric that we use it all over our code, and never a or to be seen. The one language that manages to use $ nicely, IMHO, is Haskell. Recently I noticed that it has an actual visual mnemonic in its use of $. And it's used for something I've not seen in other languages. The visual mnemonic of $ is that it looks like an opening parenthesis, with the related closing parenthesis on a line below it. 
(something (that
    (lisp folks
        (are (very (familiar with)))
    )
))
And this is also the problem that $ solves: 
something $ that $
    haskell folks $
        are $ very $ familiar with
This is a trivial feature.. but oh so useful. The implementation in Haskell of $ is simply: 
f $ x = f x
infixr 0 $
Just function application, but at a different precedence than usual. I am now very addicted to my $. Out of 15 thousand lines of code, only 87 contain )), while 10% use $.

Dirk Eddelbuettel: Rcpp 0.9.10

A new release 0.9.10 of Rcpp is now on CRAN and in Debian. This is mostly internal release with a little bit of code reorgination (some of which will be used by a forthcoming RcppArmadillo release), some changes to make R CMD check happy, and one or two new features. The complete NEWS entry for 0.9.10 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page. 
0.9.10  2012-02-16
    o   Rearrange headers so that Rcpp::Rcout can be used by RcppArmadillo et al
    o   New Rcpp sugar function mapply (for limited to two or three input vectors)
    o   Added custom version of the Rcpp sugar diff function for numeric vectors
        skipping unncesserry checks for NA
    o   Some internal code changes to reflect changes and stricter requirements
        in R CMD check in the current R-devel versions
    
    o   Corrected fixed-value initialization for IntegerVector (with thanks to
        Gregor Kastner for spotting this)
    o   New Rcpp-FAQ entry on simple way to a set compiler option for cxxfunction
Thanks to CRANberries, you can also look at a diff to the previous release 0.9.9. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Andreas Barth: Sch fflertanz

Every 7 years with the Sch fflertanz the end of the Black Death in 1517 is celebrated. After lots of people died, everyone was too scared to go out in the streets again, even after the Black Death was gone. The Sch ffler (cooper = people who traditionally build barrels) started to cheer the people in Munich up with their dance, and made them go on the streets and start their normal lifes again. This is one of the few local traditions that even survived the modern times. The barrel is signed with "Sch fflertanz 2012" and "Zur Erinnerung an das Pestjahr 1517" = "To remind of the Black Death-year 1517", and should remind us that even in the darkest times, there is still hope and life goes on.

Ben Hutchings: Another step forward for multi-arch

~$ dpkg --print-architecture
i386
~$ dpkg -s linux-image-$(uname -r)
Package  linux-image-3.2.0-1-amd64' is not installed and no info is available.
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.
~$ dpkg -s linux-image-$(uname -r):amd64
Package: linux-image-3.2.0-1-amd64
Status: install ok installed
Priority: optional
Section: kernel
Installed-Size: 104822
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: amd64
Source: linux-2.6
Version: 3.2.6-1
...

16 February 2012

C.J. Adams-Collier: NIST::NVD CWE development follow along

I m in the process of getting the tests passing for the 0.03 release of NIST::NVD::Store::SQLite3 wherein our hero imports the CWE data and cross-indexes it with CVEs and CPEs. Follow along and suggest some patches. I m developing on Debian Wheezy, but I would very much like input from devs on other platforms. http://git.colliertech.org/?p=NIST-NVD-Store-SQLite3.git;a=summary 
cjac@foxtrot:/tmp$ time git clone http://git.colliertech.org/git/NIST-NVD-Store-SQLite3.git
Cloning into 'NIST-NVD-Store-SQLite3'...
real	0m32.757s
user	0m0.200s
sys	0m0.088s
cjac@foxtrot:/tmp$ ls NIST-NVD-Store-SQLite3/t/data/
cwec_v2.1.xml  nvdcve-2.0-test.xml
Publish your patches and I ll fetch them, or you can submit them in udiff format and I ll review/apply. Thanks for playing along! [edit 20120216T1456 -0800]
Seems I need to update the NIST::NVD package as well. 
cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove  t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................read 68 entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Can't locate object method "put_idx_cwe" via package "NIST::NVD::Update" at /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve line 77.
real	0m13.072s
user	0m12.421s
sys	0m0.044s

$ time git clone http://git.colliertech.org/git/NIST-NVD.git
Cloning into 'NIST-NVD'...
real	0m2.921s
user	0m0.016s
sys	0m0.024s
[edit 20120216T16:28 -0800] 
cjac@foxtrot:/usr/src/git/f5/NIST-NVD$ git log   head -5
commit 82c72a79ee810c2b5c269a15dca5151ad67059f9
Author: C.J. Adams-Collier <cjac>
Date:   Thu Feb 16 16:25:53 2012 -0800
    added put_idx_cwe to NIST::NVD::Update
[edit 20120216T1635 -0800] 
cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove  t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................read 68 entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Done.
real	0m13.968s
user	0m13.225s
sys	0m0.064s
Alright. before going to bed, I made those w characters above mean something. Same with some of the v characters above. [edit 20120217T10:50 -0800] Now processing the Categories from the CWE files. At this point we have parsed the Description elements. Next up are: [edit 20120217T11:35 -0800]
  1. Likelihood_of_Exploit
  2. Time_of_Introduction
  3. Affected_Resources
  4. Applicable_Platforms
cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3
rm: cannot remove  t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
using store [SQLite3]
reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................Done.
read 68 nvd entries
Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone.
read 27 cwe Views
read 1 cwe Categories
read 693 cwe Weaknesses
read 1 cwe Compound Elements
Writing CPE URNs to disk...Done.
Writing NVD entries to disk....................................................................... Done.
Writing CPE index to disk...Done.
Writing CWE index to disk...Done.
real	0m14.306s
user	0m13.501s
sys	0m0.072s
[edit 20120217T1333 -0800] Alright, I ve got enough of the CWE processing done that I feel comfortable releasing this stuff. Let s make sure that the data got all the way down into the database. I ll write some tests in t/cwe.t to exercise the CRUD [edit 20120217T1345 -0800] Tests are in place and failing (this is good, by the way). Now to turn the tests green. 
cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ pushd ../NIST-NVD ; perl Makefile.PL ; make ; popd ; rm t/data/*.db *.db ; perl Makefile.PL ; make ; time prove -v -I../NIST-NVD/blib/lib -Iblib/lib t/cwe.t
/usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/svn/f5/axiom/branches/cjac/F5-Discovery
Writing Makefile for NIST::NVD
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/Base.pm (unchanged)
Skip blib/lib/NIST/NVD.pm (unchanged)
Skip blib/lib/NIST/NVD/Query.pm (unchanged)
Skip blib/lib/NIST/NVD/Update.pm (unchanged)
Skip blib/lib/NIST/NVD/Store/DB_File.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::Base.3pm
Manifying blib/man3/NIST::NVD.3pm
Manifying blib/man3/NIST::NVD::Query.3pm
Manifying blib/man3/NIST::NVD::Update.3pm
Manifying blib/man3/NIST::NVD::Store::DB_File.3pm
/usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/svn/f5/axiom/branches/cjac/F5-Discovery
rm: cannot remove  t/data/*.db': No such file or directory
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
t/cwe.t ..
1..10
ok 1 - use NIST::NVD::Query;
ok 2 - no error
ok 3 - constructor returned an object of correct class
not ok 4 - cve_for_cpe returned ARRAY ref
#   Failed test 'cve_for_cpe returned ARRAY ref'
#   at t/cwe.t line 36.
#          got: ''
#     expected: 'ARRAY'
not ok 5 - cwe_for_cpe returned ARRAY ref
#   Failed test 'cwe_for_cpe returned ARRAY ref'
#   at t/cwe.t line 37.
#          got: ''
#     expected: 'ARRAY'
Can't use an undefined value as an ARRAY reference at t/cwe.t line 39.
# Looks like you planned 10 tests but ran 5.
# Looks like you failed 2 tests of 5 run.
# Looks like your test exited with 2 just after 5.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 7/10 subtests 
Test Summary Report
-------------------
t/cwe.t (Wstat: 512 Tests: 5 Failed: 2)
  Failed tests:  4-5
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 10 tests but ran 5.
Files=1, Tests=5,  1 wallclock secs ( 0.02 usr  0.00 sys +  0.07 cusr  0.01 csys =  0.10 CPU)
Result: FAIL
real	0m0.442s
user	0m0.144s
sys	0m0.028s
[edit 20120220T1311 -0800] CWE load is passing. It takes 13 seconds to process all CWEs in cwec_v2.1.xml and some recent CVEs from the 13th of December of 2011. It seems I ve broken t/query.t, though. When it s green I ll push a new release. Everything is in master. Check it out if you like pain. 
cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db ; pushd ../NIST-NVD/ ; perl Makefile.PL ; make ; popd ; perl -I../NIST-NVD/blib/lib Makefile.PL ; make ; PERL5LIB=../NIST-NVD/blib/lib:blib/lib prove -I../NIST-NVD/blib/lib -bv t/01-load-nvdcve.t
/usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3
Writing Makefile for NIST::NVD
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/Base.pm (unchanged)
Skip blib/lib/NIST/NVD.pm (unchanged)
Skip blib/lib/NIST/NVD/Query.pm (unchanged)
Skip blib/lib/NIST/NVD/Update.pm (unchanged)
Skip blib/lib/NIST/NVD/Store/DB_File.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::Base.3pm
Manifying blib/man3/NIST::NVD.3pm
Manifying blib/man3/NIST::NVD::Query.3pm
Manifying blib/man3/NIST::NVD::Update.3pm
Manifying blib/man3/NIST::NVD::Store::DB_File.3pm
/usr/src/git/f5/NIST-NVD-Store-SQLite3
Writing Makefile for NIST::NVD::Store::SQLite3
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged)
cp bin/convert-nvdcve blib/script/convert-nvdcve
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve
Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm
t/01-load-nvdcve.t ..
1..20
ok 1 - $dist_dir is a directory
ok 2 - $test_dir is a directory
ok 3 - $data_dir is a directory
ok 4 - $convert_script is a file
ok 5 - $nvd_source_file is a file
ok 6 - $db_file does not yet exist
# running /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3 2>&1
ok 7 - took less than 15 seconds to load CWE data: 13
ok 8 - conversion script returned cleanly
ok 9 - database file exists
ok 10 - database file not empty
ok 11 - database file readable
ok 12 - database file writeable
ok 13 - database file not executable
ok 14 - file is correct type: [application/octet-stream]
ok 15 - $mtime is close
ok 16 - opened database file for reading
ok 17 - file contents indicate correct type: [application/octet-stream]
ok 18 - file contents indicate correct type: [application/octet-stream]
ok 19 - file contents indicate correct type: [SQLite 3.x database]
ok 20 - constructor returned an object of correct class
ok
All tests successful.
Files=1, Tests=20, 13 wallclock secs ( 0.02 usr  0.02 sys + 12.78 cusr  0.07 csys = 12.89 CPU)
Result: PASS
[edit 20120220T2220 -0800]
Okay, it s published. I ll need to factor the changes in to the DB_File storage engine as well.

Eugene V. Lyubimkin: bureaucratic programming

Suppose that you need to write an interface to a function which draws triangles. It could look like this: (the language is a C++-based pseudocode)

void rawDrawTriangle(size_t a, size_t b, size_t c)   ...  
bool isValidTriangle(size_t a, size_t b, size_t c)
 
  return (a + b > c) && (a + c > b) && (b + c > a);
 
void drawTriangle(size_t a, size_t b, size_t c)
 
  if (max(a, b, c) > MAX_LINE_LENGTH)
   
    throw("one of sides is too big");
   
  if (!isValidTriangle(a, b, c))
   
    throw("invalid triangle");
   
  rawDrawTriangle(a, b, c);
 
void userFunction()
 
  ...
  drawTriangle(3, 5, 7);
  ...


And the following is how a bureaucratic version of the code could look like:

class Certificate
 
  time_t getCreationTime() const   ...  
  Certificate()   ...  
 ;
class TriangleIsValidCertificate: public Certificate
 
 public:
  const size_t a;
  const size_t b;
  const size_t c;
 private:
  TriangleIsValidCertificate(size_t a, size_t b, size_t c)   ...  
 friend class TriangleCertificationAuthority;
 ;
class TriangleCertificationAuthority
 
  static TriangleIsValidCertificate getTriangleIsValidCertificate(size_t a, size_t b, size_t c)
   
    msleep(random() * 2);
    if ((a + b > c) && (a + c > b) && (b + c > a))
     
      if (a == b && a == c)
       
        msleep(random() * 10); // hm, suspicious query
       
      return TriangleIsValidCertificate(a, b, c);
     
    else
     
      throw("invalid triangle");
     
   
 ;
class LineIsDrawableCertificate: public Certificate
 
 public:
  const size_t length;
 private:
  LineIsDrawableCertificate(size_t l)   ...  
 friend class LineCertificationAuthority;
 ;
class LineCertificationAuthority
 
  static LineIsDrawableCertificate getLineIsDrawableCertificate(size_t length)
   
    msleep(rand());
    if (length <= MAX_LINE_LENGTH)
     
      return LineIsDrawableCertificate(length);
     
    else
     
      throw("the line is too long");
     
   
 ;
void drawTriangle(size_t a, size_t b, size_t c, LineIsDrawableCertificate lineCerts[3], TriangleIsValidCertificate tivCert)
 
  msleep(rand() * 5);
  if (lineCerts[0].length != a   lineCerts[1].length != b   lineCerts[2].length != c)
   
    throw("your application is rejected");
   
  if (tivCert.a != a   tivCert.b != b   tivCert.c != c)
   
    throw("your application is rejected");
   
  if (time() - tivCert.getCreationTime() > '60 milliseconds')
   
    throw("your application is rejected");
   
  msleep(rand());
  rawDrawTriangle(a, b, c);
 
void userFunction()
 
  ...
  size_t a = 3, b = 5, c = 7;
  auto aCert = LineCertificationAuthority::getLineIsDrawableCertificate(a);
  auto bCert = LineCertificationAuthority::getLineIsDrawableCertificate(b);
  auto cCert = LineCertificationAuthority::getLineIsDrawableCertificate(c);
  auto tviCert = TriangleCertificationAuthority::getTriangleIsValidCertificate(a, b, c);
  drawTriangle(a, b, c, array(aCert, bCert, cCert), tviCert);
  ...

Tanguy Ortolo: OpenSSH tip: connection sharing

The last versions of OpenSSH brought an interesting feature: sharing multiple sessions over a single connection. When enabled, this is how it works:
  1. The first time you open an SSH connection to a server (including anything that works on top of SSH, like SCP, SFTP, rsync or Git), it opens a network connection as usual. It also opens a local Unix socket and listens to it for later use.
  2. The next times you open an SSH connection to that server while the first one is still open, instead of opening a new network connection, it connects to that local Unix socket and lets the first SSH client carry its new session.
Advantages Disadvantages Just like SSH agents, there is one major side-effect to take into account: new remote sessions can be opened without a password. Never use such a feature on a public computer if you are used to temporarily leave running sessions: here, locking them would not protect other people to get a shell access to your remote account! Configuration In ~/.ssh/config or /etc/ssh/ssh_config: 
ControlPath ~/.ssh/control-%r@%h:%p
ControlMaster auto
ControlPersist 1
ControlPath
The name template for the session sharing Unix sockets. See the ssh_config(5) manpage for the syntax.
ControlMaster
Lets the SSH client use the session sharing socket if it exists or create it and listen otherwise.
ControlPersist
Usually, the first client to a given server is in charge of the connection sharing. It will then stay in foreground even if you terminate the session it is used for, until every other clients using its socket has finished, and forcefully terminating it will have the effect of terminating these clients! With that option, the master connection is put in background and will only terminate when no client needs it any more (one second after that, in fact), and you will be able to terminate the first session without kicking the other ones.
Try it! Apply this configuration and connect several times to a single host, to see the difference

Bartosz Fe&#324;ski: Automating building kernel for Zenbooks

I wrote some simple script to automate building kernels for Zenbook laptops. It downloads and apply Bluetooth patch, RC6 patch, Sentelic drivers, fixes DSDT table, and helps doing compilation of all of that. Hope it s going to be helpful for someone. Here it goes. Should work on most Debian based distros, but as usual. There is NO warranty ;)

Eddy Petri&#537;or: HOWTO: Git - reauthor/fix author and committer email and author name after a git cvsimport

You might find yourself at some moment when your git repository imported from CVS does not contain all the correct names and email addresses of the commits which were once in CVS but are now part of your project history in your git repo. Or you might do a cvsimport which missed a few authors.

Let's suppose you first import the cvs repo into git, but then you realise you missed some authors.

Before being able to do a git cvsimport, you need a checkout of the module or cvs subdir that you want to turn into its own git repo.

For ease of use I defined CVSCMD as
cvs -z9 -d :pserver:my_cvs_id@cvs.server.com:/root_dir
You will need to replace the items written in italics according to you situation, more exactly, you need to define 'my_cvs_id', 'cvs.server.com' and 'root_dir'. If your acces method to the server is not pserver, you should change that accordingly. This information should be available from your project admin or pages.


Check out the desired module or even subdir of a module

$CVSCMD checkout -d localdirname MODULE/path/to/subdir

git cvsimport -A ../authors -m -z 600 -C ../new-git-repo -R

How to find out the commits which do need rewriting

The way to limit yourself only to the commits that had no cvs-git author and commit information on git-cvsimport time is to use a filter like this:
git log -E --author='^[^@]*$' --pretty=format:%h
This tells git log to print only the abbreviated hashes (%h) for the commits that have NO '@' sign in the 'Author:', which happens if no cvs user id to git author and email was provided in the authors file and git cvsimport time.

We will use this command's output to tell later git filter-branch which commits need rewriting. *

But before that...

How do we find if our authors file is complete?

For this task we'll use a slighly modified form of the previous command and some shell script magic.
git log -E --author='^[^@]*$' --pretty=format:%an sort -u > all-leftout-cvs-authors
And now in all-leftout-cvs-authors we'll have a sorted list of all cvs id's which were not handled in the original git-cvsimport. In my case there are only 19 such ids:
$ wc -l all-leftout-cvs-authors
19 all-leftout-cvs-authors

Nice, that will be easy to fix. Now edit your all-leftout-cvs-authors file to add the relevant information in a format similar to this:
john = John van Code <john@code.temple.tld>
jimmy = Jimmy O'Document <jimmy@documenting.com>
In case you can't make a complete cvs-user-to-name-and-email map, you might want to use stubs of the following form in order to be able to easily identify later such commits, if you prefer (or you could let them unaltered at al ;-):
cvsid=cvsid <cvsid@cvs.server.com>

How to actually do the filtering to fix history (using git-filter-branch and a script)

After this is done, we'll need just one more piece, the command to do the altering itself which reads as follow (note that my final authors file is called new-authors and that I placed this in a script in order to be able to easily run it without trying to escape all spaces and such madness):

[ "$authors_file" ]   export authors_file=$HOME/new-authors

#git filter-branch -f --remap-cvs --env-filter '
git filter-branch -f --env-filter '

get_name ()  
    grep "^$1=" "$authors_file"   sed "s/^.*=\(.*\)\ .*$/\1/"
 

get_email ()  
    grep "^$1=" "$authors_file"   sed "s/^.*\ <\(.*\)>$/\1/"
 

if grep -q "^$GIT_COMMITTER_NAME" "$authors_file" ; then
    GIT_AUTHOR_NAME=$(get_name "$GIT_COMMITTER_NAME") &&
    GIT_AUTHOR_EMAIL=$(get_email "$GIT_COMMITTER_NAME") &&
    GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME" &&
    GIT_COMMITTER_EMAIL="$GIT_AUTHOR_EMAIL" &&
    export GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL &&
    export GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL
fi
' -- --all
You might wonder what's up with the commented git filter-branch line with the --remap-cvs option. This script will NOT work for you as long as you have the stock git-filter-branch script and keep the option --remap-cvs while not patching your git-filter-branch script (/usr/lib/git-core/git-filter-branch), but that option will provide a file with the mappings from the old to the new commit ids. If you want that function, too, you'll want to apply this patch to git-filter-branch:

diff --git a/git-filter-branch b/git-filter-branch
old mode 100644
new mode 100755
index ae602e3..d1f7ef6
--- a/git-filter-branch
+++ b/git-filter-branch
@@ -149,6 +149,11 @@ do
   prune_empty=t
   continue
   ;;
+ --remap-cvs)
+  shift
+  remap_cvs=t
+  continue
+  ;;
  -*)
   ;;
  *)
@@ -368,6 +373,33 @@ while read commit parents; do
    die "could not write rewritten commit"
 done <../revs
 
+# Rewrite the cvs-revisions file, if requested and the file exists
+
+ORIG_CVS_REVS_FILE="$ GIT_DIR /cvs-revisions"
+if [ -f "$ORIG_CVS_REVS_FILE" ]; then
+ if [ "$remap_cvs" ]; then
+  printf "CVS remapping requested\n"
+
+  CVS_REVS_FILE="$tempdir/cvs-revisions"
+  cp "$ORIG_CVS_REVS_FILE" "$CVS_REVS_FILE"
+  printf "\nFound $ORIG_CVS_REVS_FILE; will copy and alter it as $CVS_REVS_FILE\n"
+  cvs_remap__commit_count=0
+  newcommits="$(ls ../map/   wc -l)"
+  for commit in ../map/* ; do
+   cvs_remap__commit_count=$(($cvs_remap__commit_count+1))
+   printf "\rRemap CVS commit $commit ($cvs_remap__commit_count/$newcommits)"
+
+   oldsha1="$(basename $commit)"
+   read newsha1 < $commit
+   sed -i "s@$oldsha1\$@$newsha1@" "$CVS_REVS_FILE"
+  done
+ else
+  warn "\nNo CVS remapping requested, but cvs-revisions file found. All CVS mappings will be lost.\n"
+ fi
+elif [ "$remap_cvs" ]; then
+ warn "\nWARNING: CVS remap was ignored, since no original cvs-revisions file was found\n"
+fi
+
 # If we are filtering for paths, as in the case of a subdirectory
 # filter, it is possible that a specified head is not in the set of
 # rewritten commits, because it was pruned by the revision walker.
@@ -491,6 +523,11 @@ if [ "$filter_tag_name" ]; then
  done
 fi
 
+if [ "$remap_cvs" -a -f "$CVS_REVS_FILE" ]; then
+ mv "$ORIG_CVS_REVS_FILE" "$ORIG_CVS_REVS_FILE.original"
+ cp "$CVS_REVS_FILE" "$ORIG_CVS_REVS_FILE"
+fi
+
 cd ../..
 rm -rf "$tempdir"


Then, after running this script, let's call it filter, you should have a brand new git repo with the appropriate authors and their emails set.


P.S.: I have started writing this post some time ago but stopped just before the last part, the one with the filter script. I realise I might be missing something in the explanation, but if you have problems, please comment so I can help you fixing them.

P.P.S.: * I realised in the filter script at some point I wanted to do something like:
for R in $(git log -E --author='^[^@]*$' --pretty=format:%H   head -n 2) ; do
[the same git filter branch command above but ending in ...]
' $R
done
But I think I remember that $R didn't work on the whole history, but only on that revision, or some other weird of that sort. I know I ended up not filtering explicitly those revisions, but the entire history. I hope this helps.

Jan Hauke Rahm: Debugging getopt weirdness

I ve been working on enum again recently and got completely stuck in getopt_long() return values. I don t want to bore you with details, so let me cut to the chase: I needed a way to look into what getopt_long() is actually doing. Now, there are a few ways to figure out the doings of a function. If it s your own code, you could just go ahead and add a few printf()s to see if your variables fill up as intended. Then you could use gdb to walk through your running code. But what if you want to do so in functions outside the scope of your own code? In my case: getopt, i.e. libc? Obviously, you can download the library, add your printf() stuff, compile and install it, and see it all happening. But libc isn t exactly small, fast-compiling, and you really don t want to mess it up and have it installed on your system. So how about just taking what you actually care for and push it in? This is nothing experimental or particularly hacky . I just don t want to forget about it and maybe someone can make use of it. So here we go Download your libc sources to a new directory and copy the files you need. Try something like 
mkdir -p /tmp/getopt/mygetopt
cd /tmp/getopt
apt-get source eglibc
cp eglibc-*/posix/getopt* mygetopt/
cd mygetopt
Now write up a primitive Makefile: 
all: libmygetopt.so
libmygetopt.so: getopt.c getopt1.c getopt_int.h getopt.h
	gcc -shared -Wl,-soname,libmygetopt.so.1 -fPIC -o libmygetopt.so.1.0.0 getopt1.c getopt.c
	ln -s libmygetopt.so.1.0.0 libmygetopt.so.1
	ln -s libmygetopt.so.1 libmygetopt.so
clean:
	$(RM) libmygetopt*
In order for this to actually work, I fiddled around with a few #ifdef s in getopt.c. But that s something you ll probably find out yourselves. You should at least be able to compile it ( make it). Now put a printf statement somewhere in the function you want to inspect. Recompile the lib and then go back to your actual program. Running the program ist easy. 
src/enum -p1 1 3
1.0
2.0
3.0
Surprisingly, running it using your own getopt version is just as easy. Just push your library in: 
LD_PRELOAD=libmygetopt.so LD_LIBRARY_PATH=/tmp/getopt/mygetopt:$LD_LIBRARY_PATH src/enum -p1 1 3
Yo, Ulrich, my getopt version is waaaay cooler than yours!
Yo, Ulrich, my getopt version is waaaay cooler than yours!
1.0
2.0
3.0
If you re disappointed now, that s your own fault. I said it in the beginning, this is nothing special. It s not my idea. It s something that developers all use all the time or something like that. In case you didn t know about it, good for you. Otherwise, sorry for reading my memory dump.

Christian Perrier: Bug #660000

Beatrice Torracca reported the Debian bug #660000 on Wednesday February 15th, against wide-dhcpv6. That one makes me happy, because it is occurring during one of my numerous l10n "NMU" proposals and it is likely that I'll be the one who closes it (unless J r mie Corbier reacts to my l10n NMU proposal). It also makes me happy because Beatrice is participating to the great localization work of the Italian team where she is one of the most active contributors along with Francesca Ciceri. Hats off to italian ladies! Bug #650000 was reported as of November 25th. 2 months and 20 days for 10,000 bugs. The rate didn't really change and I expect it to increase a little bit as quality work related to release preparation will be increasing in the upcoming months. As usual, see you about 2.5 months for bug #670000 but, in the meantime, I might be "celebrating" Debian bug #666666. We all like nice numbers, isn't it?

Next.

Previous.