Search Results: "ag"

30 May 2011

Dirk Eddelbuettel: digest 0.5.0

A new version of the digest package (which generates hash function summaries for arbitrary (and possibly nested) R objects using any of the standard md5, sha-1, sha-256 or crc32 algorithms) is now on CRAN. Thanks to a patch by Mario Frasca, digest now contains a second (exported) function hmac() to generate Hash-based Message Authentication Code as defined in RFC 2104. Reference output from RFC 2104 is used to validate this functionality as shown below where we run the example code from the hmac() help page:
R> library(digest)
R> example(hmac)
hmacR> ## Standard RFC 2104 test vectors
hmacR> current <- hmac('Jefe', 'what do ya want for nothing?', "md5")
hmacR> target <- '750c783e6ab0b503eaa86e310a5db738'
hmacR> stopifnot(identical(target, as.character(current)))
hmacR> current <- hmac(rep(0x0b, 16), 'Hi There', "md5")
hmacR> target <- '9294727a3638bb1c13f48ef8158bfc9d'
hmacR> stopifnot(identical(target, as.character(current)))
hmacR> current <- hmac(rep(0xaa, 16), rep(0xdd, 50), "md5")
hmacR> target <- '56be34521d144c88dbb8c733f0e8b3f6'
hmacR> stopifnot(identical(target, as.character(current)))
hmacR> ## SHA1 tests inspired to the RFC 2104 and checked against the python
hmacR> ## hmac implementation.
hmacR> current <- hmac('Jefe', 'what do ya want for nothing?', "sha1")
hmacR> target <- 'effcdf6ae5eb2fa2d27416d5f184df9c259a7c79'
hmacR> stopifnot(identical(target, as.character(current)))
hmacR> current <- hmac(rep(0x0b, 16), 'Hi There', "sha1")
hmacR> target <- '675b0b3a1b4ddf4e124872da6c2f632bfed957e9'
hmacR> stopifnot(identical(target, as.character(current)))
hmacR> current <- hmac(rep(0xaa, 16), rep(0xdd, 50), "sha1")
hmacR> target <- 'd730594d167e35d5956fd8003d0db3d3f46dc7bb'
hmacR> stopifnot(identical(target, as.character(current)))
R> 
Also, CRANberries shows the difference to the previsious release 0.4.2 as follows:
Diff between digest versions 0.4.2 dated 2009-12-06 and 0.5.0 dated 2011-05-30
 digest-0.4.2/digest/INDEX                       only
 digest-0.4.2/digest/R/zzz.R                     only
 digest-0.4.2/digest/inst                        only
 digest-0.5.0/digest/ChangeLog                   only
 digest-0.5.0/digest/DESCRIPTION                    33 ++++++++++++-------------
 digest-0.5.0/digest/NAMESPACE                       7 +++--
 digest-0.5.0/digest/R/hmac.R                    only
 digest-0.5.0/digest/man/hmac.Rd                 only
 digest-0.5.0/digest/tests/digestTest.Rout.save      2 -
 digest-0.5.0/digest/tests/hmacTest.R            only
 digest-0.5.0/digest/tests/hmacTest.Rout.save    only
 11 files changed, 22 insertions(+), 20 deletions(-)
With that, special thanks to Mario Frasca for the patch and to Henrik Bengtsson for helpful discussion. We took some care to ensure that the existing interface to the digest() function remains unaffected.

Sam Dunne: Clarification

So earlier on I posted about my work on declarative diversions . I felt I needed to clarify a few things as a few people asked me different questions and I don t think I made the post clear enough and I also left out an important piece of information. The new control file will be located in control.tar.gz and it s syntax will be
One diversion per line Blank lines and lines with # are comments Two fields per diversion seperated by whitespace (SOURCE DESTINATION)
Now people have asked be about updating and removing diversions and how that will be accomplished. All diversions will be listed in this control file. If a diversion isn t listed in the control file on upgrade dpkg will automatically remove the diversion because it will see it is no longer needed. The same goes for changing diversions, i.i. list them just as though they re new and it will update them. This project will infer add, remove and package and will not allow you to specify them. The same is goes for rename, it is being made default and not optional (for now, I am aware of some special cases). *Removed, slight mistake here* Now, there are a number of things still left to do this week before a full draft is prepared. There is an email here by Ian Jackson which talks about ordering unpack in such a way to delay the the actual renaming of the file. If anyone has any suggestions for this process it would be greatly appreciated. Another two things that need to be done is the proper handling of diversions to non-existant directories and also escaping spaces within filenames for the control file. I should have another update tomorrow evening. Until then, ~Sam

Rapha&#235;l Hertzog: Trying to make dpkg triggers more useful and less painful

Lately I have been working on the triggers feature of dpkg. I would like to share my plan and what I have done so far. I ll first explain what triggers are, the current problems, and the work I did to try to improve the situation. Introduction Dpkg triggers are a neat feature of dpkg that package can use to send/receive notifications to/from other installed packages. Those notifications take the form a simple string. This feature is heavily used to track changes of packaged files in a list of predefined directories, and to update other files based on this. For instance, man-db is watching the directories containing manual pages so that it can update its cache (in /var/cache/man/). install-info is updating the index of info pages when there have been changes in /usr/share/info. gnome-menus is updating its own copy of the menu hierarchy (with entries from /etc/gnome/menus.blacklist blacklisted) every time that a .desktop file is installed/updated/removed. From a user s perspective You see triggers in action very often during upgrades (in fact too often as we ll see it later):
Preparing to replace zim 0.52-1 (using .../archives/zim_0.52-1_all.deb) ...
Unpacking replacement zim ...
Processing triggers for shared-mime-info ...
Processing triggers for menu ...
Processing triggers for desktop-file-utils ...
Processing triggers for man-db ...
Processing triggers for hicolor-icon-theme ...
Processing triggers for python-support ...
Processing triggers for gnome-menus ...
Setting up zim (0.52-1) ...
Processing triggers for python-support ...
Processing triggers for menu ...
As you guessed it, those Processing triggers lines correspond to the packages which received (one or more) trigger notifications and which are doing the corresponding task. By default the triggers are processed at the end of the dpkg --unpack invocation which is often too soon because APT will often call dpkg --unpack repeatedly during important upgrades. There are some options to ask APT to use dpkg s --no-triggers option in order to defer the trigger processing at the end of the APT run. You can put this in /etc/apt/apt.conf.d/triggers:
// Trigger deferred
DPkg::NoTriggers "true";
PackageManager::Configure "smart";
DPkg::ConfigurePending "true";
DPkg::TriggersPending "true";
I have now asked APT maintainers to use those options by default, I filed bug #626599 to track this. At the same time I fixed bug #526774 reported by APT maintainers. This bug forced them to put a work-around in APT which resulted in running triggers sooner than expected. (And while writing this article I filed bug #628564 and #628574 because it was clearly not normal that the menu triggers was executed twice for the installation of a single package) From a packager s perspective The implementation of triggers has several consequences on the status that packages can have. Let s assume that the package A installs a file in a directory that is watched by package B (and that B is currently in the installed state). When A is unpacked, dpkg adds B to its Triggers-Awaited field and lists the activated trigger in B s Triggers-Pending field. Package A is in unpacked state, but B has been changed to triggers-pending . When A is configured, instead of going to the installed state, it will go to the triggers-awaited state. In that state the package is assumed to NOT fulfill dependencies. However, B which is still in triggers-pending state does fulfill dependencies. A and B will switch to installed at the same time when the trigger has been processed. The fact that the triggers-awaited status does not fulfill dependencies means that some common triggers like man-db have to be processed regularly just to be able to ensure dependencies are satisfied before running the postinst of other installed packages. But a package which ships a manual page can certainly be considered as configured when its postinst has been run even if man-db has not yet updated its cache to know about the new/updated manual page. When you activate a trigger with the dpkg-trigger command you have an option --no-await to avoid awaiting the trigger processing (and thus to go directly to installed state after the postinst has been run). But with file triggers or activate trigger directives, you do not have this option. My proposal to improve the situation This is the problem that I tried to solve during my last vacation. But before changing the inner working of triggers, I wrote a non-regression test suite for that feature (commit here) so I could hack with some confidence that I did not break everything. The result has been presented on the debian-dpkg mailling list: see the discussion here. I added new directives that can be used in triggers files that work exactly like the current triggers except that they do not put triggering packages in trigger-awaited status. I believe the code to be mostly ready, but in its current form the patch brings zero benefits until all packages have been converted to use the trigger variants that do not require awaiting trigger processing (and the change requires a pre-dependency on dpkg to ensure we have the required dpkg that understands the new kind of trigger directives). Remaining question Thus I wonder if I should not change the default semantic of triggers. The packages which really provide crucial functionality to awaiting packages through triggers would then have to be updated to switch to the new directives. If you re a packager using triggers, you can thus help me by answering this question: do you know some triggers where it s important that the awaiting packages are not considered as configured before the trigger processing? In most of the cases I checked, it s important for the triggered package rather than for the triggering package. In truth, a package in triggers-awaited status is usually in a good enough shape to be able to satisfy dependencies (i.e. requirements that other packages can have), but it would still be worth to record the fact that it s not entirely configured yet because it might be true from the user s point of view: for example if the menu trigger has not yet been processed, the software might not yet be visible in the application menu.
If you appreciate this kind of groundwork that benefits to the whole Debian ecosystem, please consider supporting my work. Click here and give it a look, there are many ways to contribute and to make a difference for me.

3 comments Liked this article? Click here. My blog is Flattr-enabled.

Michael Prokop: Grml 2011.05 Codename Just Mari

I m proud to be able to announce a new stable release of Grml, the Debian based Live system for system administrators. This release is a very special one for me. On the one hand of course because of the special release name Just Mari , being dedicated to my lovely wife. But it s also special because of the way the release management worked out. I ve been the release manager for Grml since the very beginning, which turned out to be more than 6 years since the first stable release already. I developed grml-live as build framework based on FAI for generating a Grml and Debian based Linux Live system to streamline the build process. Anyway it was mainly me who managed the release chroots, doing the update management during release freeze, editing main web page etc. As I want to make the project as independent from myself as much as possible and to keep the Bus factor in balance we started to improve our project infrastructure so it s not just me who can do this kind of release management. As a result Grml core developer Christian Hofstaedtler became the release sergeant of this release. The release candidate version was even released without myself being available (the release happened behind my back during our marriage). Major work on the final stable release was also done by Grml core developers Ulrich Dangel, Christian Hofstaedtler, Frank Terbeck and Alexander Wirt and I m very happy about that. Kudos guys for all your work and all the hidden efforts going on behind my back. :) The details regarding the new Grml release are available in the official release announcement, get the ISOs from grml.org/download. I hope you enjoy the release as much as I do. Happy hacking!

Christoph Egger: Maintaining kFreeBSD buildds since one month

At April 30, I took over maintenance of of Debian's kFreeBSD autobuilders. Means getting something like 4,5k e-Mails this month (gladly no need to sign all those 4k successful builds any more!), filling nearly 30 RC Bugs (quite a lot of which got fixed just within hours after filling, wow!), investigating some of the more strange build failures and such stuff. In general it turned out to be quite some fun. Quite interesting which libraries turn out to be rather central to the Archive. I wouldn't have guessed that a uninstallable libmtp would prevent a reasonable fraction of the builds to fail -- including packages like subversion. Packages builds failing because the disk space is exhausted may be something most of us have already witnessed, especially those here that use one of these small notebook hard drives. Build failures caused by a lack of RAM might certainly be imaginable as well, especially on highly parallel builds. But have you ever seen gcc fail because the virtual address space was exhausted on 32 bit architectures? Also there's a interesting lot of packages with misspelled build dependencies which sbuild can't find and therefore can't build the package. Maybe having a lintian check for some of these problems would be a good idea? I'm also regularly seeing build failures that look easy enough to fix -- like some glob in a *.install for some python package matching lib.linux*. I try to fix some of these as I see them but my time is unfortunately limited as well. Someone interested in quick&easy noticed about these kind of issues? I could put URLs to build-logs on identi.ca or somewhere on IRC. There are also some really strange failures like llvm, which builds flawlessly on my local kFreeBSD boxes all the time, inside and outside schroot but hangs all the time in the same testcase when building on any of the autobuilders (any hints welcome!) or perl failing on kfreebsd-amd64 selectively but all the time.

Steve Kemp: How do you deploy applications?

I've got a few projects which are hosted in mercurial repositories. To deploy them I manually checkout the repository, create symlinks by hand, then update apache thttpd to make them work. When I want to update my applications I manually become the correct user, find the repository and run "hg pull --update". I think it is about time that I sat down and started doing things neatly. I made a start at this by writing a shell script for each site called .deploy then I drive it like so:
#!/bin/sh
#
# ~/bin/deploy  execute the .deploy file associated with this project.
#
while true; do
    #
    #  If we're at the root directory we're done.
    #
    if [ $PWD = "/" ]; then
        echo "Reached /"
        exit
    fi
    # found our file?
    #
    if [ -x ".deploy" ]; then
       ./.deploy
       exit
    fi
    cd ..
done
It seems the main candidate is capistrano, which was previously very Ruby on Rails centric, but these days seems to be divorced from it. Alternatively there is the python-based fabric project which has been stalled for two years, vlad the deployer (great name!) which is another Rake-based and thus Ruby-loving system, and finally whiskey disk which is limited to Git-based projects as far as I can tell. In short each of these projects is very similar, and each relies upon being able to do two things: I've automated SSH before, and I've automed SCP/rsync. The hard part is doing both "copy" and "command" over one SSH channel - such that you don't get prompted for passwords multiple times - and handling the case of runnign sudo where appropriate. Still most of the initial stages are trivial. I wonder what project I should be using: Anything I've missed? Or pointers to good documentation? ObQuote: "We need to be a little more constructive here, okay? " - Terminator 2

Bastian Venthur: I m going to DebConf11

Weee! After the excellent experience last year in New York, I couldn t resist :) See you all in Banja Luka!

Francois Marier: Code reviews with Gerrit and Gitorious

The Mahara project has just moved to mandatory code reviews for every commit that gets applied to core code.

Here is a description of how Gerrit Code Review, the peer-review system used by Android, was retrofitted into our existing git repository on Gitorious.

(If you want to know more about Gerrit, listen to this FLOSS Weekly interview.)

Replacing existing Gitorious committers with a robotThe first thing to do was to log into Gitorious and remove commit rights from everyone in the main repository. Then I created a new maharabot account with a password-less SSH key (stored under /home/gerrit/.ssh/) and made that new account the sole committer.

This is to ensure that nobody pushes to the repository by mistake since all of these changes would be overwritten by Gerrit.

Basic Gerrit installationAfter going through the installation instructions, I logged into the Gerrit admin interface and created a new "mahara" project.

I picked the "merge if necessary" submit action because "cherry-pick" would disable dependency tracking which is quite a handy feature.

Reverse proxy using NginxSince we wanted to offer Gerrit over HTTPS, I decided to run it behind an Nginx proxy. This is the Nginx configuration I ended up with:
server  
listen 443;
server_name reviews.mahara.org;
add_header Strict-Transport-Security max-age=15768000;

ssl on;
ssl_certificate /etc/ssl/certs/reviews.mahara.org.crt;
ssl_certificate_key /etc/ssl/certs/reviews.mahara.org.pem;

ssl_session_timeout 5m;
ssl_session_cache shared:SSL:1m;

ssl_protocols TLSv1;
ssl_ciphers HIGH:!ADH;
ssl_prefer_server_ciphers on;

location /
proxy_pass http://127.0.0.1:8081;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;


Things to note:
Mail setupTo enable Gerrit to email reviewers and committers, I installed Postfix and used "reviews.mahara.org" as the "System mail name".

Then I added the following to /home/gerrit/mahara_reviews/etc/gerrit.config:
[user]
email = "gerrit@reviews.mahara.org"
to fix the From address in outgoing emails.

Init script and cronFollowing the installation instructions, I created these symlinks:
ln -s /home/gerrit/mahara_reviews/bin/gerrit.sh /etc/init.d/gerrit
cd /etc/rc2.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc3.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc4.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc5.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc0.d && ln -s ../init.d/gerrit K21gerrit
cd /etc/rc1.d && ln -s ../init.d/gerrit K21gerrit
cd /etc/rc6.d && ln -s ../init.d/gerrit K21gerrit
and put the following settings into /etc/default/gerritcodereview:
GERRIT_SITE=/home/gerrit/mahara_reviews
GERRIT_USER=gerrit
GERRIT_WAR=/home/gerrit/gerrit.war
to automatically start and stop Gerrit.

I also added a cron job in /etc/cron.d/gitcleanup to ensure that the built-in git repository doesn't get bloated:
MAILTO=admin@example.com
20 4 * * * gerrit GIT_DIR=/home/gerrit/mahara_reviews/git/mahara.git git gc --quiet

Configuration enhancementsTo allow images in change requests to be displayed inside the browser, I marked them as safe in /home/gerrit/mahara_reviews/etc/gerrit.config:
[mimetype "image/*"]
safe = true

Another thing I did to enhance the review experience was to enable the gitweb repository browser:
apt-get install gitweb

and to make checkouts faster by enabling anonymous Git access:
[gerrit]
canonicalGitUrl = git://reviews.mahara.org/git/
[download]
scheme = ssh
scheme = anon_http
scheme = anon_git

which requires that you have a git daemon running and listening on port 9418:
apt-get install git-daemon-run
ln -s /home/gerrit/mahara_reviews/git/mahara.git /var/cache/git/
touch /home/gerrit/mahara_reviews/git/mahara.git/git-daemon-export-ok

Finally, I included the Mahara branding in the header and footer of each page by providing valid XHTML fragments in /home/gerrit/mahara_reviews/etc/GerritSiteHeader.html and GerritSiteFooter.html.

Initial import and replicationOnce Gerrit was fully working, I performed the initial code import by using my administrator account to push the exiting Gitorious branches to the internal git repository:
git remote add gerrit ssh://username@reviews.mahara.org:29418/mahara
git push gerrit 1.2_STABLE
git push gerrit 1.3_STABLE
git push gerrit master
Note that I had to temporarily disable "Require Change IDs" in the project settings in order to import the old commits which didn't have these.

To replicate the internal Gerrit repository back to Gitorious, I created a new /home/gerrit/mahara_reviews/etc/replication.config file:
[remote "gitorious"]
url = gitorious.org:mahara/$ name .git
push = +refs/heads/*:refs/heads/*
push = +refs/tags/*:refs/tags/*
(The $ name variable is required even when you have a single project.)

Contributor instructionsThis is how developers can get a working checkout of our code now:
git clone git://gitorious.org/mahara/mahara.git
cd mahara
git remote add gerrit ssh://username@reviews.mahara.org:29418/mahara
git fetch gerrit
scp -p -P 29418 reviews.mahara.org:hooks/commit-msg .git/hooks/
and this is how they can submit local changes to Gerrit:
git push gerrit HEAD:refs/for/master

Anybody can submit change requests or comment on them but make sure you do not have the Cookie Pie Firefox extension installed or you will be unable to log into Gerrit.

29 May 2011

Dirk Eddelbuettel: Bike The Drive 2011

Memorial Day weekend so following the annual JPMorgan Chase CC run on Thursday, it was once again time for the annual Bike The Drive on Chicago's beautiful Lakeshore Drive. The weather was craptastic---following a fog (!!) warning we had dense fog along the lake making it somewhat moist and chilly. Once we were done this was followed by an impressive thunderstorm warnings and rather heavy rain. But the girls were troopers and we did both halfes for a total of thirty miles as shown on this Google Maps record of the ride.

Evgeni Golov: I m going to DebConf11 and Wacken2011

I'm going to DebConf11Yes, you will see me in Banja Luka. This will be my first DebConf, but I have a strong feeling it won t be the last one. I hope to see some productive sessions in pkg-games and pkg-ayatana, nice people and a lot of fun in a nice country.I will travel with ICE 729 from Duesseldorf to Munich and EN 499 from Munich to Zagreb on the 22.07.2011 and return with EN 498 and ICE 728 on 31.07.2011.To make things louder, I will go to Wacken from 02.08.2011 until 07.08.2011, wonder to see some Debian people there :)

Thomas Koch: Is it really that hard?

It's 2011. Computers are here for a couple of decades now, but they still don't help me with the most simple problems. I've been asked once again about a recommendation for a GroupWare and still don't know what to say. This are the simple requirements:

Manage my Mails, Calendar, Contacts, Files, Knowledge (Wiki) and ToDos (including IssueTracker). Let me synchronize these informations with mobile devices and for offline work. Allow collaboration and sharing.

I started professional programming in 2006 with the eGroupWare project, which provides the above functionality more or less, but I still can't recommend it. Neither could I recommend any other system I know of (Tine 2.0, OpenGroupWare, OpenExchange, Horde, Kolab, ...?) since they violate one or more of my secondary requirements:



This kind of software is on the top position on my Things-To-Do-When-I-Have-Time. But on the other hand I know that many have already started such projects and either they did not deliver sufficient functionality or there code base is a PITA.

Yes, I'm ranting once again. But please show me a decent GroupWare and I'll praise you on any occassion!

Or should I make this my project for my bachelor thesis: A Contacts+Mail server/web frontend in Scala using CouchDB, Dovecot? What do you think?

Thomas Koch: Keysigning @ Berlin Buzzwords

Whenever you download some software from the Apache Software Foundation, there is a small .asc file besides every software release. This tiny file is a cryptographic signature to protect you from accidentally downloading and running malware.
Put your key signing fingerprints in your badge!
Put your key signing fingerprints in your badge!

However this signature is only as good as the web-of-trust between you and the Release Manager of the Software project.
At BerlinBuzzWords many developers and users will gather at one place. This is a unique occasion to strengthen the web-of-trust. Therefor we want to encourage and remind you to use the occassion for keysigning! For lack of time there won't be an official keysigning party, but you have occassion during lunch, coffee break or the barbecue to do quick one-to-one keysignings.
You should bring print-outs of your key fingerprints and identification documents. If you put the fingerprints in your batch (see picture) everyone can easily see
them and ask you for a quick trust exchange. We may also mark a "keysigning corner" somewhere where people can go and meet others for key signing.

Hints:

28 May 2011

Nathan Handler: Google Summer of Code Update #1

I just got off the phone with my Google Summer of Code mentor, Matt Zimmerman. Some things that we discussed were that Debian Developers are quite partial to the BTS. We agreed that using the BTS to store as much information as possible would be ideal. This would allow the Debian Maintainers to see the information that pertains to them, without them having to change their workflows to check a DEX dashboard. Usertags are a rather useful feature that we did not utilize much when processing ancient patches. In the future, we will be applying usertags to Debian bugs that DEX is tracking so that we can easily query and track them. We also plan to create a more useful overview/dashboard page. This page will grab a list of bugs with a certain usertag from the BTS. It will then allow people to see bugs involved in a DEX project, who is working on them, what the current status in Debian is, and what the current status downstream is. Matt mentioned possibly using jquery for this. I have not used jquery before, so I will investigate it a bit more. I also noted the transition tracker as another type of dashboard that we could base the DEX overview page on. In addition, we hope to take the graph idea a step further by modifying it to take time into account. This means that we will be able to clearly see how long a certain project is taking, predict based on our current rate how long it will take to complete, and detect periods of time where we might be slowing down. We also hope to have graphs that acknowledge people at the individual level for working on DEX projects. During the upcoming week, I will begin working on this overview page. In addition to coding, this will involve doing some more research to learn more about some of the tools and resources available to us. I hope to have some actual code to point people to by the time I write my next blog update.

Dirk Eddelbuettel: RcppArmadillo 0.2.21

Following up Conrad's 1.99.2 release (in RcppArmadillo 0.2.20), we now have Armadillo 1.99.3 en route to the upcoming 2.0.0 release. Armadillo is a templated C++ library for linear algebra---which RcppArmadillo brings to R based on our Rcpp library. The short NEWS file extract follows below.
0.2.21  2011-05-27
    o   Upgraded to Armadillo release 1.99.3 "v2.0 beta 3" 
          * stricter size checking for row and column vectors
          * added .count() member function to running_stat and running_stat_vec
And courtesy of CRANberries, here is the diff to the previous release.
Diff between RcppArmadillo versions 0.2.20 dated 2011-05-26 and 0.2.21 dated 2011-05-28
 ChangeLog                                                   8 
 DESCRIPTION                                                 8 
 inst/NEWS                                                   7 
 inst/include/armadillo                                      2 
 inst/include/armadillo_bits/Col_meat.hpp                   13 -
 inst/include/armadillo_bits/Mat_bones.hpp                   2 
 inst/include/armadillo_bits/Mat_meat.hpp                  147 +++++++-------
 inst/include/armadillo_bits/Row_meat.hpp                   11 -
 inst/include/armadillo_bits/arma_version.hpp                4 
 inst/include/armadillo_bits/diagview_bones.hpp              2 
 inst/include/armadillo_bits/diagview_meat.hpp               4 
 inst/include/armadillo_bits/diskio_meat.hpp                40 ++--
 inst/include/armadillo_bits/fn_accu.hpp                   167 ++++++++++-------
 inst/include/armadillo_bits/fn_det.hpp                     10 -
 inst/include/armadillo_bits/fn_prod.hpp                    15 +
 inst/include/armadillo_bits/op_dot_bones.hpp                9 
 inst/include/armadillo_bits/op_dot_meat.hpp               119 +++++++++---
 inst/include/armadillo_bits/promote_type.hpp               24 +-
 inst/include/armadillo_bits/running_stat_bones.hpp         16 -
 inst/include/armadillo_bits/running_stat_meat.hpp          47 ++--
 inst/include/armadillo_bits/running_stat_vec_bones.hpp      6 
 inst/include/armadillo_bits/running_stat_vec_meat.hpp      21 +-
 inst/include/armadillo_bits/subview_meat.hpp               23 +-
 inst/include/armadillo_bits/traits.hpp                     16 -
 inst/include/armadillo_bits/typedef_u64.hpp                21 --
 25 files changed, 446 insertions(+), 296 deletions(-)
More information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

Vincent Bernat: SNMP support for keepalived

keepalived keepalived is an high-availability and load-balacing solution. Using VRRP, it allows you to share IP addresses between several servers or routers. At a given moment, only one of them (the master) will be handed the IP address. VRRP is usually setup for redundant routers, but you can also setup redundant services. If the master becomes unavailable or fails, one of the backups will take over the IP address. VRRP is not a cluster resource manager. It is targeted at sharing one (or several) IP between a set of hosts. If you want to share other resources with complex conditions like at least two web servers should run and the database should not run on the same node as a web server , you should look at heartbeat instead. For load-balancing, keepalived relies on IPVS, a Linux subsystem featuring layer-4 switching. keepalived will check if your servers are alive and tell it to IPVS. IPVS will handle incoming connections and send them on the appropriate alive server depending on the policy you can configure. For example, one policy is round-robin: servers are handled new connections in circular order.

keepalived & SNMP keepalived did not feature a native SNMP support. I have added complete SNMP support and I hope those patches will be accepted into mainstream. Meanwhile, you can grab keepalived with SNMP support source code from GitHub. Currently, it is based on the latest version of keepalived. Here is what you can do :
  • query the configuration of the running keepalived without parsing configuration files;
  • query runtime status (like VRRP status, priority or current state of a virtual server) without looking in the logs;
  • query runtime statistics about virtual servers (how many connections were handled by this real server, for example) without parsing the output of ipvsadm;
  • get noticed with the help of SNMP traps when something changed (VRRP transition or a real server that became unavailable);
  • changing the priority of a VRRP instance (to force a transition as master);
  • changing the weight of a real server (to remove it from a pool of servers).

Compilation To use it:
1
2
3
4
5
$ git clone https://github.com/vincentbernat/keepalived.git
$ cd keepalived
$ git checkout snmp-1.2.0
$ ./configure --enable-snmp
$ make
You get bin/keepalived which is the SNMP enabled keepalived daemon. You can use it directly or install it with make install as root. On RHEL, SNMP is not properly packaged. Try the following command instead of make:
1
$ make LDFLAGS="$(net-snmp-config --agent-libs) -lpopt -lssl -lcrypto"

Use To use it, you need to ensure that your main snmpd daemon can become a master agent. Check for the presence of master agentx line in snmpd.conf. The next step is to start keepalived with the -x switch. keepalived will connect to your main snmpd daemon. You can check the logs for a ligne like this:
May 28 17:21:19 L1 Keepalived_vrrp: NET-SNMP version 5.4.3 AgentX subagent connected

Lab To test it, we can setup a little lab with UML. You can find additional details about this kind of lab in my post about network lab with UML. Grab the complete lab from GitHub. Ensure that you correct the path to keepalived sources at the top of the setup script. keepalived & SNMP lab The lab features four web servers that will be merged into a single web service with the help of keepalived. Since, keepalived now supports full IPv6 support, we will also use IPv6 in our lab. Here is the VRRP configuration of the first keepalived :
vrrp_instance VI_1  
   state MASTER
   interface eth2
   dont_track_primary
   track_interface  
      eth0
      eth1
    
   virtual_router_id 1
   priority 150
   advert_int 2
   authentication  
      auth_type PASS
      auth_pass blibli
    
   virtual_ipaddress  
      1.1.1.15/24 dev eth0
      2001:db8::15/64 dev eth0
      192.168.1.1/24 dev eth1
      fd00::1/64 dev eth1
   
 
We define two virtual servers, one for IPv4, one for IPv6. Here is an excerpt of the configuration :
virtual_server_group VS_GROUP_IPv4  
   1.1.1.15 80
 
virtual_server group VS_GROUP_IPv4  
   delay_loop 10
   lb_algo rr
   lb_kind NAT
   protocol TCP
   real_server 192.168.1.10 80  
      weight 1
      HTTP_GET  
        url  
          path /
          status_code 200
         
        connect_timeout 10
       
    
   [...]
 
When running the lab (with ./setup), we can check that everything works as expected. From the client node, we can make several requests, both with IPv4 and IPv6:
1
2
3
4
5
6
7
8
C1# curl  http://1.1.1.15 
W3
C1# curl  http://1.1.1.15
W2
C1# wget -o /dev/null -O - 'http://[2001:db8::15]'
W3
C1# wget -o /dev/null -O - 'http://[2001:db8::15]'
W2
If we put down eth1 on L1, L2 becomes master and everything works as expected. Our client node can still do the requests.

SNMP We can query the configuration of our running keepalived from L1. The appropriate MIB is KEEPALIVED-MIB. The base OID for this MIB is .1.3.6.1.4.1.9586.100.5. It is hosted in the OID space allocated by IANA to the Debian project. For example, to get the configuration of our VRRP instance:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# snmpwalk -v2c -cpublic localhost KEEPALIVED-MIB::vrrpInstanceTable
KEEPALIVED-MIB::vrrpInstanceName.1 = STRING: VI_1
KEEPALIVED-MIB::vrrpInstanceVirtualRouterId.1 = Gauge32: 1
KEEPALIVED-MIB::vrrpInstanceState.1 = INTEGER: master(2)
KEEPALIVED-MIB::vrrpInstanceInitialState.1 = INTEGER: master(2)
KEEPALIVED-MIB::vrrpInstanceWantedState.1 = INTEGER: master(2)
KEEPALIVED-MIB::vrrpInstanceBasePriority.1 = INTEGER: 150
KEEPALIVED-MIB::vrrpInstanceEffectivePriority.1 = INTEGER: 150
KEEPALIVED-MIB::vrrpInstanceVipsStatus.1 = INTEGER: allSet(1)
KEEPALIVED-MIB::vrrpInstancePrimaryInterface.1 = STRING: eth2
KEEPALIVED-MIB::vrrpInstanceTrackPrimaryIf.1 = INTEGER: tracked(1)
KEEPALIVED-MIB::vrrpInstanceAdvertisementsInt.1 = Gauge32: 2 seconds
KEEPALIVED-MIB::vrrpInstancePreempt.1 = INTEGER: preempt(1)
KEEPALIVED-MIB::vrrpInstancePreemptDelay.1 = Gauge32: 0 seconds
KEEPALIVED-MIB::vrrpInstanceAuthType.1 = INTEGER: password(1)
KEEPALIVED-MIB::vrrpInstanceLvsSyncDaemon.1 = INTEGER: disabled(2)
KEEPALIVED-MIB::vrrpInstanceGarpDelay.1 = Gauge32: 0 seconds
KEEPALIVED-MIB::vrrpInstanceSmtpAlert.1 = INTEGER: disabled(2)
KEEPALIVED-MIB::vrrpInstanceNotifyExec.1 = INTEGER: disabled(2)
We can get statistics from IPVS subsystem:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# snmpwalk -v 2c -c public localhost KEEPALIVED-MIB::virtualServerTable   grep Stats
KEEPALIVED-MIB::virtualServerStatsConns.1 = Gauge32: 6 connections
KEEPALIVED-MIB::virtualServerStatsConns.2 = Gauge32: 6 connections
KEEPALIVED-MIB::virtualServerStatsInPkts.1 = Counter32: 36 packets
KEEPALIVED-MIB::virtualServerStatsInPkts.2 = Counter32: 36 packets
KEEPALIVED-MIB::virtualServerStatsOutPkts.1 = Counter32: 24 packets
KEEPALIVED-MIB::virtualServerStatsOutPkts.2 = Counter32: 24 packets
KEEPALIVED-MIB::virtualServerStatsInBytes.1 = Counter64: 2970 bytes
KEEPALIVED-MIB::virtualServerStatsInBytes.2 = Counter64: 3312 bytes
KEEPALIVED-MIB::virtualServerStatsOutBytes.1 = Counter64: 2592 bytes
KEEPALIVED-MIB::virtualServerStatsOutBytes.2 = Counter64: 3072 bytes
We can also disable a real server, for example W1:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# snmpget -v2c -cpublic localhost KEEPALIVED-MIB::realServerAddrType.1.1 \
                                KEEPALIVED-MIB::realServerAddress.1.1
KEEPALIVED-MIB::realServerAddrType.1.1 = INTEGER: ipv4(1)
KEEPALIVED-MIB::realServerAddress.1.1 = Hex-STRING: C0 A8 01 0A 
# snmpget -v2c -cpublic localhost KEEPALIVED-MIB::realServerAddress.2.1 \
                                KEEPALIVED-MIB::realServerAddress.2.1
KEEPALIVED-MIB::realServerAddress.2.1 = Hex-STRING: FD 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10 
KEEPALIVED-MIB::realServerAddress.2.1 = Hex-STRING: FD 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10 
# snmpset -v2c -cprivate localhost KEEPALIVED-MIB::realServerWeight.1.1 = 0
KEEPALIVED-MIB::realServerWeight.1.1 = INTEGER: 0
# snmpset -v2c -cprivate localhost KEEPALIVED-MIB::realServerWeight.2.1 = 0
KEEPALIVED-MIB::realServerWeight.2.1 = INTEGER: 0
From C1, you can now check that W1 is not queried anymore. Let s graph the number of connections per second each real server gets. With a simple script using rrdtool, we can get the following graph: rrdtool graph built from stats returned by keepalived SNMP agent

Evgeni Golov: Debian/Ubuntu Games Team Meeting #3

As it hasn t been announced on planet yet, I d like to invite you to the thrird Debian/Ubuntu Games Team Meeting on May 29th (yes, tomorrow!) at 11:00 UTC in #debian-games.
  • What: Debian/Ubuntu Games Team Meeting #3
  • When: 29.05.2011, 11:00:00 UTC
  • Where: #debian-games on irc.deb ian.org
  • Who: YOU!
The agenda is listed in the wiki and Kamping_Kaiser will try to stick to it (not as I did the last meeting).So see you tomorrow 11:00 UTC in #debian-games on irc.debian.org!

27 May 2011

Tanguy Ortolo: Wrapper to use su as sudo (or sudo as su)

The problem Sudo logo: a shield with a hash sign, a lightning and SUDO written on Some programs require root privileges for part of their work: most notably Debian building tools such as dpkg-buildpackage or pdebuild. So, several command can be used to obtain real or fake root privileges: su, sudo, fakeroot, etc. Among them, su is quite distinct, because it only takes a quoted command to pass it to a shell. That is why its interface is completely incompatible with the others':
$ su -c "pbuilder --build package.dsc"
$ sudo pbuild --build package.dsc
This is annoying when real root privileges are needed, for instance by pbuilder, a tool to build Debian packages inside a minimal chroot. Indeed, in such a case I only know one command to obtain the root privileges, sudo, and some people (read: me) prefer using su.A wrapper Long story short: as I saw no existing tool to do that (but I would be glad to hear if there was already one), I wrote a wrapper that emulates a (very basic) sudo-like interface by using su, and vice-versa. It is available from my Git repository:
$ git clone git://git.ortolo.eu/sudo-su.git
It is polymorphic, BusyBox-style: call it sudo-su and it will emulate sudo with su; call it su-sudo and it will emulate su with sudo. For instance:
$ pdebuild --pbuilderroot /usr/local/bin/sudo-su
It is WTFPL-licensed, it implements no options from sudo and su (except sudo's -c, of course), but I can do something better if it may be useful to other people.

Joachim Breitner: Conditional Elimination through Code Duplication

My Advanced Compilers course at IIT Bombay under Prof. D. M. Dhamdhere also involved a group project. Our group had the task to pick an existing compiler optimization which works by restructuring the control flow graph of the code, and implement that in LLVM. As we did not quickly find a suitable optimization, I came up with a simple idea by myself. At first, our professor was not really pleased by this move, but after the second presentation or so, when the ideas were more streamlined and we noticed how simple yet possibly widely applicable the transformation is, he allowed us to carry on with it. I concentrated on the theoretical part, writing a paper describing it, while the others in the team implemented parts of it as a pass for the LLVM compiler. The idea is simple: If within one execution of a program the same conditional expression (e.g. a variable debug_enabled) is evaluated several times, then it might be faster to copy the code in between the conditional jumps and thus remember the result without further conditional jumps. Of course this increases the code size, and there is a trade-off, but in some instances the benefits could well be the worth the effort, especially inside tight inner loops. You can read more about it in my paper about CECD, and I m considering uploading it to arxiv.org as well.

Erich Schubert: Dear Lazyweb, how to write multi-locale python code

Dear Lazyweb,
I've been toying around with a python WSGI application, i.e. a multi-threaded persistent web application. Now I'd like to add multi-language support to this application. I need to format datetimes to human readable formats, but I havn't found a way yet to do this in a sane way using strftime. Essentially, strftime will use the current application locale; however since I'm running multi-threaded, different threads might want to use different locales. So changing the locale is bound to cause race conditions.
So what is the best way to pretty-print (including week day names!) datetime, currency and similar values in a multi-threaded multi-locale context in python? Gettext and manually emulating strftime doesn't sound that sensible to me. And of course, I don't want to have to translate the weekday names myself into any language I choose to support...

Matt Zimmerman: Why I m excited about joining Singly

This summer, I ll be taking a bit of time off, moving back to San Francisco and starting a new job. I can t wait to get back to work. Here s why. Me and my data
I have a singular relationship with my data. I have a copy of every email I ve sent since I first got an Internet email address in 1994 (82,000 messages and counting). I have even older files downloaded from BBSes, and passed between friends on floppy disks. Chat logs, text messages, voicemail I hold onto them all. Anything which is relevant to me personally, I tend to save. This must seem banal to people who are first getting online today. In the age of Gmail and Flickr, it s easy to assume that all of your data will be preserved indefinitely, with little or no effort on your part. But for me, it has been hard work over the years, because I ve done it myself. I ve carried my data with me to countless new computers, operating systems, storage technologies, file formats and cities over the years. Everywhere I ve lived, I ve brought it with me. Physically. Really? Why do I do this? Why have I gone to such trouble for a collection of bits? Especially now, why is most of my data still at home?
One pragmatic answer is that I can simply do more with my data when I have a copy. I can work with it using any software I want, including software that I write myself. I don t have to worry about whether I can transfer it from one web service to another. I m never stuck using yesterday s services because my data is never trapped in them. My personal data is always available to me me, always raw, ready and waiting for the next wave of software to come along. When it does, I can load my data into it and keep going. The fact that Facebook and Google disagree over sharing their users data doesn t bother me in the least. Another reason is that I want to be in control of it. I decide who to share my data with, and when. Some of it, I prefer not to share at all, with any person or company, and I have that choice. Even if a powerful government wants to access my data, I am afforded certain protection under the law, at least in the countries where I ve lived. If I turned over my data to service providers, my choices and protection would likely be much more limited. I have a deeper emotional attachment to my data as well. Enfolded within that vast pattern of bits is some part of my self. By sharing my personal data with other people, I show them something of who I am. Increasingly, my personal data is part of my identity. This is more than just a state of mind: it s been shown that even our non-identifying personal data can reveal who we are. In other words, it s not just my data it s me data . Singly I m joining Singly because I want to take this concept much further, and combine people, data and software into a different shape with people at the center. Today, we are creating vastly greater amounts of personal data, and it s stored in many more places. We leave our trail on the Internet in the form of activity streams, messages and content, spread across different web sites, each with their own inscrutable terms of service and (if we re lucky) their own API. These disconnected silos prevent us from using all of this information effectively. Meanwhile, we want and need to connect with each other in more ways than ever before. We need applications which can connect us, through our personal data, to the services we need. Singly is building the technology to make this possible. It will be designed with the deepest respect for the relationship that we have with our personal data, and with a vision for truly personal computing. Singly is This opportunity is a great fit for my interests and experience. Singly aims to be the commercial part of a vibrant open source community, and I m looking forward to building on what I ve learned in Debian, Canonical and Ubuntu to help make it a success. I ll have lots more to say about it as time goes on. Meanwhile, if you re interested in following what we re doing, here s where:

Next.

Previous.