Search Results: "ag"

18 January 2011

John Goerzen: Backing Up to the Cloud

I m recently taking some big-picture looks at how we do things, and one thing that I think could be useful would be for us to back up a limited set of data to an offsite location states away. Prices are cheap enough for this to make it useful. Services such as Amazon S3 and Rackspace Cloud Files (I ve heard particularly good things about that one) seem to be perfect for this. I m not quite finding software that does what I want, though. Here are my general criteria:
  1. Storage fees of $0.15 per gigabyte-month or less
  2. Free or cheap ($0.20 per gigabyte or less) bandwidth fees
  3. rsync-like protocol to avoid having to re-send those 20GB files that have 20MB of changes in their entirety every night
  4. Open Source and cross-platform (Linux, Windows, Mac, Solaris ideally; Linux and Windows at a minimum)
  5. Compression and encryption
  6. Easy way to restore the entire backup set or individual files
  7. Versatile include/exclude rules
  8. Must be runnable from scripts, cron, etc. without a GUI
  9. Nice to have: block or file-level de-duplication
  10. Nice to have: support for accurately backing up POSIX (user, group, permission bits, symlink, hard links, sparse files) and Windows filesystem attributes
  11. Nice to have: a point-and-click interface for the non-Unix folks to use to restore Windows files and routine restore requests
So far, here s what I ve found. I should note that not a single one of these solutions appears to handle hard links or sparse files correctly, meaning I can t rely on them for complete system-level backups. That doesn t mean they re useless I could still use them to back up critical user data just less useful. Of the Free Software solutions, Duplicity is a leading contender. It has built-in support for Amazon S3 and Rackspace Cloud Files storage. It uses rdiff, which is a standalone implementation of the rsync binary delta algorithm. So you send up a full backup, then binary deltas from that for incrementals. That makes it bandwidth-efficient for incremental backups, and storage-efficient. However, periodic full backups will have to be run, which will make it less bandwidth-efficient. (Perhaps not incredibly *often*, but they will still be needed.) Duplicity doesn t offer block-level de-deuplication or a GUI for the point-and-click folks. But it DOES offer the most Unixy approach and feels like a decent match for the task overall. The other service relying on Free Software is rsync.net, which supports rsync, sftp, scp, etc. protocols directly. That would be great, as it could preserve hard links and be compatible with any number of rsync-based backup systems. The downside is that it s expensive really expensive. Their cheapest rate is $0.32 per GB-month and that s only realized if you store more than 2TB with them. The base rate is $0.80 per GB-month. They promise premium support and such, but I just don t think I can justify that for what is, essentially, secondary backup. On the non-Open Source side, there s JungleDisk, which has a Server Edition that looks like a good fit. The files are stored on either S3 or Rackspace, and it seems to be a very slick and full-featured solution. The client, however, is proprietary though it does seem to offer a non-GUI command-line interface. They claim to offer block-level de-duplication which could be very nice. The other nice thing is that the server management is centralized, which presumably lets you easily automate things like not running more than one backup at a time in order to not monopolize an Internet link. This can, of course, be managed with something like duplicity with appropriate ssh jobs kicked off from appropriate places, but it would be easier if the agent just handled it automatically. What are people s thoughts about this sort of thing?

Stefano Zacchiroli: cross-distribution meeting on application installer

saluting the cross-distro app installer meeting I just want to wish luck to the cross-distribution meeting on applications installers, which is starting today. For some background, check out Vincent's nice blog post on the subject. As far as my memory goes, this is one of the first attempts if not the first one to get representative of as many distros as possible around the same table to design, hack, and discuss a common topic of interest. We don't know yet how it will turn out, but the approach to the meeting has been a commendable one: aim at as much diversity as possible (first gathering people informally to get some ballpark dates, then with an announcement sent to the most appropriate place), be transparent about the organization, and engage in keeping others informed ex post. As they say here in France, chapeau for this attempt at factual cross-distro collaboration. Debian is present at the meeting, represented by Enrico Zini and David Kalnischkies. Thanks to them (for being there), to Vincent Untz (for the organization), to Novell/SUSE (for sponsoring part of Debian attendees' expenses), and to Debian donors (whose donations have been used to sponsor the rest).

Michal Čihař: Going to FOSDEM 2011

It looks like I will make it again to visit FOSDEM this year. This time it won't be meeting of whole phpMyAdmin team, but just few members but I hope to meet other people in person as well. If you have something you would like to discuss with me in person there, it's best time to let me know so that we can schedule it. I'm going to FOSDEM, the Free and Open Source Software Developers' European Meeting PS: It looks like it might be good timing for Debian release party as well :-).

Filed under: Debian English Phpmyadmin Suse 0 comments Flattr this!

Benjamin Drung: Poll: How to call the library?

Dear lazy web, I am writing a library that implements the Units Policy of Ubuntu. This library can be used to format sizes (of files, disks, memory, etc.) for displaying them and the other way around. For example, the size 12345 bytes can be formatted to 12.3 kB, 12.1 KiB, or 12.1 KB depending on the configuration. An other function will give you the possibility to convert 12.3 kB, 12.1 KiB, or 12.1 KB back to 12345 bytes. The preferred binary unit can be configured by an text file (per system and per user) and by an environment variable. The library is written in C and doesn t have any dependency. Therefore it can be used by any program on any desktop environment. Binding to other languages like C++, Python, Java are planned. Before creating a project for this library, I need a name for the library. The current prototype is called libdisplayunits, but this name reflect only the half of the project. It does not indicate that the library can be used for inputting sizes. What do you think? What is the best name for the library? Please poll and/or comment below.

<script charset="utf-8" language="javascript" src="http://static.polldaddy.com/p/3972978.js" type="text/javascript"></script> <noscript> View This Poll
survey software </noscript>

Benjamin Drung: Poll: libkibi VS libbyteprefix (or: How to call the library? 2)

Dear lazy web, thanks for voting the best name for the library that helps implementing the Units Policy of Ubuntu. The winner with the most votes has been libkibi. There was one suggestion that got my attention: libbyteprefix. The suggestion came too late to have a chance. Therefore I ask you again to vote for the library name, but this time only these two names are open for voting: libkibi and libbyteprefix. libkibi is short and sounds good. On the other hand, libbyteprefix describes exactly what the library is supposed to do. I was asked to use a free (as in freedom) poll-service. So please go to the Selectricity unitspolity vote and select your favorite. Thanks.

17 January 2011

Joey Hess: the conversion

It's summer in Spain, and outside there's a dry heat, but in the hack lab, it's frigid. I'm sitting across the table from Otavio, and each of our laptops is showing the same identical screen of text, editing the same file. Slowly. The file is located in Colorado, and we're fighting serious conference lag, plus transatlantic lag. Type a few lines, wait a minute, hope it shows up, and restart the converter to see if it worked. We're struggling with figuring out how to make it work, and pair programming across the table like this helps -- although we also struggle with each others accents sometimes, so we're eventually mostly communicating via text on the computers. Eventually Otavio rolls off to bed. Later, everyone else in the conference goes to bed or whatever -- I don't really notice. At some point a cleaning woman comes in, and seems suprised to see me. I muster up enough Spanish to ask if this muy fio air conditioning can be turned off. A bit later, I make this commit and then go out, look at storks in early morning light, and go to bed.
r59731   joeyh   2009-07-24 12:30:52 -0400 (Fri, 24 Jul 2009)   1 line
up to r15000; also skip the packages/po accient revs
That would have been 6 am local time. The log shows us back at it by 11 am local. There's so much detail in these logs, and it's hard to tell how any individual bit will turn out useful, but the value in aggregate is undeniable and that's why we struggle to retain it as technology marches on.
A few days later and the conversion is caught up to the commits documenting it. Everyone goes home. Later still, and work is done to verify the conversion. Problems are found, fixed. The worst of them involves the Danish language, and absolute evil. Tempers flare, and I have the last argument I'll ever have the pleasure of having with Frans. At some point in there, I do rather more coding in C++ than I ever have before, or ever wanted to. Frans's objections are addressed by that, more or less. We're using a weird tool that was developed as a kind of one-off for another project, but happens to meet our needs pretty well -- but we're stuck dealing with its bugs. And its need for copious handholding and historical research. The file we started out editing grows to two thousand lines, and now covers ten years of odd bits of history.
Suddenly it's a year and a half later. I'm in a cabin in winter, Otavio is down in summer, and I'm once again dealing with lag (dialup lag this time). We run the converter again -- one last time. And a time more.. and four or five more "last times". Finally, after a longish day, it's done.
This was not a major project, just some little bits of time here and there, maybe a week's worth in total, spread out over a few years and four or five people. It hardly seems worth writing about, just another codebase converted from Subversion to Git, a bit behind most of the other ones. Just thought I'd give you a glimpse behind the curtain.

Aigars Mahinovs: Nexus S review incoming

This is just a quick note that I broke down and replaced my ageing iPhone 3G with a brand new unlocked Google Nexus S. I got it in my hands last Saturday and by the end of the first day it did 90% of what I used my iPhone for and by the end of day two it did 100% of what I used my phone for before this and did it better than the iPhone. I will give it a week and then write up a bigger thing on all the good and bad things I notices from my migration from iPhone to Nexus S. If you want to know something specific about Nexus S, Android 2.3, its interaction with Debian/Ubuntu, this is the time to ask.

Phil Hands: Cloud Expo Europe

Debian will be exhibiting at next month's Cloud Expo Europe on the 2nd & 3rd of Feb.
(well I will be if you fancy keeping me company please add yourself to this page)If you fancy visiting the show, you can register for free before midnight on the 1st of February (after that it costs 15.00 GBP)oh, and later the same week,I'm going to FOSDEM, the Free and Open Source Software Developers' European Meeting

Michal &#268;iha&#345;: Gammu documentation

I've made cleanup in Gammu documentation and the only place where to look for it is now The Gammu Manual. Reason for this is simple - single documentation format allows easy cross-referencing and is easier to maintain than mixed sources we used to have before. Thanks to using Sphinx it is possible to generate various outputs, you can see the HTML version on the website, you can download PDF variants and the same source is being used for generating man pages. PS: You can find more details on generating other formats in the documentation.

Filed under: English Gammu 0 comments Flattr this!

Julien Viard de Galbert: Triaging bugs is not that hard !

There is a big thread on -devel that started about Forwarding bugs upstream, the idea is that it should be the maintainer s job. While it s true in general, some understaffed teams are so flooded by bugs that they just can t do it.
I ve seen more than once the X Strike Force asking a user to report upstream (pointing to the relevant BTS) and come back with the tracking URL to be attached to the bug. This was mostly for driver issue if I remember well.
And most users did it, so thanks! Then the thread evolved in lots of directions, I m not following all of it, but there was the question of the bug triaging and as I m doing it for the X packages, I want to share about my experience here. The available triaging docs The debian wiki already have a general Bug Triage page.
The thread pointed to the KDE Bug triaging page.
The X Strike Force has some pages, one on the wiki and a new one more specific to bug triaging on the alioth project s doc.
And when it comes to closing bugs with version information, there is a good read on the wiki: Bugs Version Tracking.
Finally don t hesitate to reopen the BTS documentation whenever you need to write to control@b.d.o ! Now if all those docs don t help you, then try to improve the docs as a first step ! ;) Talk with the maintainer(s) Well I did met Julien Cristau and Cyril Brulebois in person during mini-DebConf Paris/2010 that certainly helped. But we exchanged several mails, or met on IRC channel #debian-x.
But the triaging rules of the X Strike Force (linked above) were pretty clear anyway. My current process This process has evolved over time with the response of several contributors either to help or to fix to my mistakes, thanks!
  1. List bug from UDD: XSF unstable bugs sorted by date.
  2. Pick an old bug (once you answer and UDD runs, the bug moves down the list, so there are always old bugs on the top).
  3. Read the report, don t be afraid by the subject, it might be unrelated, the bug might be merged with other bugs, then reading the other bugs can help.
  4. If relevant try to reproduce it. (most bugs in X are hardware related or specific to one setup, so it s not that often that I have something to test).
  5. Decide what to do, most of the time, I will ping the submitter.
  6. I m a mutt user, and to my help the bts command as a way to open the bug mbox in mutt, that makes it really easy to answer the bug.
    So I launch bts -m show $n where $n is the bug number, and I can then answer the bug from mutt.
    The triaging procedure suggest to ping -submitter, but using mutt it s easier to just reply and add a -quiet to the bug number (this avoid spamming the debian-x with all my pings).
  7. In the message, always try to be kind and helpful, most of the time ask to test newer versions.
  8. Keep track of the pinged bugs, well I do it for my blog but it will be useful anyway, see below.
  9. When the submitter answers, just react accordingly, if the bug can be closed, then close it ! ;)
  10. If after some time, the bug are still not answered, consider closing them (I haven t started that yet, but this will probably get the bug number down faster:) ).
Note: Sometime I don t send the bug to -quiet so that other debian-x subscriber can have a look at the bug. They then can give me advices, or close the bug directly with a stronger reason than any I could have found. ;) To submitters, please keep the bug CCed when you answer, thanks. For the skeptics Yes as a newbie I did some mistakes, but I was always corrected in a nice way, with explanations. Also I try to have a conservative approach, I don t close bugs if I m not sure, I ask if I can close them, so it might take longer, but in the end the effect is the same for the bug, but not the same to the submitter ! ;) Also this work might look boring, but it s actually rewarding, most of the answers I received included a cheerful message thanking me for following up on the bug. Not to mention all I ve learned on Xorg. Conclusion As long as I have time for it, I ll continue to triage bugs and I d love to see others coming to help me on X or probably better go and help some other teams that also need help, and flood the planet with bug triaging reports ! ;) Get involved !

Dirk Eddelbuettel: Keeping simple things simple

My friend Jeff deserves a sincere congratulation for finally unveiling his rebranded R consultancy Lemnica. One notable feature of the new website is a section called esoteric R which discusses less frequently-visited corners of the R world. It even boasts its own CRAN package esotericR with the example sources. esoteric R currently holds two articles. Jeff had sent me the one about introducing closures a while back, and I like it and may comment at another time. What caught me by surprise when Lemnica finally opened was the other article: R calling C. It is a fine article motivated by all the usual reasons that are e.g. mentioned in the Google Tech Talk which Romain and I gave last October about our work around Rcpp. But it is just not simple. Allow me to explain. When Jeff showed this C language file
#include <R.h>
#include <Rinternals.h>
SEXP esoteric_rev (SEXP x)  
  SEXP res;
  int i, r, P=0;
  PROTECT(res = allocVector(REALSXP, length(x))); P++;
  for(i=length(x), r=0; i>0; i--, r++)  
     REAL(res)[r] = REAL(x)[i-1];
   
  copyMostAttrib(x, res);
  UNPROTECT(P);
  return res;
 
and then needs several paragraphs to explain what is going on, what is needed to compile and then how to load it --- I simply could not resist. Almost immediately, I emailed back to him something as simple as this using both our Rcpp package as well as the wonderful inline package by Oleg which Romain and I more or less adopted:
library(inline)  ## for cxxfunction()
src <- 'Rcpp::NumericVector x = Rcpp::NumericVector(xs);
        std::reverse(x.begin(), x.end());
        return(x);'
fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
fun( seq(0, 1, 0.1) )
Here we load inline, and then define a three-line C++ program using facilities from our Rcpp package. All we need to revert a vector is to first access its R object in C++ by instantiating the R vector as a NumericVector. These C++ classes then provide iterators which are compatible with the Standard Template Library (STL). So we simply call the STL function reverse pointing the beginning and end of the vector, and are done! Rcpp then allows us the return the C++ vector which it turns into an R vector. Efficient in-place reversal, just like Jeff had motivated, in three lines. Best of all, we can execute this from within R itself:
R> library(inline)  ## for cxxfunction()
R> src <- 'Rcpp::NumericVector x = Rcpp::NumericVector(xs);
+         std::reverse(x.begin(), x.end());
+         return(x);'
R> fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
R> fun( seq(0, 1, 0.1) )
 [1] 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0
R>
Lastly, Jeff shows a more complete example wherein a new vector is created, and any potential attributes are copied as well. Naturally, we can do that too. First, we used clone() to make a deep copy (ie forcing creation of a new object rather than a mere proxy) and use the same R API function he accessed---but it our case both prefixed with ::Rf_ for R remapping (to protect clashed with other functions with identical names) and a global namespace identifier (as it is a global C function from R).
R> library(inline)
R> src <- 'Rcpp::NumericVector x = Rcpp::clone<Rcpp::NumericVector>(xs);
+         std::reverse(x.begin(), x.end());
+         ::Rf_copyMostAttrib(xs, x);
+         return(x);'
R> fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
R> obj <- structure(seq(0, 1, 0.1), obligatory="hello, world!")
R> fun(obj)
 [1] 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0
attr(,"obligatory")
[1] "hello, world!"
R> obj
 [1] 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
attr(,"obligatory")
[1] "hello, world!"
R>
Both the obj variable and the new copy contain the desired data attribute, the new copy is reversed, the original is untouched---and all in four lines of C++ called via one inline call. I have now been going on for over one hundred lines yet I never had to mention memory management, pointers, PROTECT or other components of the R API for C. Hopefully, this short writeup provided an idea of why Romain and I think Rcpp is the way to go for creating C/C++ functions for extending and enhancing R.

Julien Danjou: OrgCamp 2011

I'll be at the OrgCamp 2011 in Paris next Saturday. Flattr this

Rapha&#235;l Hertzog: Review of my Debian related goals for 2010

Last year I shared my Debian related goals for 2010 . I announced that I would not be able to complete them all and indeed I have not, but I have still done more than what I expected. Let s have a look. Translate my Debian book into English and get it published: NOT DONE Or rather not yet done. It s still an important project of mine and I will do it this year. When I wrote this last year, I expected to find a publisher that would take care of everything but we failed to find a suitable one so we re going to do it ourselves. Cleanup the dpkg perl API and create libdpkg-perl: DONE libdpkg-perl has been introduced with dpkg 1.15.6. Create dpkg-buildflags: DONE dpkg-buildflags has been introduced with dpkg 1.15.7. It s not widely used yet and it won t really be until debhelper 7 supports it (see #544844). It would be nice to see progress on this front this year. Ensure the new source formats continue to gain acceptance: DONE The adoption rate has been steady, it clearly slowed down since the freeze though. I have implemented quite a few features to satisfy the needs of users, like the possibility to unapply the patches after the build, or the possibility to fail in case of unwanted upstream changes. Design a generic vcs-buildpackage infrastructure to be integrated in dpkg-dev: NOT DONE I believe it s something important on the long term but it never made it to my short-term TODO list and it s unlikely to change any time soon. Continue fixing dpkg bugs faster than they are reported: PARTLY DONE We have dealt with many bugs over the year, but we still have 20 more bugs than at the start of last year (370 vs 350). We re not doing bad compared to many other Debian teams but we can still benefit from some help. Start here if you re interested. Enhance our infrastructure to ease interaction between contributors and to have a better view of how each package is maintained: NOT DONE I am convinced that we need something to have a clearer idea of the commitments made by each contributor. I don t put the same amount of care in maintaining smarty-gettext that I do on dpkg. If we had a database of the stuff that we know we don t do well enough, it s easier to point new contributors towards those. Anyway, this project is still unlikely to come to the top of my priorities any time soon. Work on the developers-reference: NOT DONE We have switched the Maintainer field to debian-policy@lists.debian.org to have more review of the changes suggested through the bug tracking system but that has not changed much on the global situation. I still hope to become more active on it sometimes this year. Maybe by trying to make it more fun and creating the text for some of the wishlist bugs as blog articles first. Rewrite in C the last Perl scripts provided by the dpkg binary package: DONE Dpkg 1.15.8 was the first version working without Perl, I announced it in July. Integrate the 3-way merge tool for Debian changelogs in dpkg-dev: DONE dpkg-mergechangelogs is part of dpkg-dev since 1.15.7. I enjoy it regularly. Unfortunately it doesn t work well for cherry-picks. Would be nice to see this fixed, anyone up to the task? :-)
Click here to subscribe to my free newsletter and get my monthly analysis on what s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

One comment Liked this article? Click here. My blog is Flattr-enabled.

David Welton: The Kindle and Book Sharing

I'm very pleased with the Kindle; all of a sudden, I can quickly, easily, and fairly cheaply buy lots of English language books that otherwise I would have had to order (not cheap, not quick). Furthermore, they're not cluttering up my house, either! Not that I view books as clutter, but sooner or later we're going to have to move out of the apartment we're in, and the less I have to haul around, the better. However, the fact that it's not easy to share the books with my wife is starting to annoy me. I could always give her the Kindle itself to read something, but that would deprive me of not only that book, but all the others I have as well. Indeed, the difficulty of the problem lies in the difference between digital goods and physical goods. Real books are: Ebooks are: Setting aside the costs and benefits to society of DRM, intellectual property, and so on, let's take the point of view of Amazon, book publishers and authors, who all want to maximize their income, and of readers, who want as much freedom as possible, and also, for the sake of argument, believe that paying for books leads to more money for authors, which leads to more books. I think the trick is to try and make the ebook market as similar as possible to that for real books. For instance, if you had the freedom to resell an ebook quickly and easily, the price would likely start dropping quickly, as people started selling their "used" copies of books they didn't want to keep. This would be good for the consumer, but potentially quite bad for authors, as fewer new copies would be sold. Sure, we have used book stores, but the practical limitations involved, and the fact that real books show some wear and tear, mean that there's still an incentive to buy new books. On the other hand, an ebook ought to be a bit cheaper than a real book if it's something you'd consider reading and then selling, because you no longer have that option, so the lack of freedom should be priced in to the original price. The other problem they seem to be having is with the lending of books. If you lend a real book, you can only lend it to someone who you see with some frequency, so there's already a big geographical limitation. Also, while they're reading it, you obviously can't. You have to have some trust in them, that they'll give it back to you sooner or later, so you won't just loan it to anyone: if you wanted a transaction with more finality, you'd simply sell it. Currently, Kindle books can only be lent for two weeks, and you can't access it while the other person has it. I think the latter part of that is fair, but two weeks seems like an arbitrary number. Remember though, that they don't want you to be able to just give someone a book, with no strings attached and no return-by date, because then you could also sell it, undercutting their store prices, and so on, as above. So to make lending work, they have to have some "strings attached". I think "two weeks" is a fairly arbitrary string though. What would work better? Perhaps a way that permanently differentiates the original purchaser from anyone borrowing it. For instance, a "recall now" button? This button would, when the original purchaser pressed it, disable it on the borrower's device and reenable it on the buyer's device. This would prevent secondary markets, or at least limit them, because the 'borrowed' books would be inferior to the originals. And yet it might still work well for lending; I could lend books to people who trust me without problems. Interestingly, I became aware of this site the other day: http://www.kindlelendingclub.com/ so it looks like whatever Amazon does, there are going to be tradeoffs involved. And of course, there is the point of view that all of the costs involved by DRM are too high, but that's a fairly involved discussion in its own right. What do you think?

Michal &#268;iha&#345;: Useful parameters for Lintian

Every time I review some package for sponsoring, I usually end up suggesting to fix some of more picky things which lintian finds. It looks like everybody is using default set of checks and does not look at others. Indeed the defaults tell about things which are really errors or need override, but --pedantic and --display-info show also useful hints. However you need to take them more as a suggestions than as a errors. What I usually find interesting is --pedantic, because it tells about several things, which might be good idea to have in your package. Missing Homepage in debian/control is one of them and I believe this should exist for every package which has a homepage. The --display-info (or just -I) is also useful, for example it does spell checking of texts in binaries and man pages. Registration into doc-base is also useful integration into the system and you should definitely do it if the package provides documentation in HTML. To run these checks every time, you can create alias in your ~/.bashrc:
alias lintian='lintian -I --pedantic'

Filed under: Debian English 7 comments Flattr this!

16 January 2011

Gregor Herrmann: RC bugs 2011/01 - 2011/02

new year, new RC bugs most of the fixes below are for newly reported bugs in versions that don't affect squeeze. the rest are mainly uploads of packages or patches prepared by others.

Dirk Eddelbuettel: Plotting overbought / oversold regions in R

The good folks at Bespoke Investment Group frequently show charts of so-called overbought or oversold levels; see e.g. here for the most recent global markets snapshot. Classifying markets as overbought or oversold is a popular heuristic. It starts from computing a rolling smoothed estimate of the prices, usually via a (exponential or standard) moving average over a suitable number of days (where Bespoke uses 50 days, see here). This is typically coupled with a (simple) rolling standard deviation. Overbought and oversold regions are then constructed by taking the smoothed mean plus/minus one and two standard deviations. Doing this is in R is pretty easy thanks to the combination of R's rich base functions and its add-on packages from CRAN. Below is a simply function I wrote a couple of months ago---and I figured I might as well release. It relies on the powerful packages quantmod and TTR by my pals Jeff Ryan and Josh Ulrich, respectively.
## plotOBOS -- displaying overbough/oversold as eg in Bespoke's plots
##
## Copyright (C) 2010 - 2011  Dirk Eddelbuettel
##
## This is free software: you can redistribute it and/or modify it
## under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 2 of the License, or
## (at your option) any later version.
suppressMessages(library(quantmod))     # for getSymbols(), brings in xts too
suppressMessages(library(TTR))          # for various moving averages
plotOBOS <- function(symbol, n=50, type=c("sma", "ema", "zlema"), years=1, blue=TRUE)  
    today <- Sys.Date()
    X <- getSymbols(symbol, src="yahoo", from=format(today-365*years-2*n), auto.assign=FALSE)
    x <- X[,6]                          # use Adjusted
    type <- match.arg(type)
    xd <- switch(type,                  # compute xd as the central location via selected MA smoother
                 sma = SMA(x,n),
                 ema = EMA(x,n),
                 zlema = ZLEMA(x,n))
    xv <- runSD(x, n)                   # compute xv as the rolling volatility
    strt <- paste(format(today-365*years), "::", sep="")
    x  <- x[strt]                       # subset plotting range using xts' nice functionality
    xd <- xd[strt]
    xv <- xv[strt]
    xyd <- xy.coords(.index(xd),xd[,1]) # xy coordinates for direct plot commands
    xyv <- xy.coords(.index(xv),xv[,1])
    n <- length(xyd$x)
    xx <- xyd$x[c(1,1:n,n:1)]           # for polygon(): from first point to last and back
    if (blue)  
        blues5 <- c("#EFF3FF", "#BDD7E7", "#6BAED6", "#3182BD", "#08519C") # cf brewer.pal(5, "Blues")
        fairlylight <- rgb(189/255, 215/255, 231/255, alpha=0.625) # aka blues5[2]
        verylight <- rgb(239/255, 243/255, 255/255, alpha=0.625) # aka blues5[1]
        dark <- rgb(8/255, 81/255, 156/255, alpha=0.625) # aka blues5[5]
      else  
        fairlylight <- rgb(204/255, 204/255, 204/255, alpha=0.5)         # grays with alpha-blending at 50%
        verylight <- rgb(242/255, 242/255, 242/255, alpha=0.5)
        dark <- 'black'
     
    plot(x, ylim=range(range(xd+2*xv, xd-2*xv, na.rm=TRUE)), main=symbol, col=fairlylight) 		# basic xts plot
    polygon(x=xx, y=c(xyd$y[1]+xyv$y[1], xyd$y+2*xyv$y, rev(xyd$y+xyv$y)), border=NA, col=fairlylight) 	# upper
    polygon(x=xx, y=c(xyd$y[1]-1*xyv$y[1], xyd$y+1*xyv$y, rev(xyd$y-1*xyv$y)), border=NA, col=verylight)# center
    polygon(x=xx, y=c(xyd$y[1]-xyv$y[1], xyd$y-2*xyv$y, rev(xyd$y-xyv$y)), border=NA, col=fairlylight) 	# lower
    lines(xd, lwd=2, col=fairlylight)   # central smooted location
    lines(x, lwd=3, col=dark)           # actual price, thicker
    invisible(NULL)
 
After downloading data and computing the rolling smoothed mean and standard deviation, it really is just a matter of plotting (appropriate) filled polygons. Here I used colors from the neat RColorBrewer package with some alpha blending. Colors can be turned off via an option to the function; ranges, data length and type of smoother can also be picked. To call this in R, simply source the file and the call, say, plotOBOS("^GSPC", years=2) which creates a two-year plot of the SP500 as shown here: Example chart of overbought/oversold levels from plotOBOS() function This shows the market did indeed bounce off the oversold lows nicely on a few occassions in 2009 and 2010 --- but also continued to slide after hitting the condition. Nothing is foolproof, and certainly nothing as simple as this is, so buyer beware. But it may prove useful in conjunction with other tools. The code for the script is here and of course available under GPL 2 or later. I'd be happy to help incorporate it into some other finance package. Lastly, if you read this post this far, also consider our R / Finance conference coming at the end of April.

Edit: Corrected several typos with thanks to Josh.

John Goerzen: 24 hours with Jacob

Friday, I wrote about the train trip Jacob and I were planning to take. Here s the story about it. Friday night, Jacob was super excited. He was running around the house, talking about trains. I had him pack his own backpack with toys this time, which were you guessed it trains. Plus train track. His usual bedtime is around 7. He was still awake in his room at about 11, too excited to sleep. The train was an hour late into Newton, so got up, got ready, and then went into Jacob s room at 3:15AM. I put my arm around him and said his name softly. No response. I said, just a little louder, Jacob, it s time to wake up to go to the train station. There was about a 2-second pause and then he sat bolt upright rubbing his eyes. A couple seconds later, in a very tired but clear voice, OK dad, let s go! That is, I believe, a record for waking up speed for Jacob. We went downstairs, got coats, mittens, hats, etc. on, made sure we had the stuffed butterfly he always sleeps with, and went out the door. As usual, Jacob chattered happily during the entire 15-minute drive to the Amtrak station. One of these days I need to remember to record it because it s unique. He described things to me ranging from the difference between freight and passenger trains, to what the dining car is all about, to tractors and how to ride them safely. Newton has some winter lights , and a few places still had Christmas lights, which were of course big hits. We had to wait a few minutes at the Amtrak station, and Jacob hadn t shown any signs of slowing down yet. He wanted to look at every Amtrak poster, picture, logo, or sign in the building. This generally meant me holding him up high while he leaned over to touch it and make out a few words. Then, of course, he would pick out minute details about the trains, such as how many coach cars he thought they had, and we d visit about that for awhile. We got on at about 4:20. We found our seats, and Jacob showed no signs of calming down, despite having had only 4 hours of sleep (instead of his usual 11) so far. We checked out the buttons for lights. And, of course, he excitedly yelled out, Dad, the train is moving! He spent the next while mostly watching out his window, but also still exploring his space. Finally at about 5, I said, Jacob, I am really tired. I am going to sleep now. Will you sleep too? His response: Oh sure dad, I will sleep with my eyes open! As a result, no sleep was had for Jacob, and only a little for me. The dining car opens for breakfast at 6:30, which is normally a rather foreign time for breakfast on the train for us. But we were both awake so I figured might as well go. So Jacob and I went to the dining car. We sat with a woman going from New Mexico to Lawrence for her grandpa s funeral, though it was expected and she was having a good time on the train. Jacob turned completely shy, and refused to say a word, except maybe a few whispered into my ear. He got his favorite railroad French toast, and had me drizzle some syrup on it. I used the word drizzle for syrup the first time he had French toast on the train, and if I fail to use that word in the dining car, I will hear about it in no uncertain terms from Jacob. He loved his dining car breakfast, but we spent about an hour and a half there. He was really slow at eating because his face was pressed up against the window so much. But that was just fine; we had nowhere else to be, the person eating breakfast with us enjoyed visiting (and, apparently, scaring the dining car staff with tales of bears in the New Mexico mountains). This was what the train trip was all about, after all. We played in the lounge car for awhile. The almost floor-to-ceiling wrap-around windows provided a great view for him, and more opportunities to press his face against a window. We talked about freight trains that he saw, noticed the snow on some of them. Then we found the back of the train and he got to look out the back window. Back at our seat, he played with his toys for about 10 minutes, which was all he used them on the entire trip. There was just too much else to enjoy. When we used the restroom on the train, he d comment on how much he liked the Amtrak soap. It smells SO very very good! He wanted to wash his hands on the train. By late morning, he had decided: Dad, I LOVE this Amtrak soap. It smells like peaches! Shall your hands smell like peaches too? And, when we d get back up to our seats, he d put his hands in my face, saying, Dad, smell that! My hands smell like peaches! It was from the AMTRAK SOAP! At some point, he discovered the airline-style safety brochures in the seat back pockets. These were filled with diagrams of the train car, a few photos, and lots of icons with descriptions. I don t know how many times I read the thing to him, or really how many times he then recited it to me from memory. It was a lot. He spent hours with those brochures. Jacob had already told me that he wanted pizza for lunch, so I got him the kid-sized pizza. It wasn t all that big, and he could have devoured at least half of it when hungry. But he was getting really tired and ate only a few bites of pizza and a few chips. Pretty soon he was leaning up against me, the window, and eventually had his head on the table in some tomato sauce. But he didn t quite fall asleep by the time we went back to our seats, and of course was wide awake by that point. Jacob loves spotting the word Amtrak on things. It was very exciting when he noticed his orange juice at breakfast, and milk at lunch, were Amtrak juice and Amtrak milk due to the logo on the cups. At dinner he noticed we had Amtrak plates, and when I pointed out that his metal fork had the Amtrak logo on it, he got very excited and had to check every piece of silverware within reach. Dad, I have an Amtrak fork too! . And dad, YOU also have an Amtrak fork! We ALL have Amtrak forks! *cackling laughter* I finally insisted that Jacob lay down for some quiet time. I closed the curtains, and he finally did fall asleep less than an hour before our arrival into Galesburg. So by 2:15 he was up to 4.75 hours of sleep, I guess. We stopped in the train station briefly, then started our walk to the Discovery Depot Children s Museum, which was right nearby. Although I made no comment about it, Jacob said, Dad, there is a train museum RIGHT HERE! Yes, you re right Jacob. I can see a steam engine and some cars here. Let s go in! I don t think it s open today. It IS open shall we go check? It wasn t, and that was mighty sad though when he spotted another old caboose sitting outside the children s museum, the day suddenly seemed brighter. He complained of how cold he was, although my suggestion that he stop walking through the big piles of snowdrifts was met with a whiny, But dad, I WANT to do that! We went inside the museum (having to walk right buy the locked caboose thankfully the people at the desk promised to unlock it for us when we were ready) and Jacob started to explore. There was some wooden play trains big enough for children to climb in which he enjoyed, but in general he went from one thing to the next every minute or two as he does when he s really tired or overstimulated. Until, that is, he discovered the giant toy train table. It had a multi-level wooden track setup, and many toy trains with magnetic hitches. It was like what we have at home, only much bigger and fancier. He spent a LONG time with that. We then briefly explored the rest of the museum and went out into the caboose. It wasn t the hit it might have been, possibly because there are several at the Great Plains Transportation Museum that he gets to go in on a somewhat regular basis. After that, he was ready to go back into the museum, but I was feeling rather over-stimulated. On a day when the highs were still well below freezing, it seemed just about every family in Galesburg was crowded into the children s museum, making it loud and crowded which I don t enjoy at all. So I suggested maybe it was snack time instead. A moment s thought, then he started to pull me out of the caboose before I could get my gloves back on Yes dad, I think it IS snack time. Let s go. Let s go NOW! We walked over to Uncle Billy s Bakery (Google link or minimal website). Jacob spotted some sugar cookies shaped like mittens. Despite my reluctance to get him more sugar, he was so excited plus I had barely prevented a meltdown at lunch by promising him that he would get dessert later in the day so he picked two red mitten cookies. I got myself a wonderful peach muffin and a croissant and we sat down at one of the tables by the window. I taught Jacob how to hang his coat on his chair and he lit into those cookies. I spotted a guy at the next table over wearing a BNSF jacket, and asked him if he worked for the railroad. He had retired as an engineer a couple of years ago, and had worked various jobs before that. He grew up in Manhattan, KS and so was interested in our trip and very friendly. While we visited, Jacob devoured his cookies and increasing portions of my snack as well. He told us about a new shop The Stray Cat just two stores down that was having a grand opening event today. They make decorations and art out of basically discarded items, and had some really nifty things that I may have bought had I not been wanting for space in our backpack. Then I spotted Sweets Old-Fashioned Ice Cream, Candy, and Soda Shop across the road. I figured he d love it and I was already in for the sugar so might as well. He picked out some birthday cake flavor ice cream for himself. I got huckleberry ice cream, which he insisted on calling purpleberry and managed to get some tastes of as well. After that, we went to the train station. It was about an hour until our train would be there. I wasn t sure if we d find enough to do, but I shouldn t have worried. Earlier, we had made the happy discovery that the station s restroom featured the Amtrak soap, so there was that. Then there was the model Amtrak train in the ticket window, which Jacob kept wanting to look at while I d hold him. And also, the California Zephyr came in. We watched it arrive from the station window, saw people get off and on, and saw it leave maybe the first time Jacob has witnessed all that in person. And, of course, we looked at the pictures in that train station. The ticketmaster gave Jacob a paper conductor s hat with puzzles and mazes on the back side. And then it was time to get onto our train back home. We ate dinner Jacob again ate little and almost fell asleep and got back to our seats. I let Jacob stay awake until about 8, when he was starting to get a bit fragile. It took him awhile to fall asleep, but he finally did at about 8:30. Today he s still been all excited. He will randomly tell us about bits of the trip, that the man at supper called his grilled cheese sandwich piece little when it was really big, what we did at the ice cream store, etc. And I do think that he is now a train safety expert. All in all, I think that is probably the most excitement he s ever had in 24 hours and it was a lot of fun to be with him for it!

Nikita Youshchenko: Render PDFs on GPU?

Recently I tried once more to read a pdf file on n900. And as usual, it worked bad. Not only because of small screen that particular pdf had two-column formatting, and n900 s screen is wide enough to display one of these two columns with a readable font. Much worse was extremely slow redrawing. Finger scrolling on a touch screen is cool but only if system has enough power to immediately show content as it is scrolled to be visible. In practice I try to scroll with finger, then a second or two pass, then page on the screen starts to move, replaying all my finger movement during that second or two (which was not very direct because of no quick visual feedback), by the way text at borders goes out of screen for a moment, and that area becomes broken and is redrawn only after several more seconds. And then, in less than a minute (after reading displayed content), all that has to be repeated. In total, this makes UI almost unusable. While looking at this, I thought why not use GPU here? This device has a powerful GPU, enough even for non-trivial games. So why not convert a pdf page into a scene description for GPU, and then get zero-delay scrolling and scaling at least? This could be useful even on desktops sometimes pdf viewers do have interactivity problems, especially at very-large scale (400% or more) which is needed to view board schematics diagrams. Anyone ever implemented this? Or maybe wants to implement?.. :)

Daniel Kahn Gillmor: resetting passphrases for mapped LUKS volumes

Let's say you set up a machine using an encrypted disk with LUKS (debian-installer's partman makes this wonderfully easy!). You choose an initial passphrase, get the machine working, and it's working great. Then, you need to restart it, and realize that (for whatever reason) you've forgotten or lost the passphrase for the volume. oops! (i'm sure this has never happened to you -- let's just pretend it's your less-fortunate friend). If your system is still running, and you have superuser access to it, you can actually set a new passphrase for the LUKS volume using information that the dm-crypt kernel module has about the in-use mapping. In my examples, i'll imagine that the source volume is /dev/XXX2 and the exported cleartext volume is known by the device-mapper as XXX2_crypt In the bigger picture, this should serve as a reminder that even though your disk is encrypted, if someone gets live access to the superuser account on a system with the encryption keys loaded, your data is no longer secret. Before you do any tweaking, you might want to back up your LUKS header, just in case:
0 root@example:~# umask 077
0 root@example:~# cryptsetup luksHeaderBackup /dev/XXX2 --header-backup-file XXX2.luksheader.backup
0 root@example:~# 
Maybe also copy that off the machine, since a copy of the LUKS header stored within its own volume isn't terribly useful for a backup-recovery situation. You might also be interested in looking at the contents of the LUKS header:
0 root@example:~# cryptsetup luksDump /dev/XXX2
LUKS header information for /dev/XXX2
Version:       	1
Cipher name:   	aes
Cipher mode:   	cbc-essiv:sha256
Hash spec:     	sha1
Payload offset:	2056
MK bits:       	256
MK digest:      93 51 6c 66 ec ce 32 54 6f 6b 52 d1 27 9b 5a 62 6f 6b 52 d1
MK salt:       	b2 ca 20 49 9f 78 49 c2 fe 15 b4 0f 74 11 23 49
               	64 9e 61 bb f2 82 60 47 a5 76 fa a4 24 0e 5a 7e
MK iterations: 	10
UUID:          	052f1da0-21a1-11e0-ac64-0800200c9a66
Key Slot 0: ENABLED
	Iterations:         	218733
	Salt:               	f2 ae 8c 53 48 a5 f0 bf e1 2c 06 5f 5a bd ff d9
	                      	9a 2e d1 49 3a 63 f8 49 82 ed ae 86 7b 7b 7e 76 
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
0 root@example:~# 
Now, the fix: We pull the live "master key" from the running device map, and fill a new luksKeySlot from it (this example uses bash's <() syntax for process substitution -- if you use a different shell, i'm sure you can find a different way to do it):
0 root@example:~# cryptsetup --master-key-file <( 
> dmsetup --showkeys table   awk '/^XXX2_crypt: /  print $6  '   tr -d '\n'   perl -e 'print pack("H*", <STDIN>);' 
> ) luksAddKey /dev/XXX2
Enter new passphrase for key slot: abc123
Verify passphrase: abc123
2 root@example:~# 
(note that the luksAddKey invocation above returned an error code of 2 even though it succeeded. I think this is a bug in cryptsetup's return code, not a bug in the password resetting -- it should have returned 0 instead of 2). You can check to see that a new key slot was enabled by re-running cryptsetup luksDump And if you really want to double-check before you reboot, you can try enabling a third keyslot using the passphrase you just added, since this would not succeed if your new passphrase failed to unlock any of the existing keyslots:
0 root@example:~# cryptsetup luksAddKey /dev/XXX2
Enter any passphrase: abc123
Enter new passphrase for key slot: anotherpassphrase
Verify passphrase: anotherpassphrase
0 root@example:~# 
You can also get rid of the original keyslot (which you don't know the passphrase to) like this:
0 root@example:~# cryptsetup luksKillSlot /dev/XXX2 0
Enter any remaining LUKS passphrase: abc123
0 root@example:~# 
(the above commands were all demonstrated using debian testing, with cryptsetup 2:1.1.3-4 and dmsetup 2:1.02.48-4) Tags: cryptsetup, dmsetup, luks, tip

Next.

Previous.