Search Results: "nate"

19 January 2011

Nikita Youshchenko: Trinity project: KDE3 alive?

Trinity project in an attempt to keep KDE3 desktop compatible with ongoing changes elsewhere, and to some extent to fix bugs and add features into KDE3. It is for those us long-time KDE users who is upset by the way that KDE has chosen for KDE4. Unfortunately Trinity project looks small and underpowered, so it s future is unclear. Did not try it yet. There are debs, although some people think that it is hard to deal with those, because of KDE3 package namespace now used by KDE4, and also because of binary compatibility issues within Trinity. Why I don t like KDE4? Because I have no interest in eye candy, but due to heavy use of thin clients with apps running on remote server I do care about desktop resource consumption. A medium-class server (2 dual-core 2 GHz cpus, 16G ram) perfectly hosts 50 KDE3 sessions, served over vnc. Anything near to that with KDE4? I m not aware of any good-working kde4-over-vnc-over-network configuration, even for a single user over unloaded network. Looks like within KDE4, even with all effects turned off, every tiny operation results into near-fullscreen update. Spice is promising but requires per-session VMs which is a serious drawback when number of users is high.

Dirk Eddelbuettel: RcppBDT 0.1.0

The family of Rcpp packages just grew by one: the first 0.1.0 release of RcppBDT is now on CRAN. RcppBDT stands for Rcpp Boost Date_Time. It employs what we call Rcpp modules: a mechanism which provides easier ways to expose C++ functions and classes to R (and which bears some resemblance to Boost.Python---see this vignette about Rcpp modules for more details). And thus RcppBDT provides R users with access to (some) Boost Date.Time functionality. I used Boost Date.Time because it is widely known, useful and well tested. It can also be used in pure template mode not requiring linking (if one foregoes string parsing and formatting which is fine here as we get this from R already; it can be added via a #define in the header file). With that, the package is rather portable as it only needs to find the Boost headers. (And no, I have not yet added a configure option for that.) Basic usage follows Rcpp modules and provides the C++ class via a Reference Class in R. That means using a new constructor and assessing functions via the $ operator. Here is an example:
R> library(RcppBDT)
Creating a new generic function for "print" in "RcppBDT"
Creating a new generic function for "format" in "RcppBDT"
R> d <- new(bdtMod$date, 2011, 1, 18)
R> format(d)
[1] "2011-01-18"
R>
Not very exiting yet: we create a date, using explicit year, months and date arguments and then format it. Something more useful follows:
R> d$getDayOfWeek()
[1] 2
R> d$getModJulian()
[1] 55579
R> d$getFirstOfNextMonth()
[1] "2011-02-01"
R>
We can use accessors to extracts parts of the date, or use functions to convert the date to different representations such as a modified Julian date. Moreover, given a date, we can apply helper functions such as get me the date of the beginning of the next month. More interesting still are helper functions such as the ones below. Note that this also shows the alternate access method using wrapper functions I added for the package, this may be more familiar to most R users:
R> getNthDayOfWeek(third, Wed, Dec, 2010)
[1] "2010-12-15"
R> getLastDayOfWeekInMonth(Sat, Dec, 2010)
[1] "2010-12-25"
R> getFirstDayOfWeekInMonth(Sat, Dec, 2010)
[1] "2010-12-04"
R> getFirstDayOfWeekAfter(Wed, as.Date("2010-12-31"))
[1] "2011-01-05"
R>
This uses some of the constants Jan, Feb, ... , Dec; Sun, Mon, ..., Sat and first ... fifth defined in the package; normal integers can also be used. The third Wednesday in a month is also known as the IMM Date to Finance-heads; the example was borrowed from Whit's earlier rboostdatetime code on github. More examples are in the demo available with the package and accessible via demo(RcppBDT) once you install the package. As always, feedback would be welcome, both on the usefulness (or lack thereof) of the Boost Date.Time functionality as well as on the Rcpp modules wrapping. Additional Boost Date.Time functionality (durations, times, ...) may be added; contributions would be welcome. The rcpp-devel mailing list off the R-Forge page for Rcpp is the best place to start a discussion.

18 January 2011

Russell Coker: Conferences and People on the Autism Spectrum

There have been some recent discussions about issues related to the treatment of women at Free Software conferences, I ve written posts about Aspie Social Skills and Free Software [1] and Empathy, Autism, and Geeks [2] about this. But of course discussion continued on how Aspies supposedly cause problems that no-one seems to be noticing.Lisa of Chaotic Idealism wrote an interesting post about the NT social bias [3]. In summary NTs seem to think that Autism Spectrum Disorders are only about socialisation, not realising that for many people on the Spectrum it s sensory issues that are their main problem. Social problems are exacerbated by sensory issues and other causes of stress, so I think it s worth considering ways in which conferences can be planned to be less stressful for people on the Autism Spectrum, people who have SPD [4] that isn t associated with an ASD, and NTs who just get annoyed by loud noises etc.What Autism Conferences DoAutscape is one of only two conferences for people on the Autism Spectrum and the only one which clearly documents how they plan their conference [5]. The first noteworthy thing that they do is have badge colors to indicate what level of social interaction is desired by each delegate, I don t think that this is relevant to Free Software conferences as people who don t have sufficient social skills to suit at least a green badge probably won t be attending. But I think that when attending a conference about Free Software or any other equally geeky topic (if there is an equally geeky topic) it s worth keeping in mind the fact that there are probably a lot of people who would like to talk to you but lack the social skills to start a conversation.
No initiationRedPlease do not initiate any interaction with me.
Prior PermissionYellowPlease do not initiate unless I have already given you permission to approach me on a yellow badge.
Please initiateGreenI would like to socialise, but I have difficulty initiating. Please initiate with me.
NeutralWhite (or no badge)I am able to regulate my own interaction.
The next thing that they document is a black circle badge which indicates that the wearer shouldn t be photographed. Prior to reading that web site I wasn t aware of this being an Autistic issue, I was only aware of it being an issue for women who don t want zoomed-in pictures of themselves appearing on guys web sites (even pictures that aren t up-skirt or down-blouse can be unwelcome). A conference policy that prohibited photographs that zoom in on one person without that person s consent (or parental consent in the case of minors) would probably be a good idea.Another thing about badges is that it s a really good idea to have the delegate s name on both sides of the badge if the badge is attached in a way that permits it to turn. People on the Autism Spectrum tend to have some difficulty in recognising people and in remembering names. I find it inconvenient when someone expects me to recognise them but has their badge turned around so I can t see their name. Some people get really unhappy if they think that someone doesn t recognise them.Autscape has long breaks between activities and a leisure session each afternoon to allow delegates to recover from the stress of dealing with people. It seems to me that computer conferences in some cases could do with longer breaks between sessions. I find that a lot of the benefit of a conference is in what happens outside sessions and the standard practice of publishing videos of presentations makes personal meetings a more important part of the conference.Autscape has designated quiet spaces. I think that for computer conferences which have hack-labs (which seems to be most Free Software conferences nowadays even if they aren t labelled as such) it would be good to have some lab areas designated as quiet zones. I think that it is a really good thing to meet people you ve only known by email and then play some LAN based games against them and that this should be encouraged as part of a conference, but having that sort of thing separated from people who want to do some quiet coding is a really good idea. Whether people want to do quiet coding, read email, or just escape from the stress of a conference a quiet hack-lab would be a good place for it.An issue that s related to quiet spaces is the ability to escape from social situations. One of my pet hates is corporate meetings on boats, when the boat is in motion there is no escape. Corporate meetings that are only accessible by coach are also bad. Fortunately most conferences aren t like that.The Autscape web site states that they prohibit people from wearing perfume or aftershave to help people who are sensitive to smell. My observation of Free Software conferences is that encouraging everyone to have a shower every morning would be a good idea as there are stronger smells from unwashed people than from perfume.One thing that s interesting about the Autscape web site is that they have different color schemes available and have tested it in multiple web browsers including Safari and Konqueror (which apparently don t work so well). It s interesting to note that they test with such a variety of browsers including free software ones most corporations don t do that.Sensory Processing IssuesThe main SPD issues related to conferences seem to be noise and light related.For social events one difficulty that most people on the Spectrum seem to face is in listening to one person in a noisy crowded room. As well as that sudden noises and loud background noises can be very stressful. While it s sometimes impossible to avoid crowds (which can be another problem) it is often possible to select venues that have less noise problems. A venue with carpet on the floor and soft walls (anything other than brick or concrete) will be a lot quieter than one with hard surfaces that reflect noise. It seems that a quiet venue will benefit NTs as well, there seems to be a strong correlation between the price of food at restaurants and the amount of sound absorbing material on the floor and walls people who can afford a good dining experience seem to want it quiet.In terms of visual issues the relevant problems seem to be related to sudden transitions and lecture halls that are extremely dark. The only thing that can be done by conference organisers is to seek to have the lights in the lecture halls as bright as possible without preventing the viewing of the projector screen, this is usually done anyway.For social events the whole binge drinking at crowded bars thing doesn t work too well due to noise, poor lighting, crowds, and the smell of vomit. But there s no shortage of reasons to discourage binge drinking at conferences.FoodPsychology Today has a good article about preferences for food and picky eaters [6]. Lots of people on the Autism Spectrum have similar issues. The thing to do when arranging a conference meal is to have things neat and without needless combinations. Think about making food look more like what you would expect to see in a Japanese restaurant and less like Paella. Also having some very plain food on offer is a good thing, I think that bread-rolls for dinner and ice-cream for desert makes a viable meal. But any dish with a word like hash or mixed in it s name isn t a good option. Finally some combinations are really bad, I always find rare steak and potatoes on the same plate to be rather disgusting potato that s blood-stained doesn t appeal. The majority of conference meals satisfy these criteria.Autscape provides options of eating outside the main dining hall for people who can t tolerate the noise. For a Free Software conference it might be a better idea to provide seating outside the main area for people who are going to make noise. I don t mind people who want to get really drunk at the conference dinner, but I would prefer them to be in a separate room. Also when assisting a drunk friend to leave the venue before they cause more problems it would be convenient if there was a good place to take them to. The one occasion when I had to strongly encourage a friend to leave a conference dinner to reduce his embarrassment the next day (and reduce the annoyance for everyone else) there was no good place to take him so he just came back!ConclusionI don t expect that anyone will make any significant changes to conference plans after reading this post. I have tried to focus on things that would benefit NTs as well, but there are practical issues that get in the way such as the cost and availability of venues that are large enough. But there are some smaller ideas that can be implemented with relative ease.I hope that the people who are making claims about Autism and conference behavior will refer to this in future. If you think that there is a problem with the way people on the Spectrum act at conferences then the solution will more likely involve the suggestions I make here than anything else.

17 January 2011

Rapha&#235;l Hertzog: Review of my Debian related goals for 2010

Last year I shared my Debian related goals for 2010 . I announced that I would not be able to complete them all and indeed I have not, but I have still done more than what I expected. Let s have a look. Translate my Debian book into English and get it published: NOT DONE Or rather not yet done. It s still an important project of mine and I will do it this year. When I wrote this last year, I expected to find a publisher that would take care of everything but we failed to find a suitable one so we re going to do it ourselves. Cleanup the dpkg perl API and create libdpkg-perl: DONE libdpkg-perl has been introduced with dpkg 1.15.6. Create dpkg-buildflags: DONE dpkg-buildflags has been introduced with dpkg 1.15.7. It s not widely used yet and it won t really be until debhelper 7 supports it (see #544844). It would be nice to see progress on this front this year. Ensure the new source formats continue to gain acceptance: DONE The adoption rate has been steady, it clearly slowed down since the freeze though. I have implemented quite a few features to satisfy the needs of users, like the possibility to unapply the patches after the build, or the possibility to fail in case of unwanted upstream changes. Design a generic vcs-buildpackage infrastructure to be integrated in dpkg-dev: NOT DONE I believe it s something important on the long term but it never made it to my short-term TODO list and it s unlikely to change any time soon. Continue fixing dpkg bugs faster than they are reported: PARTLY DONE We have dealt with many bugs over the year, but we still have 20 more bugs than at the start of last year (370 vs 350). We re not doing bad compared to many other Debian teams but we can still benefit from some help. Start here if you re interested. Enhance our infrastructure to ease interaction between contributors and to have a better view of how each package is maintained: NOT DONE I am convinced that we need something to have a clearer idea of the commitments made by each contributor. I don t put the same amount of care in maintaining smarty-gettext that I do on dpkg. If we had a database of the stuff that we know we don t do well enough, it s easier to point new contributors towards those. Anyway, this project is still unlikely to come to the top of my priorities any time soon. Work on the developers-reference: NOT DONE We have switched the Maintainer field to debian-policy@lists.debian.org to have more review of the changes suggested through the bug tracking system but that has not changed much on the global situation. I still hope to become more active on it sometimes this year. Maybe by trying to make it more fun and creating the text for some of the wishlist bugs as blog articles first. Rewrite in C the last Perl scripts provided by the dpkg binary package: DONE Dpkg 1.15.8 was the first version working without Perl, I announced it in July. Integrate the 3-way merge tool for Debian changelogs in dpkg-dev: DONE dpkg-mergechangelogs is part of dpkg-dev since 1.15.7. I enjoy it regularly. Unfortunately it doesn t work well for cherry-picks. Would be nice to see this fixed, anyone up to the task? :-)
Click here to subscribe to my free newsletter and get my monthly analysis on what s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

One comment Liked this article? Click here. My blog is Flattr-enabled.

16 January 2011

Dirk Eddelbuettel: Plotting overbought / oversold regions in R

The good folks at Bespoke Investment Group frequently show charts of so-called overbought or oversold levels; see e.g. here for the most recent global markets snapshot. Classifying markets as overbought or oversold is a popular heuristic. It starts from computing a rolling smoothed estimate of the prices, usually via a (exponential or standard) moving average over a suitable number of days (where Bespoke uses 50 days, see here). This is typically coupled with a (simple) rolling standard deviation. Overbought and oversold regions are then constructed by taking the smoothed mean plus/minus one and two standard deviations. Doing this is in R is pretty easy thanks to the combination of R's rich base functions and its add-on packages from CRAN. Below is a simply function I wrote a couple of months ago---and I figured I might as well release. It relies on the powerful packages quantmod and TTR by my pals Jeff Ryan and Josh Ulrich, respectively.
## plotOBOS -- displaying overbough/oversold as eg in Bespoke's plots
##
## Copyright (C) 2010 - 2011  Dirk Eddelbuettel
##
## This is free software: you can redistribute it and/or modify it
## under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 2 of the License, or
## (at your option) any later version.
suppressMessages(library(quantmod))     # for getSymbols(), brings in xts too
suppressMessages(library(TTR))          # for various moving averages
plotOBOS <- function(symbol, n=50, type=c("sma", "ema", "zlema"), years=1, blue=TRUE)  
    today <- Sys.Date()
    X <- getSymbols(symbol, src="yahoo", from=format(today-365*years-2*n), auto.assign=FALSE)
    x <- X[,6]                          # use Adjusted
    type <- match.arg(type)
    xd <- switch(type,                  # compute xd as the central location via selected MA smoother
                 sma = SMA(x,n),
                 ema = EMA(x,n),
                 zlema = ZLEMA(x,n))
    xv <- runSD(x, n)                   # compute xv as the rolling volatility
    strt <- paste(format(today-365*years), "::", sep="")
    x  <- x[strt]                       # subset plotting range using xts' nice functionality
    xd <- xd[strt]
    xv <- xv[strt]
    xyd <- xy.coords(.index(xd),xd[,1]) # xy coordinates for direct plot commands
    xyv <- xy.coords(.index(xv),xv[,1])
    n <- length(xyd$x)
    xx <- xyd$x[c(1,1:n,n:1)]           # for polygon(): from first point to last and back
    if (blue)  
        blues5 <- c("#EFF3FF", "#BDD7E7", "#6BAED6", "#3182BD", "#08519C") # cf brewer.pal(5, "Blues")
        fairlylight <- rgb(189/255, 215/255, 231/255, alpha=0.625) # aka blues5[2]
        verylight <- rgb(239/255, 243/255, 255/255, alpha=0.625) # aka blues5[1]
        dark <- rgb(8/255, 81/255, 156/255, alpha=0.625) # aka blues5[5]
      else  
        fairlylight <- rgb(204/255, 204/255, 204/255, alpha=0.5)         # grays with alpha-blending at 50%
        verylight <- rgb(242/255, 242/255, 242/255, alpha=0.5)
        dark <- 'black'
     
    plot(x, ylim=range(range(xd+2*xv, xd-2*xv, na.rm=TRUE)), main=symbol, col=fairlylight) 		# basic xts plot
    polygon(x=xx, y=c(xyd$y[1]+xyv$y[1], xyd$y+2*xyv$y, rev(xyd$y+xyv$y)), border=NA, col=fairlylight) 	# upper
    polygon(x=xx, y=c(xyd$y[1]-1*xyv$y[1], xyd$y+1*xyv$y, rev(xyd$y-1*xyv$y)), border=NA, col=verylight)# center
    polygon(x=xx, y=c(xyd$y[1]-xyv$y[1], xyd$y-2*xyv$y, rev(xyd$y-xyv$y)), border=NA, col=fairlylight) 	# lower
    lines(xd, lwd=2, col=fairlylight)   # central smooted location
    lines(x, lwd=3, col=dark)           # actual price, thicker
    invisible(NULL)
 
After downloading data and computing the rolling smoothed mean and standard deviation, it really is just a matter of plotting (appropriate) filled polygons. Here I used colors from the neat RColorBrewer package with some alpha blending. Colors can be turned off via an option to the function; ranges, data length and type of smoother can also be picked. To call this in R, simply source the file and the call, say, plotOBOS("^GSPC", years=2) which creates a two-year plot of the SP500 as shown here: Example chart of overbought/oversold levels from plotOBOS() function This shows the market did indeed bounce off the oversold lows nicely on a few occassions in 2009 and 2010 --- but also continued to slide after hitting the condition. Nothing is foolproof, and certainly nothing as simple as this is, so buyer beware. But it may prove useful in conjunction with other tools. The code for the script is here and of course available under GPL 2 or later. I'd be happy to help incorporate it into some other finance package. Lastly, if you read this post this far, also consider our R / Finance conference coming at the end of April.

Edit: Corrected several typos with thanks to Josh.

Daniel Kahn Gillmor: resetting passphrases for mapped LUKS volumes

Let's say you set up a machine using an encrypted disk with LUKS (debian-installer's partman makes this wonderfully easy!). You choose an initial passphrase, get the machine working, and it's working great. Then, you need to restart it, and realize that (for whatever reason) you've forgotten or lost the passphrase for the volume. oops! (i'm sure this has never happened to you -- let's just pretend it's your less-fortunate friend). If your system is still running, and you have superuser access to it, you can actually set a new passphrase for the LUKS volume using information that the dm-crypt kernel module has about the in-use mapping. In my examples, i'll imagine that the source volume is /dev/XXX2 and the exported cleartext volume is known by the device-mapper as XXX2_crypt In the bigger picture, this should serve as a reminder that even though your disk is encrypted, if someone gets live access to the superuser account on a system with the encryption keys loaded, your data is no longer secret. Before you do any tweaking, you might want to back up your LUKS header, just in case:
0 root@example:~# umask 077
0 root@example:~# cryptsetup luksHeaderBackup /dev/XXX2 --header-backup-file XXX2.luksheader.backup
0 root@example:~# 
Maybe also copy that off the machine, since a copy of the LUKS header stored within its own volume isn't terribly useful for a backup-recovery situation. You might also be interested in looking at the contents of the LUKS header:
0 root@example:~# cryptsetup luksDump /dev/XXX2
LUKS header information for /dev/XXX2
Version:       	1
Cipher name:   	aes
Cipher mode:   	cbc-essiv:sha256
Hash spec:     	sha1
Payload offset:	2056
MK bits:       	256
MK digest:      93 51 6c 66 ec ce 32 54 6f 6b 52 d1 27 9b 5a 62 6f 6b 52 d1
MK salt:       	b2 ca 20 49 9f 78 49 c2 fe 15 b4 0f 74 11 23 49
               	64 9e 61 bb f2 82 60 47 a5 76 fa a4 24 0e 5a 7e
MK iterations: 	10
UUID:          	052f1da0-21a1-11e0-ac64-0800200c9a66
Key Slot 0: ENABLED
	Iterations:         	218733
	Salt:               	f2 ae 8c 53 48 a5 f0 bf e1 2c 06 5f 5a bd ff d9
	                      	9a 2e d1 49 3a 63 f8 49 82 ed ae 86 7b 7b 7e 76 
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
0 root@example:~# 
Now, the fix: We pull the live "master key" from the running device map, and fill a new luksKeySlot from it (this example uses bash's <() syntax for process substitution -- if you use a different shell, i'm sure you can find a different way to do it):
0 root@example:~# cryptsetup --master-key-file <( 
> dmsetup --showkeys table   awk '/^XXX2_crypt: /  print $6  '   tr -d '\n'   perl -e 'print pack("H*", <STDIN>);' 
> ) luksAddKey /dev/XXX2
Enter new passphrase for key slot: abc123
Verify passphrase: abc123
2 root@example:~# 
(note that the luksAddKey invocation above returned an error code of 2 even though it succeeded. I think this is a bug in cryptsetup's return code, not a bug in the password resetting -- it should have returned 0 instead of 2). You can check to see that a new key slot was enabled by re-running cryptsetup luksDump And if you really want to double-check before you reboot, you can try enabling a third keyslot using the passphrase you just added, since this would not succeed if your new passphrase failed to unlock any of the existing keyslots:
0 root@example:~# cryptsetup luksAddKey /dev/XXX2
Enter any passphrase: abc123
Enter new passphrase for key slot: anotherpassphrase
Verify passphrase: anotherpassphrase
0 root@example:~# 
You can also get rid of the original keyslot (which you don't know the passphrase to) like this:
0 root@example:~# cryptsetup luksKillSlot /dev/XXX2 0
Enter any remaining LUKS passphrase: abc123
0 root@example:~# 
(the above commands were all demonstrated using debian testing, with cryptsetup 2:1.1.3-4 and dmsetup 2:1.02.48-4) Tags: cryptsetup, dmsetup, luks, tip

Nikita Youshchenko: Avoiding apt-get update on n900 over gprs

While browsing n900 FAQ (in Russian) on maemoworld.ru site, I ve found some hints on avoiding automatic package list update over gprs. Problem is that gprs here in Russia is relatively expensive, and downloading several-megabytes package lists at unexpected moments costs money that users don t want to pay. Of course, Nokia ignores this problem. Hints linked above may be useful. But actually those don t disable updates over gprs, but disable updates completely, which is inconvenient. Much better is to still have automatic updates when on WiFi connection. Fortunately it is very easy to get that. Just create /etc/network/if-up.d/zz_block_apt_over_gprs with following content:
#!/bin/sh
# If on GPRS, set fake apt proxy to stop automatic update from working.
# Otherwise, remove fake apt proxy.
if [ "GPRS" = "$ICD_CONNECTION_TYPE" ]; then
  echo 'Acquire::http::Proxy "http://127.0.0.1:8768";' > /etc/apt/apt.conf.d/zz-fake-proxy
else
  rm -f /etc/apt/apt.conf.d/zz-fake-proxy
fi
exit 0
Well, this actually also disables package installation while on gprs. Bit IMO this is not usually needed, and if you happen to need to install something over gprs, you may just remove /etc/apt/apt.conf.d/zz-fake-proxy

Russell Coker: Links January 2011

Halla Tomasdottir gave an interesting TED talk about her financial services company which uses feminine values and thus survived the Icelandic financial crash [1]. She also has a lot of other interesting ideas about how to run a corporation or a country.Hillary Clinton gave an interesting TED talk about national security and women s rights [2]. The US State Department is now making women s rights a priority due to the fact that places where women are mistreated tend to pose a security threat to the US. Unfortunately this talk can t be viewed without flash.Richard Dawkins reads some of his hate-mail while sitting by the fire [3], it s interesting to know that a loving God wants his minions to send hate-mail to scientists. There are a few cheap laughs here.Emily Pilloton gave an interesting TED talk about fixing some of the problems with the education system in rural USA by incorporating design [4]. Instead of having useless projects in shop class the students will complete real projects for their community which have to be designed, planned, and made. The students who end up going to university will learn some generally useful skills through this and the ones who don t go to university will probably have an immediate opportunity to apply the skills that they learn.Elizabeth Pisani gave an interesting TED talk about rational behavior and HIV [5]. She explains why certain behavior that carries great risk can be rational for the people who do it. One of her interesting points is the fact that the poor health system in the US makes it economically viable for the government to refrain from providing free needles to junkies as they don t pay to treat them when they get sick. She makes a good case for political change.Tony Porter gave an inspiring TED talk titled A Call to Men about the way the macho culture socialises men towards violence against women [6].Fraud Magazine has an interesting Interview with Dr. Robert D. Hare and Dr. Paul Babiak about psychopathic fraudsters [7]. Most of the interview is generally about psychopaths and the cases of psychopaths in business, there isn t much that is specific to financial fraud. It has a good summary of what some psychopaths are like.Barbara Fister wrote a Liberation Bibliography a call for knowledge to be free [8]. Currently universities pay researchers who then pay for corporations to accept copyright for their work and then resell it back to the universities. This is sucking money from the university system and denying access to knowledge to the people who can t pay.Charles Limb gave an interesting TED talk about using an MRI to study the brains of Jazz musicians doing Improv and Rappers doing Freestyle [9].Deborah Rhodes gave an interesting TED talk about her new method of screening for breast cancer for women with dense breast tissue [10].Cory Doctorow wrote an interesting article about Net Neutrality and it s importance in helping the less famous writers (the majority of writers) earn money [11]. He also makes some interesting points about Google and about the corporate welfare given to telephone companies.Beyond Zero Emissions has an interesting summary of the current situation regarding Carbon Capture and Sequestration (CCS), among other things the only system that s currently operational is leaking CO2 and killing plants and animals [12].The Brisbane council knew that the flood risks were greater than suggested by published information, they kept this news to themselves to assist property developers in selling low-lying land, and this led to increased problems now that the predicted floods have occurred [13]. Sounds like corruption, we could probably do with a Royal Commission into this.Psychology Today has an interesting article titled Wise Words From Fellow Introverts which is a collection of blog comments from introverts [14].

15 January 2011

Mike Hommey: Replacing the Iceweasel button with an icon

Recent Iceweasel betas allows to replace the menu bar with a Iceweasel button. This is not enabled by default, but right-clicking on the menu bar allows to disable the menu bar, which enables the Iceweasel button.
The button is not exactly very appealing, and takes quite a lot of horizontal space on the tab bar. But with a few lines of CSS, this can fortunately be changed. Edit the chrome/userChrome.css file under your user profile, and add the following lines:
#appmenu-toolbar-button  
  list-style-image: url("chrome://branding/content/icon16.png");
 
#appmenu-toolbar-button > .toolbarbutton-text,
#appmenu-toolbar-button > .toolbarbutton-menu-dropmarker  
  display: none !important;
 
This what Iceweasel looks like, then:

14 January 2011

Jo Shields: The phantom fifth freedom

Not for the first time, I ve seen the suggestion in the echo chamber that Mono packages should be moved from Debian into the non-free repository, which is not formally part of Debian. The reason, as it so often is, is patents specifically this time, the searing risk posed to Debian and its users that Mono s packaging does not (and technically could not without forking from upstream) provide base class libraries which implement only the content of ECMA-335 4th Edition. As I understand it, this implies three things about the suggestion/demand: firstly, that the individual in question believes that Mono end users are at risk from patent litigation from Microsoft Corp because Mono s implementation of Microsoft.NET beyond the content of ECMA 334/335 infringes on Microsoft patents; secondly, that the Microsoft Community Promise which promises not to assert legal claims over third party implementations of ECMA-335 4th Edition (and ECMA-334 4th Edition which defines the C# language) would render a pure ECMA-only runtime safe if it existed (which it does not); thirdly that without the protection offered by the Microsoft Community Promise, the source code licenses of Mono are irrelevant the patent risk renders the software non-Free. It appears, unfortunately, that the community of Free Software Advocates don t actually understand what Free Software actually IS. That explains an awful lot, but should surprise nobody. So here s a lesson on what, exactly, is being advocated for. The Free Software Foundation defines four Software Freedoms these are the minimum criteria required for something to be considered Free Software by the FSF: Other groups have their own variants on these, but those are really just clarifications on the FSF definition - for example, the Debian Free Software Guidelines mostly line up, but have some additional clauses such as clause 4 which allows software to be considered Free if source code may be redistributed without modifications, as long as patches may be shipped alongside. These four freedoms are offered to you by the software s copyright holders only, and apply regardless of their choice of license any Free license, from a lengthy legal tome like the MPL to the completely-Free WTFPL, will offer you these four basic freedoms as a minimum, and any additional clauses in their licenses cannot seek to restrict these freedoms. These four freedoms represent the beginning, and end, of whether a piece of software is Free or not. Software does not need to be developed in the open, in a community-responsive way, in order to qualify as Free projects such as Google s Android, which are developed under a throw a final release over the wall, bugs and all, and expect people to thank you for it model, are still free, even if contribution is difficult. Actually, on a related note, software does not need to solicit upstream contribution of any kind in order to qualify as Free as long as you personally can redistribute the code with changes, then that s enough. Software does not need to serve a fully or even partially legal purpose in order to qualify as Free the favoured tool for causing distributed denial of service attacks, Low Orbit Ion Cannon, is Free Software, even though realistically it serves no legal purpose. DeCSS, the code initially used to allow DVD media to play on Linux (by breaking the CSS encryption mechanism) is Free Software. Software does not need to be useful or tasteful in order to qualify as Free the Last Measure Operating System, a minimalist OS primarily designed to loudly display the famous shock site images from goatse and related, is Free Software. Even in somewhat less clear-cut cases of taste, your personal opinion of software has no bearing on whether it is Free Software, as long as the four freedoms are guaranteed by the author(s). Software does not need to have only Free dependencies to qualify as Free Software it is entirely permissible to write software which relies upon a non-Free framework or library, and release your code under a Free license. It is the downstream recipient s problem to provide the dependencies, including their choice to craft a Free replacement for any non-Free code you make use of. Debian has a special repository called contrib, where Free software which only works with non-Free data, lives for example, Free game engines which require the insertion of proprietary game data in order to operate. You could write Free addons for expensive proprietary software such as Matlab, and as long as your code is Free, your responsibilities are met. Software does not need to avoid patents software, algorithmic, or otherwise in order to qualify as Free. The Freetype font library was still entirely Free Software when Apple were slinging threats around regarding font hinting data. FFmpeg, the Swiss Army knife of media codec libraries, is Free Software regardless of the number of codec patents it breaks. Software does not need any third party approval to be Free Software the rights of Free Software can only be offered by the copyright holders, and the opinion of third parties is an irrelevance as to whether software is Free. The GPL d clone of Blizzard Entertainment s Battle.net servers, bnetd, is Free Software, regardless of legal takedown notices. Third parties cannot influence whether or not a piece of software is Free. They can influence tangentially related topics, such as whether the software can be legally used, but that s the limit. And even within a given piece of software, where copyright is shared by contributors, the author of one component has no say on other components. And you can t make code which is already released as Free, suddenly un-Free you can, if you hold all the copyrights, close up future versions, but your existing code remains Free forever. Reasonably, you can opt to avoid using a piece of software because you have requirements beyond it merely being Free Software Cdrtools has been avoided in Debian for a long time due to the upstream author s legal threats and rambling but that is a side issue as to the question of whether or not the software is Free. Patents are simply not involved in the question of whether or not something is Free Software, except for one narrow case: where Free Software is released by somebody who also holds related patents, and uses a license such as Ms-PL or Apache 2.0 or GPLv3 which requires them to also release those patents to those using/distributing the software. Outside that narrow situation, patents do not relate to the question of whether something is Free Software even if a company releases some source code under a license like BSD then demands patent fees from end users. So, on to the original topic of Mono. Every piece of Mono s source code is released by its authors under a license which guarantees the FSF s Four Freedoms. Whether or not you find Mono useful or tasteful does not affect that Free status. Whether or not Mono infringes upon any laws or patents does not affect that Free status. That Mono contains some libraries whose upstream author is Microsoft does not give Microsoft even the remotest claim to a single line of code outside the code they wrote and even then, it wouldn t be an issue, since the licenses they use are Free. In fact, both the licenses used in the Microsoft portions of the source base make patent grants to all users, in addition to guaranteeing the FSF s Four Freedoms and any license contamination would decrease, not increase, any risk of legal attack from Microsoft. There s even plenty of Microsoft code available for re-use at a lower level than the currently re-used libraries: The Microsoft.NET Micro Framework (for use on embedded platforms direct to the metal) is under the Free Software Apache 2.0 license, and would provide access to some of Microsoft s runtime and class library code, complete with a patent grant, if it were desired. Please try to keep your thought processes straight. If you want to argue that you re all for Free Software, please remember that there s plenty of Free Software you might not approve of and don t claim to be a Free Software advocate then use bogus arguments to claim that Free Software is not Free. Free Software includes LMOS and LOIC and Mono, whether you like it or not.

12 January 2011

Mike Hommey: Attempting to track I/O with systemtap

There are several ways a program can hit the disk, and it can be hard to know exactly what s going on, especially when you want to take into account the kernel caches. This includes, but is not limited to: There are various ways to track system calls (e.g. strace) allowing to track what a program is doing with files and directories, but that doesn t tell you if you re actually hitting the disk or not. There are also various ways to track block I/O (e.g. blktrace) allowing to track actual I/O on the block devices, but it is then hard to back-track what part of which files or directories these I/O relate to. To the best of my knowledge, there are unfortunately no tools to do such tracking easily. Systemtap, however, allows to access the kernel s internals and to gather almost any kind of information from any place in a running kernel. The downside is that it means you need to know how the kernel works internally to gather the data you need ; that will limit the focus of this post. I had been playing, in the past, with Taras script, which he used a while ago to track I/O during startup. Unfortunately, it became clear something was missing in the picture, so I had to investigate what s going on in the kernel. Setting up systemtap On Debian systems, you need to install the following packages: That should be enough to pull all the required dependencies. You may want to add yourself to the stapdev and stapusr groups, if you don t want to run systemtap as root. If, like in my case, you don t have enough space left for all the files in /usr/lib/debug, you can trick dpkg into not unpacking files you don t need:
# echo path-exclude /usr/lib/debug/lib/modules/*/kernel/drivers/* > /etc/dpkg/dpkg.cfg.d/kernel-dbg
The file in /etc/dpkg/dpkg.cfg.d can obviously be named as you like, and you can adjust the path-exclude pattern to what you (don t) want. In the above case, kernel drivers debugging symbols will be ignored. Please note that this feature requires dpkg version 1.15.8 or greater. Small digression One of the first problems I had with Taras script is that systemtap would complain that it doesn t know the kernel.function("ext4_get_block") probe. It is due to a very unfortunate misfeature of systemtap, where the kernel.* probes refer to whatever is in the vmlinux image. Modules probes have a separate namespace, namely module("name").*. So for the ext4_get_block() function, this means you need to set a probe for either kernel.function("ext4_get_block") or module("ext4").function("ext4_get_block"), depending how your kernel was compiled. And you can t even use both in your script, because systemtap will complain about either being an unknown probe Tracking the right thing I was very recently pointed to a Red Hat document containing 4 useful systemtap scripts ported from dtrace, which gives me a good occasion to explain the issue at hand with the first of these scripts. This script attempts to track I/O by following read() and write() system calls. Which is not tracking I/O, it is merely tracking some system calls (Taras script had the same kind of problem with read()/write() induced I/O). You could just do the very same with existing tools like strace, and that wouldn t even require some system privileges. To demonstrate the script doesn t actually track the use of storage devices as the document claims, consider the following source code:
#include <fcntl.h>
#include <sys/stat.h>
#include <unistd.h>
/* Ensure the resulting binary is decently sized */
const char dummy[1024 * 1024] = "a";
int main(int argc, char *argv[])  
  struct stat st;
  char buf[65536];
  int i, fd = open(argv[0], O_RDONLY);
  for (i = 0; i < 100000; i++)  
    read(fd, buf, 65536);
    lseek(fd, 0, SEEK_SET);
    read(fd, buf, 65536);
    lseek(fd, 512 * 1024, SEEK_SET);
   
  close(fd);
  return 0;
 
All it does is reading some parts of the executable a lot of times (notice the trick to make the executable size at least 1MB). Not a lot of programs will actually do something as bold as reading the same data again and again (though we could probably be surprised), but this easily points the problem. Here is what the output of the systemtap script looks like for this program (stripping other unrelevant processes):
         process     read   KB tot    write   KB tot
            test   400002 25600001        0        0
Now, do you really believe 25MB have actually been read from the disk? (Note that the read() count seems odd as well, as there should only be around 200000 calls) Read-ahead, page cache and mmap() What the kernel actually does is that a read() on a file is going to check the page cache first. If there is nothing corresponding to the read() request in the page cache, then it goes down to the disk, and fills page cache. But as loading only a few bytes or kilobytes from the disk could be wasteful, the kernel also reads a few more blocks ahead, apparently with some heuristic. But read() and write() aren t the sole way a program may hit the disk. On UNIX systems, a file can be mapped in memory with the mmap() system call, and all accesses in the memory range corresponding to this mapping will be reflected on the file. There are exceptions, depending on how the mapping is established, but let s keep it simple. There is a lot of litterature on the subject if you want to document yourself on mmap(). The way the kernel will read from the file, however, is quite similar to that of read(), and uses page cache and read ahead. The systemtap script debunked above doesn t track these. I ll skip write accesses, because for now, they haven t been in my scope. Tracking some I/O with systemtap What I ve been trying to track so far has been limited to disk reads, which happen to be the only accesses occurring on shared library files. Programs and shared libraries are first being read() from, so that the dynamic linker gets the ELF headers and knows what to load, and then are mmap()ed following PT_LOAD entries in these headers. As far as my investigation in the Linux kernel code goes, fortunately, both accesses, before they end up actually hitting the disk, go through the __do_page_cache_readahead kernel function (this function was tracked in Taras script). Unfortunately, while it is called with an offset and a number of pages to read for a given file, it turns out the last pages in that number are not necessarily being read from the disk. I don t know for sure, because the latter had an effect on my observations, but some might even already be in the page cache. Going further down, we reach the VFS layer, which ends up being file-system specific. But fortunately, a bunch of (common) file-systems actually share page mapping code, and commonly use the mpage_readpage and mpage_readpages functions, both calling do_mpage_readpage to do the actual work. And this function seems to be properly called only once for each page not in the page cache already. If my reading of the kernel source is right, this however is not really where it ends, and do_mpage_readpage doesn t actually hit the disk. It seems to only gather some information (basically, a mapping between storage blocks and memory) that is going to be submitted to the block I/O layer, which itself may do some fancy stuff with it, such as reordering the I/Os depending on other requests it got from other processes, the position of the disk heads, etc. And when I say do_mpage_readpage doesn t actually hit the disk, I m again simplifying the issue, because it actually might, as there might be a need to read some metadata from the disk to know where some blocks are located. But tracking metadata reads is much harder, and I haven t investigated it. Anyways, skipping metadata, going further down after do_mpage_readpage is hard because it s difficult to back-track which block I/O is related to which read-ahead, corresponding to what read at what position in which file. do_mpage_readpage already has part of this problem because it is not called with any reference to the corresponding file. But __do_page_cache_readahead is. So knowing all the above, here is my script, the one I used to get the most recent Firefox startup data you can find in my last posts:
global targetpid;
global file_path;
probe begin  
  targetpid = target();
 
probe kernel.function("__do_page_cache_readahead")  
  if (targetpid == pid())
    file_path[tid()] = d_path(&$filp->f_path);
 
probe kernel.function("do_mpage_readpage")  
  if (targetpid == pid() && (tid() in file_path))  
    now = gettimeofday_us();
    printf("%d %s %d\n", now, file_path[tid()], $page->index*4096);
   
 
probe kernel.function("__do_page_cache_readahead").return  
  if (targetpid == pid())
    delete file_path[tid()];
 
This script needs to be used with a command given to systemtap, with the -c option, such as in the following command line:
# stap readpage.stp -c firefox
Each line of output represents a page (i.e. 4,096 bytes) being read, and contains a timestamp, the name of the file being read, and the offset in the file. As discussed above, do_mpage_readpage is not really the place the I/O actually occurs, so the timestamps are not entirely accurate, and the actual read order from disk might be slightly different, but it still is a quite reliable view in that the result should be reproducible with the same files even when they re not located on the same blocks on disk, and provided their page cache status is the same when starting the program. This systemtap script ignores writes, as well as metadata accesses (including but not limited to inodes, dentries, bitmap blocks and indirect blocks). It also doesn t account accesses to files opened with the O_DIRECT flag or similar constructs (raw devices, etc.) Read-ahead in action Back to the small example program, my systemtap script records 102 page accesses, that is, 417,792 bytes, much less than the actual binary size on my system (1,055,747 bytes). We are still far from the 25MB figure from the other systemtap script. But we are also far from the 128KiB the program actively reads (64KiB twice, leaving a hole between the two blocks). At this point, it is important to note that the ELF headers and program code all fit within a single page (4 KiB), and following the 1MiB section corresponding to the dummy variable, there are only 5,219 bytes of other ELF sections, including the .dynamic section. So even counting everything the dynamic linker needs to read, and the program code itself, we re still far from what my systemtap script records. Grouping consecutive blocks with consecutive timestamps, here is what can be observed:
Offset Length
0 16,384
983,040 73,728
16,384 262,144
524,288 65,536
(By now, you should have guessed why I wanted that big hole between the read()s ; if you want to reproduce at home, I suggest you also use my page cache flush helper) As earlier investigations showed, the first accesses by the dynamic loader are to read the ELF headers and .dynamic section. As mentioned above, these are really small. Yet, the kernel actually reads much more: 16KiB at the beginning of the file when reading the ELF headers, and 72KiB at the end when reading the .dynamic section. Subsequent accesses from the dynamic loader are obviously already covered by these accesses. Next accesses are those due to the program itself. The program actively reads 64KiB at the beginning of the file, then 64KiB starting at offset 524,288. For the first read, the kernel already had 16KiB in the page cache, so it didn t read them again, but instead of reading the remainder, it reads 256KiB. For the second read, however, it only reads the requested 64KiB. As you can see, this is far from being something like you wanted n KiB, I ll read that fixed amount now . Further testing with different read patterns (e.g. changing the hole size, read size, or reading from the dummy variable directly instead of read()ing the binary) is left as an exercise to the reader.

Russell Coker: icmptx Tunneling IP over ICMP Echo

I ve just been playing with icmptx, a system for tunneling IP over ICMP Echo which could be handy if I ever find myself blocked by firewalls. Unfortunately the documentation is lacking. Below is a sample configuration that works for me, all you have to do is to put the correct IP address in for SERVERIP in both scripts and it should work. I m not sure what the ideal value for the MTU is, 65535 is the largest possible. For transmission it usually won t make any difference as the occasions when I need such things will usually be download-only sessions and the ACK packets will be quite small. For receiving data the server has an MTU of 1500 on the Ethernet port so nothing bigger than that will come in. Presumably when downloading data the packets will be transmitted in two ICMP fragments.One interesting feature of the program is that it doesn t match requests and replies. I presume this is because any firewall that only allows one reply per echo request will probably ensure that the reply contents match the request contents, so they just assume that a firewall will let all ICMP echo/reply traffic through. The upside of this is that it should give lower round trip times than any tunneling system that polls for return data.I ve filed some Debian bug reports about it, bug #609413 is a request for it to set icmp_echo_ignore_all when it s running and also emulate the regular PING functionality. Bug #609412 is a request for it to assign the IP address to the tun0 interface. Bug #609414 is a request for the server side of it to call daemon(0,0).I won t leave this running. Having to run a virtual server with the regular ICMP functionality disabled is too much effort for the small benefit that using ICMP tunneling may offer over DNS tunneling.My configuration scripts (with the IP address removed) are below.ConfigurationServer#!/bin/sh
set -e
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all
icmptx -s SERVERIP &
sleep 0.5
ifconfig tun0 mtu 65535 10.10.10.1 netmask 255.255.255.0Client#!/bin/sh
set -e
killall icmptx true
icmptx -c SERVERIP &
sleep 0.5
ifconfig tun0 mtu 65535 10.10.10.2 netmask 255.255.255.0
wait

10 January 2011

Christian Perrier: [life] How to discourage the use of public transport

As some of my readers may now, I'm a regular user of public transport. Even though I'm living in an area (far suburbs of Paris) where the use of cars for nearly everything is common, I managed up to now to not use my car for my daily commute. I was using the public bus line of our town, that starts close from our house and ends up at the local train station where I can commute to U line of Paris transport system (then C line, then another bus ride). You can compare this complicated 1h30 travel to the 35 minutes drive I would have if I was using my car. I *was*. I'll stop from tomorrow. The reason is simple: my U train start hour is 6h57. The bus formerly started at 6h36, then reached the train station at 6h51, which was a quite good commute. That made me leave our house at about 6h25, considering a 10min walk from the house to the bus stop. That's 32 minutes for....3 kilometers. Already needs some motivation, right? This changed today (with minimal notice). To catch the same train, I should have a bus at 6h31, thus leave 5 minutes earlier, so *nearly 40 minutes before the train schedule*. Considering that driving from our place to the train station is about 12 minutes, the deal is now made. I'll stop being the only non captive person using the bus in our neighbourhood (all others are people who can't use a car). So much for environment, etc. This is an 'interesting' way to understand why it is so easy to discourage proper use of public transport and minimize the impact on environment. A small change can lead to such effects. Of course, as you may expect, I won't stop there. Letters to the (public-funded) bus company and people in charge of public transport in our city (the bus line deficit is funded by the city), will follow very soon. Tehre are indeed many other reasons why people don't use our local buses that much. One being that, even though there are 2 trains every 15 minutes leaving the station (U and N lines), both alternated with a 7.5 minutes interval....there is only one bus every 15 minutes....on each bus line. Sometimes, even, you don't know why, there's a 23 minutes gap. Imagine waiting for your bus for more than 20 minutes at night in winter...when you're about 2-3 kilometers away from your home... Sure, that would need a lot of funding to have a really convenient system. But how about spending the billions of euros spent in super-highways, toll tunnels, etc., in public transport infrastructure? Funding my local bus line to have 2 buses every 15 minutes at peak hours would cost about the price of 100m of a 3-lane highway, for one year. Make the deal. (All that will indeed reinforce my backup system : run from my house to the train station, for 3km and 15 minutes...which involves another run at the end of my train ride, for 4km and 20 minutes. That, however, involves some "logistics" organization to be sure that I don't spend the day working in my stinking running suit. That also means up to 56km run every week, which I can't always do, particularly if I plan a race for the next week-end!) Another side effect: I will have less time for Debian as my bus commutes were used for hacking, while it will be much more difficult when driving my car to the train station, isn't it?

9 January 2011

Nikita Youshchenko: checkrestart

After aptitude or similar upgrade tool completes its job, new versions of files are installed into filesystem. However, some running processes may be still using old files (and actually Linux file systems continue to store invisible copies of old files while those are mapped by running processes). In case of a security update, having old versions running keeps system vulnerable even after update completes, which is not good. A reboot is possible, but that s an ugly solution. Fortunately tools exist to find processes that have deleted files mapped. One of these tools is checkrestart (from debian-goodies package). Just run checkrestart -p after upgrade, and you will see what to restart to be safe and even what init.d scripts to run with restart argument. P.S.
On a server with hundreds of running processes, checkrestart -p is terribly slow, likely because it makes separate queries to dpkg about files mapped by each process. Perhaps it could be seriously accelerated if first all names to query are gathered, and then each name is queried only once. Any volunteers to do that? :)

8 January 2011

Nikita Youshchenko: Psi / Psi+: want to avoid auto-online after auto-offline

Older versions of Psi jabber client required manual changing status to online to leave auto-offline state. It was very handy for me: I connect to my jabber account from two locations (from home and from office), so after I left one of these locations, auto-offline soon happened, and Psi at that location did not return online until I returned and manually instructed it to re-login. Unfortunately newer versions of Psi (I guess, starting from commit 4a54500c on git://git.psi-im.org/psi.git, which was between 0.11 and 0.12 releases), and also Psi+, behave differently: if user activity is detected on the desktop, auto-offline state is left, and status becomes online again similar to leaving auto-away and auto-xa states. This change is not good for me, at least in two situations: In both these situations, Psi client not at my current location becomes online, which can cause routing incoming messages to wrong location. I don t state that new behaviour is somehow worse than old behaviour. But for my use cases, I want to get old behaviour back. I ve done some code reading. I think that point to fix is near the bottom of PsiAccount::secondsIdle() method, where any auto status is disabled if no timeout condition currently exists. This should be changed so that auto-offline state won t be left if some config key is set. I will try to implement this change soon (or at least I hope I will try ) P.S.
To make things even better, a check of some external condition could be performed to decide if client should return online from auto-offline or not. Then fancy things could be set up, e.g. return to online at office only if my mobile phone is currently registered in the office wifi network not really required but cool.

7 January 2011

Rapha&#235;l Hertzog: How to create custom RSS feeds with WordPress

WordPress has many alternate built-in feeds: per category, per tag, per author, per search-keyword. But in some cases, you want feeds built with some more advanced logic. Let s look at the available options. WordPress advanced built-in feeds You can create feeds for unions or intersections of tags, you just have to use a URL like /tag/foo,bar/feed/ (all articles tagged with foo or bar) or /tag/foo+bar/feed/ (all articles tagged with foo and bar). You can also have feeds excluding a category, although that requires you to know the category identifier (and hardcode it in the URL like this: /?feed=rss2&cat=-123 where 123 is the category id that you want to exclude). But there s no simple way to have a feed that excludes articles with a given tag. The best solution found involves creating a custom feed. I ll show you a variation of this below. Creating a custom feed
  1. First of, install the Feed Wrangler plugin, it will take care of registering our custom feeds with wordpress.
  2. Go to Settings > Feed Wrangler in your WordPress administrative interface and create a new feed, let s call it myfeed .
  3. You should now create a feed-myfeed.php file and put it in your current theme s directory. The initial content of that file should be this:
    <?php
    include('wp-includes/feed-rss2.php');
    ?>
  4. At this point, you already have a new feed that you can access at /feed/myfeed/ (or /?feed=myfeed). It s a complete feed like the main one.
Now, we re going to look at ways to customize this feed. We re going to do this by changing/overriding the default query that feed-rss2.php s loop will use. A feed excluding articles with a tag If you want to create a feed that excludes the tag foo , you could use this:
<?php
global $wp_query;
$tag = get_term_by("slug", "foo", "post_tag");
$args = array_merge(
        $wp_query->query,
        array('tag__not_in' => array($tag->term_id))
);
query_posts($args);
include('wp-includes/feed-rss2.php');
?>
That was relatively easy, thanks to the tag__not_in query parameter. Now you can further customize the feed by adding supplementary query parameters to the $args array. The documentation of query_posts details the various parameters that you can use. A feed excluding articles with a custom field (meta-data) I went further because I did not want to use a tag to exclude some posts: that tag would have been public even if it was only meaningful to me. So I decided to use a custom field to mark the posts to exclude from my new feed. I named the field no_syndication and I always give it the value 1 . This time it s not so easy because we have no query parameter that can be used to exclude posts based on custom fields. We re going to use the post__not_in parameter that can be used to exclude a list of posts. But we must first generate the list of posts that we want to exclude. Here we go:
<?php
global $wp_query;
$excluded = array();
$args_excluded = array(
    'numberposts'     => -1,
    'meta_key'        => 'no_syndication',
    'meta_value'      => 1,
    'post_type'       => 'post',
    'post_status'     => 'published'
);
foreach (get_posts($args_excluded) as $item)  
        $excluded[] = $item->ID;
 
$args = array_merge(
        $wp_query->query,
        array('post__not_in' => $excluded)
);
query_posts($args);
include('wp-includes/feed-rss2.php');
?>
A feed with modified content You might want to add a footer to the articles that are syndicated. I use the Ozh Better Feed plugin for this but it applies to all your feeds. You could do that sort of transformation only in your customized feed by using the WordPress filter named the_content_feed. Here s a simple example:
<?php
function myfeed_add_footer($content)  
        return $content . "<hr/>My footer here";
 
add_filter('the_content_feed', 'myfeed_add_footer');
include('wp-includes/feed-rss2.php');
?>
I ll stop here but obviously you have lots of options and many ways to tweak all the snippets above. They have been tested with WordPress 3.0.4. Note that in all those examples, I took care to not duplicate the code from feed-rss2.php, instead I used include() to execute it. That way my custom feeds will automatically benefit from all the future enhancements and fixes made by the WordPress developers. But if you have to modify the XML structure of your custom feeds, you can paste the content of feed-rss2.php in your file and change it like you want

One comment Liked this article? Click here. My blog is Flattr-enabled.

Simon Josefsson: On Password Hashing and RFC 6070

The RFC Editor has announced a new document, RFC 6070, with test vectors for PKCS5 PBKDF2. The document grow out of my implementation of SCRAM for GNU SASL. During interop testing, more than one other implementation turned out to have mistakes in the PBKDF2 implementation. It didn t help that there weren t any stable test vectors for PBKDF2, so that we could do black-box testing of our PBKDF2 implementations against well-known and stable test vectors. Debugging this was time consuming. The document addresses this problem. So what is PBKDF2?

Briefly, the PBKDF2 is the state of the art way to convert a password string into a cryptographic binary key. There are many ways to design such an algorithm. To understand why PBKDF2 is designed the way it is we need some basic understanding. The typical attack you would be concerned with is if someone compromises your authentication servers and steals the credential database. This is what happened with the recent Gawker attack, where passwords were apparently stored in clear text. The first security improvement compared to storing passwords in clear text is to only store a hashed or encrypted form of the password. Hashing is preferable over encryption in this context, because with encryption it is possible to go back to the unencrypted form (if you know the key) whereas with a hash it is computationally infeasible to do so. The downside of hashing is that it is impossible to be compatible with all kind of authentication systems: the hash of the password in the database needs to be computed using the same algorithm as the system will be using. Unfortunately there are many different schemes out there to do password hashing, including the old Unix crypt(3) algorithm and the Windows LM-hash. You normally never know in advance what kind of systems your hashed credential database needs to work for, so you have to pick some set of algorithms and hope for the best. The next security improvements are iterations and salting. Iterations means to run the hash function not once, but hundreds or even thousands of times, on the input data (e.g., password). The extra time is hardly noticeable for the user, but for an attacker who is testing an entire dictionary of common passwords against your hash, making the work cost thousand times higher quickly makes the attack vector computationally infeasible. Salting means to include something other than the password in the input to the hash function, typically a random string. Without salting, a pre-computed rainbow table would provide a dictionary attack against all hashed password databases (one rainbow table per hashing algorithm). By using a sufficiently long and random salt, a rainbow table for one system is not applicable to another system. Since generating a rainbow table is time consuming, this adds another barrier to the attacker. PBKDF2 combines these three concepts hashing, iteration and salting into one construct, and it is designed flexible enough so that it can work on any pseudo-random function. In practice, I ve only ever seen HMAC-SHA1 be used. Some final words on the publication behind RFC 6070: Out of my earlier publications, this was the document that (by a large margin) spent the least time in the IETF process. You see various milestones of the document in the RFC 6070 datatracker history page. The time in the publication process was from 2010-08-09 to 2011-01-06. One month was in the AUTH48 state, of which the initial 20 days were waiting for me to respond. Thank you to my sponsoring AD, Sean Turner, for making this possible!

Paul Wise: Another year, another log entry

It has been almost a full year since my last log entry. It has been a busy work year, I attended some nice conferences and did minimal FLOSS stuff. On the work side of things I was a third of an Australian VoIP startup that came and went. I setup Debian servers, installed OpenSIPS and associated software, wrote OpenSIPS scripts, wrote peripheral software and did customer support. We had a good thing going there for a while, some fans on the Whirlpool forums but in the end there wasn't enough money for the requisite marketing and local market circumstances were squeezing Australian VoIP providers anyway. On the conference side of things I went to LCA 2010, the Thai Mini-DebCamp 2010, DebConf10 and FOSSASIA 2010. Had a great time at all of them. At LCA 2010 in windy Wellington, New Zealand the distributions summit organised by Martin Krafft was one of the highlights. It was dominated by Debian/Ubuntu talks but there were some other interesting ones, especially the one on GoboLinux's integration of domain-specific package managers. Also excellent were the keynotes given by Gabriella Coleman (Best & worst of times), Mako Hill (Antifeatures) and others, which I felt gave LCA an improved and very welcome focus on software freedom. There were quite a few Debian folks at LCA, it was great to hang out with them during the week and afterwards. Monopedal sumo with mako and others was hilarious fun. At the Thailand Mini-DebCamp 2010 in Khon Kaen, I was glad to see Andrew Lee (Taiwan) and Christian Perrier (France) again and meet Yukiharu YABUKI (Japan) and Daiki Ueno (Japan). In addition to the five international folks, there were quite a few locals, including Thailand's currently sole Debian member, Theppitak Karoonboonyanan. The event was hosted at Khon Kaen University and opened with my talk about the Debian Social Contract and the Debian Free Software Guidelines. This was followed by a number of talks about Debian package building, a 3-day BSP where we touched 57 bugs, a great day of sightseeing and talks about i18n, derivative distros, keysigning, mirrors, contribution and a discussion about DebConf. During the week there was also the usual beersigning, combined with eating of unfamiliar and "interesting" Thai snacks. After the conference Andrew and I roamed some markets in Bangkok and got Thai massages. Beforehand I also visited a friend from my travels on the RV Heraclitus in Chiang Mai, once again experiencing the awesomeness of trains in Asia, unfortunately during the dry season this time. I took a lot of photos during my time in Thailand and ate a lot of great and spicy food. As a vegetarian I especially appreciated the organiser's efforts to accommodate this during the conference. At DebConf10 in New York City, by far the highlight was Eben Moglen's vision of the FreedomBox. Negotiating the hot rickety subways was fun, the party at the NYC Resistor space was most excellent, Coney Island was hot and the water a bit yuck, zack threw a ball, the food and campus was really nice. Really enjoyed the lintian BoF, ARM discussions, shy folks, GPLv3 question time, paulproteus' comments & insights, wiki BoF, puppet BoF, derivatives BoF, Sita, astronomy rooftop, cheese, virt BoF, Libravatar, DebConf11, Brave new Multimedia World, bagels for breakfast, CUT, OpenStreetMap & lightning talks. Having my power supply die was not fun at all. Afterwards I hung out with a couple of the exhausted organisers, ate awesome vegan food and fell asleep watching a movie about dreams. One weird thing about DebConf10 was that relatively few folks used the DebConf gallery to host their photos, months later only myself and Aigars Mahinovs posted any photos there. At FOSSASIA 2010 in H Ch Minh City (HCMC) was a mini-DebConf. I arrived at the HCMC airport and was greeted by Huyen (thanks!!), one of FOSSASIA's numerous volunteers, who bundled me into a taxi bound for the speakers accommodation and pre-event meetup at The Spotted Cow Bar. The next day the conference opened at the Raffles International College and after looking at the schedule I noticed that I was to give a talk about Debian that day. Since I didn't volunteer for such a talk and had nothing prepared, the schedule took me by surprise. So shortly after an awesome lunch of Vietnamese pancakes we gathered some Debian folks and a random Fedora dude and prepared a short intro to Debian. The rest of the day the highlights were the intro, video greetings and the fonts, YaCy and HTML5 talks. The next day the Debian MiniConf began with Arne Goetje and everyone trying to get Debian Live LXDE USB keys booted on as many machines in the classroom as possible (many didn't boot). Once people started showing up we kicked off with Thomas Goirand's introduction to the breadth of Debian. Others talked about Debian pure blends, Gnuk and building community and packages. The second last session was about showing the Vietnamese folks in the room how to do l10n and translation since Debian had only one Vietnamese translator (Clytie Siddall). After manually switching keyboard layouts (seems LXDE doesn't have a GUI for that) on the English LXDE installs, the two Cambodian folks were able to do some Khmer translation too. This was a great session and it resulted in two extra Vietnamese translators joining Debian. It went over time so I didn't end up doing my presentation about package reviewing. We rushed off to a university where the random Fedora ch^Wambassador was hosting a Fedora 14 release party in a huge packed classroom. There were a lot of excited faces, interesting and advanced questions and it was in general a success. Afterwards we had some food, joined up with some other speakers and ended up in a bar in the gross tourist zone. On the final day we hung around in the Debian room, went downstairs for the group photo and final goodbyes. Later we found a place with baked goods, coffee and juices and navigated the crazy traffic to a nice local restaurant. The next morning Arne & I went to the airport, others went on a Mekong Delta tour and Jonas hung out with the organisers. I took less photos than at other events but got a few interesting ones. I avoided doing a lot of FLOSS stuff over the last year, I hope to work on some things in the coming months; I'm also planning some interesting travel and acquiring some new technological goods, more on those in some later posts.

6 January 2011

Gunnar Wolf: Back with Ruby on Debian polemics

Once again, a polemic regarding how to properly integrate the Ruby language and libraries with the Debian distribution has been ignited. Similar arguments were presented in November and December 2008 and September 2010 And excuse me for just refering to my own blog, but there are links to some other posts from there... and there will surely be many others I just missed. There were even nice collaboration attempts (such as DebGem, announced in January 2009... But which apparently never stuck on, as it is still listed as in its public Beta period and lists only support for Debian 4.0 and Ubuntu 8.04 and 8.10). Some days ago, while I was on vacation, I received a mail from Lucas Nussbaum expressing the burnout he had been suffering from this situation. Some days after that, he posted a corresponding blog post - Giving up on Ruby packaging. The comments following his post are most interesting The one comment I'd like to highlight (noting I skimmed a whole deal of them) is by Paul Brannan, one of the original RubyGems authors. Yes, possibly the design criteria for Gems included some mutually-conflicting goals, and they cover some of Debian's goals. Unfortunately, this conflicting criteria... Was resolved to the opposite way we would have wanted (specifically, that a sysadmin should be able to install software with the same tools he was already familiar with). And of course, there are deeper disagreements, which are rightly stemmed from different priorities: Debian (+derivatives) is meant to make user's and sysadmin's lifes simpler. Gems are meant to make developers' lifes simpler. And while all developers are some sort of users, and sysadminsthe reverse is (obviously) not so. A developer maintaining several different codebases will surely suffer (well, this particular developer has surely suffered) if he has to maintain them all using coherent versions of system libraries. When a system is programmed, its developers want to use the greatest, latest tools to offer the best experience and functionality. And it's natural for libraries to evolve over time. However, any sysadmin will grind their teeth at the prospect of having many different versions of libraries, as the Gem model proposes. Why? Well, I was bitten by a minor example of it Bug fixes do not get backported. Of course, if simple bugs are not backported, security and stability of the system as a whole suffers. Often, new software versions will not only add functionality, but improve on how things are done. Not only bugs get corrected, but we get better response times, better memory handling, etc. It might sound harsh to say this, and even more, as a developer it feels I am talking against myself But while development time should be minimized during the system design/implementation, once systems are in production, it should be used to allow for friendlier sysadmin relations. I have to wear both hats at my real-life job. So, what is my take on this? Of course, blaming myself for choosing the wrong version is a no-go. When evaluating a library before using it in my projects, I try to look at its history and API-stability. APIs that change often speak about immature libraries which are still trying to get the right way to implement their functionality; just-started projects might be great and revolutionary, but they do not yet show any kind of long-term committment... And can lead (and have often led me) to painful rewrites when the Next Big Thing is reached. As for the users: Even if your favorite language has the best and friendliest distribution method, users just do not care about what language was a particular piece of software implemented in. Users want to be able to install and uninstall with their system's usual tools. Directly using Gems, CPAN, PEAR and whatnot is just unnatural for users, however convenient they are for developers. Distributions offer a non-technical advantage in this regard as well: A human filter. As an example, as I write this there are 19392 Ruby Gems and 19092 Perl modules in CPAN (note that CPAN stores some older versions but discourages authors from keeping too many - So no, they are still very different in size). Debian has around 30,000 packages. Why is that? Because all Debian packages _must_ be human-generated, human-reviewed, human-submitted. This means, a person must think each packaged piece of code is worth packaging, is stable enough and provides value to users as it is, and is fit for being part of a stable release. I am not saying with this that 90% of CPAN and Gems are crap I am saying that they are probably early implementations, to be installed, tested and improved by developers, and still not apt for general public use. Or maybe not interesting enough to be packaged as a service for the non-techie public at large. Oh, and one last point: Ruby is not a new language anymore. It is a mature, powerful language with different implementations, every time more stable... But it is a language deeply affected by the (not so new anymore either) the appearance of Rails. Why do I say this? Because although the language is in no way tied to Web development, many of its strongest uses are Web-oriented. How does this affect the current discussion? Well, because many people argue that users are no longer needed to install the software. Web systems are installed by sysadmins and used via a Web browser, and sysadmins are expected be more skilled than casual users. Still, in Debian (and in other distributions, surely) we try to make sysadmin's lives simpler I have (again, talking out of personal experience) installed several webapps (and system tools, and whatnot) for which I never followed any instructions besides aptitude install foo Using different languages, frameworks, and so on. Can I troubleshoot their installs? Probably, as there is a common logic for how the distribution I have chosen and specialized in works. Can I find causes for bugs in them? Possibly, although there are some languages and frameworks I dare not touch. Can I get help on getting them out of a tight spot? Surely, as there is a central bug tracking system for my distribution And one of the maintainer's tasks is to separate the problems related to the distribution (packaging, installing, simple user questions and misconceptions) to those derived from real bugs upstream. Anyway I am not saying that our way is the best way. No, by a long shot. Again, developers should have an easy, convenient way for installing whatever they want to play with. And to publish it without jumping through hoops. With this post, I'm only trying to express again why Debian works the way it does. And hope for better cooperation in the future. And as for several comments of what I read in Lucas' post, I think that there is interest for this synergy among some of the most committed Ruby people.

Enrico Zini: Streaming JSON objects with python

Streaming JSON objects with python Dear Lazyweb, Suppose I'd like to create an HTTP service that gives you a looong list of JSON objects. For example, one JSON object for every package in Debian. One way I could do that is to just transfer a big, dozens of megabytes JSON array with one element per package. With the standard json module in python, this requires the client to buffer all the data in memory for decoding, as the JSON result would be something like this:
[
    info for package 1  ,
    info for package 2  ,
  ...dozens of megabytes...
    info for package 30000  ,
  ...
]
However, it would be reasonable to engineer things so that the client can process packages one at a time, as they are produced. This can dramatically reduce client memory usage, as well as remove a large latency between the start of the request and the start of processing. One trivial idea would be to just send JSON objects one after the other, and sort of call load multiple times. It doesn't work:
import json
from cStringIO import StringIO
a = StringIO()
json.dump( 1:3 , a)
json.dump( 2:4 , a)
a.reset()
print a.getvalue()
print json.load(a)
gives:
$ python z1.py 
 "1": 3 "2": 4 
Traceback (most recent call last):
  File "z1.py", line 8, in <module>
    print json.load(a)
  File "/usr/lib/python2.6/json/__init__.py", line 267, in load
    parse_constant=parse_constant, **kw)
  File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/json/decoder.py", line 322, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 8 - line 1 column 16 (char 8 - 16)
And the json module documentation does not provide any way of telling the "stream" decoder to just stop parsing after the first valid object. An alternative can be to use chunked transfer encoding and transfer each JSON object in a separate HTTP chunk. It would add the overhead of a little HTTP header per JSON object, which could be significant when generating lots of small JSON objects, but it would be an implementation that strictly follows HTTP and JSON standards to the letter in a clean way. Except:
  1. How do you stream HTTP chunks in WSGI? The WSGI specification's only mention of chunks is that if you produce a result as a generator of "chunks", they are just sent one after the other (effectively, concatenated) in the response stream.
  2. How do you read and process one chunk at a time with urllib/urllib2? I searched on Google but I only found people in tears.
Another alternative is to wrap the JSON objects in some custom container protocol, but then we are not talking standards anymore. Is this all you can do with the standard python library? Can this only be done by adding a dependency on a C library with two obscure separately maintained bindings, or by writing my own JSON parser? Ideas? Please mail enrico@enricozini.org and I'll add them here. Updates Wouter van Heyst mentions that he heard of ijson, which indeed has my exact use case in the synopsis, but after some research looks like a third binding for yajl. Aigars Mahinovs suggests using YAML instead of JSON. Indeed YAML supports streaming very well (it's even already supported in DDE, but last time I tried to do anything serious with it, PyYaml was slooow. Of course, this being the Python ecosystem, There Are Lots Other Ways To Do It And You Shall Be Patronised For Not Chosing The Proper One, but Debian only has PyYaml. Another itch with YAML is that there is no YAML parser in the standard python library, which requires clients to add an extra dependency. Jyrki Pulliainen has so far mailed my favourite solution: disable formatting in the encoder and separate JSON objects with newlines:
we built a streaming JSON app as a part of our platform. However, we
didn't stream a single JSON object as that was a bit cumbersome. When
having syntax like:
[
  <json-obj1>,
  <json-obj2>,
  ....
  <json-objN>
]
The reader has some problems. Should it read the whole stream to find
the closing bracket ] even if the stream would me tens of megabytes?
Should it try to split on , and parse single objects? What if the
objects contain , so should there be some regexp magic?
We ended up streaming separate JSON objects separated by newlines
(\n). This way the reader can read until hitting \n or end of stream
and then split the result on newlines. The new stream looked like
this:
<json-obj1>\n
<json-obj2>\n
...
<json-objN>\n
This way we could efficiently stream hundreds of thousands of JSON
objects from server to client without reserving a load of memory. For
the WSGI part how to do this, the best would probably to do a
generator that would yield a single JSON object with the newline (we
didn't use WSGI back then). Reading can be done using any reader
capable of handling streams. We used pycurl for reading, but I can't
really recommend it as it is way too complicated for a simple task
like this :p
Hopefully this helps!
This means that if the client has no access to a JSON reader that supports a stream of concatenated toplevel objects, they can still trivially code it on top of any JSON library. This means that working clients can be written based just on Squeeze's python standard library. That's what I'm going to do; I'll probably implement it as a separate 'jsons' output format for DDE, because producing a JSON response with multiple toplevel obejcts somehow breaks the JSON standard. Thanks to all who provided this excellent information!

Next.

Previous.