There are several ways a program can hit the disk, and it can be hard to know exactly what s going on, especially when you want to take into account the kernel caches. This includes, but is not limited to:
- any access to a file, which may lead to I/O to read its parent directories if they are not already in the inode or dentry caches
- enumerating a directory with
readdir(), which may lead to I/O on the directory for the same reason
read()/write() on a file, which may lead to I/O on the file if it is not in the page cache- accesses in a
mmap()ed area of memory, which may lead to I/O on the underlying file if it is not in the page cache
- etc.
There are various ways to track system calls (e.g.
strace) allowing to track what a program is doing with files and directories, but that doesn t tell you if you re actually hitting the disk or not. There are also various ways to track block I/O (e.g.
blktrace) allowing to track actual I/O on the block devices, but it is then hard to back-track what part of which files or directories these I/O relate to. To the best of my knowledge, there are unfortunately no tools to do such tracking easily.
Systemtap, however, allows to access the kernel s internals and to gather almost any kind of information from any place in a running kernel. The downside is that it means you need to know how the kernel works internally to gather the data you need ; that will limit the focus of this post.
I had been playing, in the past, with
Taras script, which he
used a while ago to track I/O during startup. Unfortunately, it became clear something was missing in the picture, so I had to investigate what s going on in the kernel.
Setting up systemtap
On Debian systems, you need to install the following packages:
- systemtap
- linux-image-2.6.x-y-$arch-dbg (where x, y, and $arch correspond to the kernel package you are using)
- linux-headers-2.6.x-y-$arch (likewise)
- make
That should be enough to pull all the required dependencies. You may want to add yourself to the
stapdev and
stapusr groups, if you don t want to run systemtap as root. If, like in my case, you don t have enough space left for all the files in
/usr/lib/debug, you can trick dpkg into not unpacking files you don t need:
# echo path-exclude /usr/lib/debug/lib/modules/*/kernel/drivers/* > /etc/dpkg/dpkg.cfg.d/kernel-dbg
The file in
/etc/dpkg/dpkg.cfg.d can obviously be named as you like, and you can adjust the
path-exclude pattern to what you (don t) want. In the above case, kernel drivers debugging symbols will be ignored. Please note that this feature requires dpkg version 1.15.8 or greater.
Small digression
One of the first problems I had with Taras script is that systemtap would complain that it doesn t know the
kernel.function("ext4_get_block") probe. It is due to a very unfortunate misfeature of systemtap, where the
kernel.* probes refer to whatever is in the vmlinux image. Modules probes have a separate namespace, namely
module("name").*.
So for the
ext4_get_block() function, this means you need to set a probe for either
kernel.function("ext4_get_block") or
module("ext4").function("ext4_get_block"), depending how your kernel was compiled. And you can t even use both in your script, because systemtap will complain about either being an unknown probe
Tracking the right thing
I was
very recently pointed to a Red Hat document containing
4 useful systemtap scripts ported from dtrace, which gives me a good occasion to explain the issue at hand with the first of these scripts.
This script attempts to track I/O by following
read() and
write() system calls. Which is not tracking I/O, it is merely tracking some system calls (Taras script had the same kind of problem with
read()/
write() induced I/O). You could just do the very same with existing tools like strace, and that wouldn t even require some system privileges.
To demonstrate the script doesn t actually track the use of storage devices as the document claims, consider the following source code:
#include <fcntl.h>
#include <sys/stat.h>
#include <unistd.h>
/* Ensure the resulting binary is decently sized */
const char dummy[1024 * 1024] = "a";
int main(int argc, char *argv[])
struct stat st;
char buf[65536];
int i, fd = open(argv[0], O_RDONLY);
for (i = 0; i < 100000; i++)
read(fd, buf, 65536);
lseek(fd, 0, SEEK_SET);
read(fd, buf, 65536);
lseek(fd, 512 * 1024, SEEK_SET);
close(fd);
return 0;
All it does is reading some parts of the executable a lot of times (notice the trick to make the executable size at least 1MB). Not a lot of programs will actually do something as bold as reading the same data again and again (though we could probably be surprised), but this easily points the problem. Here is what the output of the systemtap script looks like for this program (stripping other unrelevant processes):
process read KB tot write KB tot
test 400002 25600001 0 0
Now, do you really believe 25MB have actually been read from the disk? (Note that the
read() count seems odd as well, as there should only be around 200000 calls)
Read-ahead, page cache and
mmap()
What the kernel actually does is that a
read() on a file is going to check the page cache first. If there is nothing corresponding to the
read() request in the page cache, then it goes down to the disk, and fills page cache. But as loading only a few bytes or kilobytes from the disk could be wasteful, the kernel also reads a few more blocks ahead, apparently with some heuristic.
But
read() and
write() aren t the sole way a program may hit the disk. On UNIX systems, a file can be mapped in memory with the
mmap() system call, and all accesses in the memory range corresponding to this mapping will be reflected on the file. There are exceptions, depending on how the mapping is established, but let s keep it simple. There is a lot of litterature on the subject if you want to document yourself on
mmap().
The way the kernel will read from the file, however, is quite similar to that of
read(), and uses page cache and read ahead. The systemtap script debunked above doesn t track these.
I ll skip write accesses, because for now, they haven t been in my scope.
Tracking some I/O with systemtap
What I ve been trying to track so far has been limited to disk reads, which happen to be the only accesses occurring on shared library files. Programs and shared libraries are first being
read() from, so that the dynamic linker gets the ELF headers and knows what to load, and then are
mmap()ed following
PT_LOAD entries in these headers.
As far as my investigation in the Linux kernel code goes, fortunately, both accesses, before they end up actually hitting the disk, go through the
__do_page_cache_readahead kernel function (this function was tracked in Taras script). Unfortunately, while it is called with an offset and a number of pages to read for a given file, it turns out the last pages in that number are not necessarily being read from the disk. I don t know for sure, because the latter had an effect on my observations, but some might even already be in the page cache.
Going further down, we reach the VFS layer, which ends up being file-system specific. But fortunately, a bunch of (common) file-systems actually share page mapping code, and commonly use the
mpage_readpage and
mpage_readpages functions, both calling
do_mpage_readpage to do the actual work. And this function seems to be properly called only once for each page not in the page cache already.
If my reading of the kernel source is right, this however is not really where it ends, and
do_mpage_readpage doesn t actually hit the disk. It seems to only gather some information (basically, a mapping between storage blocks and memory) that is going to be submitted to the block I/O layer, which itself may do some fancy stuff with it, such as reordering the I/Os depending on other requests it got from other processes, the position of the disk heads, etc.
And when I say
do_mpage_readpage doesn t actually hit the disk, I m again simplifying the issue, because it actually might, as there might be a need to read some metadata from the disk to know where some blocks are located. But tracking metadata reads is much harder, and I haven t investigated it.
Anyways, skipping metadata, going further down after
do_mpage_readpage is hard because it s difficult to back-track which block I/O is related to which read-ahead, corresponding to what read at what position in which file.
do_mpage_readpage already has part of this problem because it is not called with any reference to the corresponding file. But
__do_page_cache_readahead is.
So knowing all the above, here is my script, the one I used to get the most recent Firefox startup data you can find in my last posts:
global targetpid;
global file_path;
probe begin
targetpid = target();
probe kernel.function("__do_page_cache_readahead")
if (targetpid == pid())
file_path[tid()] = d_path(&$filp->f_path);
probe kernel.function("do_mpage_readpage")
if (targetpid == pid() && (tid() in file_path))
now = gettimeofday_us();
printf("%d %s %d\n", now, file_path[tid()], $page->index*4096);
probe kernel.function("__do_page_cache_readahead").return
if (targetpid == pid())
delete file_path[tid()];
This script needs to be used with a command given to systemtap, with the
-c option, such as in the following command line:
# stap readpage.stp -c firefox
Each line of output represents a page (i.e. 4,096 bytes) being read, and contains a timestamp, the name of the file being read, and the offset in the file. As discussed above,
do_mpage_readpage is not really the place the I/O actually occurs, so the timestamps are not entirely accurate, and the actual read order from disk might be slightly different, but it still is a quite reliable view in that the result should be reproducible with the same files even when they re not located on the same blocks on disk, and provided their page cache status is the same when starting the program.
This systemtap script ignores writes, as well as metadata accesses (including but not limited to inodes, dentries, bitmap blocks and indirect blocks). It also doesn t account accesses to files opened with the
O_DIRECT flag or similar constructs (raw devices, etc.)
Read-ahead in action
Back to the small example program, my systemtap script records 102 page accesses, that is, 417,792 bytes, much less than the actual binary size on my system (1,055,747 bytes). We are still far from the 25MB figure from the other systemtap script. But we are also far from the 128KiB the program actively reads (64KiB twice, leaving a hole between the two blocks).
At this point, it is important to note that the ELF headers and program code all fit within a single page (4 KiB), and following the 1MiB section corresponding to the
dummy variable, there are only 5,219 bytes of other ELF sections, including the
.dynamic section. So even counting everything the dynamic linker needs to read, and the program code itself, we re still far from what my systemtap script records.
Grouping consecutive blocks with consecutive timestamps, here is what can be observed:
| Offset |
Length |
| 0 |
16,384 |
| 983,040 |
73,728 |
| 16,384 |
262,144 |
| 524,288 |
65,536 |
(By now, you should have guessed why I wanted that big hole between the
read()s ; if you want to reproduce at home, I suggest you also use my
page cache flush helper)
As
earlier investigations showed, the first accesses by the dynamic loader are to read the ELF headers and
.dynamic section. As mentioned above, these are really small. Yet, the kernel actually reads much more: 16KiB at the beginning of the file when reading the ELF headers, and 72KiB at the end when reading the
.dynamic section. Subsequent accesses from the dynamic loader are obviously already covered by these accesses.
Next accesses are those due to the program itself. The program actively reads 64KiB at the beginning of the file, then 64KiB starting at offset 524,288. For the first read, the kernel already had 16KiB in the page cache, so it didn t read them again, but instead of reading the remainder, it reads 256KiB. For the second read, however, it only reads the requested 64KiB.
As you can see, this is far from being something like you wanted
n KiB, I ll read that fixed amount now .
Further testing with different read patterns (e.g. changing the hole size, read size, or reading from the
dummy variable directly instead of
read()ing the binary) is left as an exercise to the reader.
Debian's announcement to release
I lived in Israel between 1994 and 1996 This was just after the signature of the Oslo agreements, the Rabin-Arafat signature. 1994-95 was a beautiful time to be in Israel; there was the excitement of a prospect of real peace with the Palestinians, something that had been just seen as unattainable for the past 50 years.
But we also saw the buildup of a huge tension, of real hatred. And the dream of the end of the conflict, of two people living in peace side by side was heavily shattered on November 4 1995, when prime minister Rabin was assassinated. But for the sake of where I'm trying to get to (this post is meant to be about current Mexico, not of Israel 15 years ago), lets go a couple of months back.
The Israeli society is a deeply democratic one (although I understand many people feel it is losing that trait). Lots of political propaganda of all sorts, many of it very creative, are posted on the streets And not by the government or by political parties, but by citizen movements. I tried to follow it... I remember a particular sticker that I started seeing very often: Rabin has no mandate over the Golan . At first, I understod that the Golan residents rejected Rabin's authority (but, why was it posted all over the country? And what specifically was it about?), until a friend explained to me that it meant that it rather meant that the right wing all over the country felt that Rabin had no right to push for negotiations involving in any way the Golan heights return to Syria because that was not his electoral platform.
So, fast-forward in time, and 12500Km to the west.
If you have followed my ramblings for the past ~five years, you are aware of what I think about Mexico's government It is an illegitimate government, which got the power thanks to a dirty and illegal (as per Mexico's legal system) campaign and to voting fraud. And, according to what I have read, around 30% of the population still believe firmly the elections were stolen.
But even if they were legal, valid and legitimate What has happened since 2006 should not be seen as valid.
In Mexico, we don't have such a democratic culture. The society is not used to demanding anything from the politicians. We often see, as an example, that when talking about political organs, there is a divide everybody understands although makes no sense: We talk about citizens as people unaffiliated to political parties (v.gr. the 1996 Federal Electoral Institute was very well regarded as it was "el IFE ciudadano", and has since then been overtaken by the parties). Why are the parties' members not seen as citizens? Well, maybe because of the same distortion field that makes us believe our de-facto president when he stated that, from the (then) over 28000 people killed in the 
Up to now, 
There has been a quite long hiatus in my Debian activity, first due to 
My friend Jeff deserves a sincere congratulation for finally unveiling his rebranded R
consultancy 
Last year I shared my 
This shows the market did indeed bounce off the oversold lows nicely
on a few occassions in 2009 and 2010 --- but also continued to slide after
hitting the condition. Nothing is foolproof, and certainly nothing as simple
as this is, so buyer beware. But it may prove useful in conjunction
with other tools.
The code for the script is
Let's say you set up a machine using an encrypted disk with LUKS (debian-installer's partman makes this wonderfully easy!). You choose an initial passphrase, get the machine working, and it's working great. Then, you need to restart it, and realize that (for whatever reason) you've forgotten or lost the passphrase for the volume. oops! (i'm sure this has never happened to you -- let's just pretend it's your less-fortunate friend). If your system is still running, and you have superuser access to it, you can actually set a new passphrase for the LUKS volume using information that the dm-crypt kernel module has about the in-use mapping. In my examples, i'll imagine that the source volume is /dev/XXX2 and the exported cleartext volume is known by the device-mapper as XXX2_crypt In the bigger picture, this should serve as a reminder that even though your disk is encrypted, if someone gets live access to the superuser account on a system with the encryption keys loaded, your data is no longer secret. Before you do any tweaking, you might want to back up your LUKS header, just in case: 
Tomorrow, at this time, I'll probably be unsuccessfully trying to find a
comfortable position on a seat of the Bilmanbus to Irun. Very early
on Saturday, as soon as we get off the bus, Maria and I will quickly head to
Hondarribia's beach in Cape Higer to symbolically wet our feet in the waters
of the Cantabrian Sea.
We won't have much time to enjoy the cold waters of the ocean, though.
Soon after that, we'll have to take a deep breath, look East, and start
walking if we want to achieve our utmost objective: take a bath in the
beautiful beaches of the Cap de Creus, in the Mediterranean sea. In between,
30 days and 840 kilometres of thick woods, deep valleys, high peaks and cold
waters, all of which shape incredible landscapes.
Not for the first time, I ve seen the suggestion in the echo chamber that Mono packages should be moved from Debian into the non-free repository, which is not formally part of Debian. The reason, as it so often is, is patents specifically this time, the searing risk posed to Debian and its users that Mono s packaging does not (and technically could not without forking from upstream) provide base class libraries which implement only the content of ECMA-335 4th Edition. As I understand it, this implies three things about the suggestion/demand: firstly, that the individual in question believes that Mono end users are at risk from patent litigation from Microsoft Corp because Mono s implementation of Microsoft.NET beyond the content of ECMA 334/335 infringes on Microsoft patents; secondly, that the Microsoft Community Promise which promises not to assert legal claims over third party implementations of ECMA-335 4th Edition (and ECMA-334 4th Edition which defines the C# language) would render a pure ECMA-only runtime safe if it existed (which it does not); thirdly that without the protection offered by the Microsoft Community Promise, the source code licenses of Mono are irrelevant the patent risk renders the software non-Free.
It appears, unfortunately, that the community of Free Software Advocates don t actually understand what Free Software actually IS. That explains an awful lot, but should surprise nobody. So here s a lesson on what, exactly, is being advocated for.
The Free Software Foundation defines four Software Freedoms these are the minimum criteria required for something to be considered Free Software by the FSF:
