What is Valgrind?
Valgrind is a framework for building dynamic analysis tools. Several useful tools are
included in the Valgrind distribution including tools to check dynamic memory usage (
memcheck), a cache profiler (
cachegrind), heap profiler (
massif) and thread debugger (
helgrind) among others. Valgrind also provides a framework which can be used to build other tools.
The Valgrind tool which I find most useful and the one which I have most experience with is
memcheck. This tool can detect all manner of memory management problems, including use after free, using uninitialized data, memory leaks, double free. Between them these can result in savings of many hours of staring a core dumps and gdb backtraces.
How Does
memcheck Work?
At its core Valgrind is a dynamic binary translation engine, which is used to instrument the code at runtime. In the case of
memcheck this is used to track properties of the memory in a process, including aspects such as whether each bit (yes, bit) of memory has been initialized since it was allocated. It also tracks this information as data is loaded into registers, so it can know if a given register is currently tainted with uninitialized data or not. As well as instrumentation through binary translation Valgrind also includes instrumented versions of the C dynamic memory allocation functions which are used to track whether a each bit of memory is currently considered free or allocated, as well as tainting registers when they contain a pointer to memory which has been freed.
A large part of
memcheck's functionality is built upon Valgrind's ability to determine when memory has been initialized. However although binary translation can be used to instrument when the application itself has initialized some memory it is not currently possible to instrument the behaviour of system calls, in other words Valgrind cannot automatically determine when memory has been initialised e.g. after a
read(2) system call. For this reason Valgrind has baked in knowledge about the behaviour of system calls on various platforms.
For example lets consider the
read(2) system call. The prototype of
read(2) is:
ssize_t read(int fd, void *buf, size_t count);
This system call reads up to
count bytes from the file descriptor
fd into the memory pointed to by
buf and returns the number of bytes read. Here is the code within Valgrind which handles this system call (found in
coregrind/m_syswrap/syswrap-generic.c):
PRE(sys_read)
*flags = SfMayBlock;
PRINT("sys_read ( %ld, %#lx, %llu )", ARG1, ARG2, (ULong)ARG3);
PRE_REG_READ3(ssize_t, "read",
unsigned int, fd, char *, buf, vki_size_t, count);
if (!ML_(fd_allowed)(ARG1, "read", tid, False))
SET_STATUS_Failure( VKI_EBADF );
else
PRE_MEM_WRITE( "read(buf)", ARG2, ARG3 );
POST(sys_read)
vg_assert(SUCCESS);
POST_MEM_WRITE( ARG2, RES );
These two functions are called before and after the system call respectively. The main piece of functionality is the calls to
PRE_MEM_WRITE and
POST_MEM_WRITE. In this case
PRE_MEM_WRITE is used to check that the memory starting at
ARG2 (recall that this is the supplied buffer) and extending for
ARG3 bytes (the system calls
count argument) consists of allocated memory. After
the system call in complete the post function calls
POST_MEM_WRITE to indicate to Valgrind that
RES bytes of the buffer have know been initialized.
The above is a relatively simple case, but the above two hooks must be supplied for each system call on each platform which Valgrind wishes to support. One of the most problematic system calls is the
ioctl(2) system call. The
ioctl call takes a file descriptor, a request number and a pointer to a per-ioctl argument structure and implements per-device control requests. There
are literally dozens of devices many with their own specific ioctls. For this reason a sizable portion of Valgrind code is dedicate to decoding the behaviour of the myriad of potential ioctls
and providing pre- and post-call instrumentation for them.
Interacting with the Hypervisor from Userspace
Under Xen the toolstack is a normal userspace process running in the control domain (typically domain 0). As part of its operation the toolstack needs to communicate with the hypervisor and to do this it uses hypercalls. However a userspace process cannot simply make hypercalls on its own, instead it must request that the OS kernel make the hypercall for it. This prevents just any userspace process making a hypercall, since the kernel will first check the privilege of the process making the request.
Under Linux a userspace process makes hypercalls by using the
ioctl(2) system call on a special "Privileged Command" (
privcmd) device. In this case the toolstack uses the
IOCTL_PRIVCMD_HYPERCALL ioctl request which takes a hypercall number and the hypercall arguments as its argument. Other OSes which can be used as a control domain posses similar interfaces.
Making
memcheck Work With Xen Toolstacks?
As might be expected the majority of the work to allow Valgrind to work on Xen toolstack processes was teaching it about the
IOCTL_PRIVCMD_HYPERCALL ioctl. Fortunately it was not necessary to teach Valgrind about every hypercall but only about those which are used by toolstacks. This is a smallish subset of all hypercalls including the
domctl interfaces and a handful of others.
The initial batch of patches cover all of the system calls needed to start new domains, shut them down and migrate them using the
XL toolstack. New hypercalls will be added as they are discovered to be missing.
Getting and Using This Functionality
The initial set of patches were accepted into Valgrind's
Subversion repository trunk revision 13081 in October 2012 and are expected to be part of the 3.9.0 release.
Until 3.9.0 is released it will be necessary to get this functionality from SVN. Fortunately this is relatively simple and follows the usual
autofoo dance. Although the usual caveats regarding running unreleased software apply.
$ svn co svn://svn.valgrind.org/valgrind/trunk valgrind
$ cd valgrind
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install
Once Valgrind is built and installed simply call it passing the command to be debugged as an option:
# valgrind xl list
==16186== Memcheck, a memory error detector
==16186== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==16186== Using Valgrind-3.9.0.SVN and LibVEX; rerun with -h for copyright info
==16186== Command: xl list
==16186==
Name ID Mem VCPUs State Time(s)
Domain-0 0 512 4 r----- 332.5
==16186==
==16186== HEAP SUMMARY:
==16186== in use at exit: 0 bytes in 0 blocks
==16186== total heap usage: 16 allocs, 16 frees, 93,529 bytes allocated
==16186==
==16186== All heap blocks were freed -- no leaks are possible
==16186==
==16186== For counts of detected and suppressed errors, rerun with: -v
==16186== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)
Here we see that
xl list has no leaks! However if I comment out the call to
libxl_dominfo_list_free in
tools/libxl/xl_cmdimpl.c:main_list() then instead Valgrind reports:
[...]
==16203== HEAP SUMMARY:
==16203== in use at exit: 90,112 bytes in 1 blocks
==16203== total heap usage: 16 allocs, 15 frees, 93,529 bytes allocated
==16203==
==16203== LEAK SUMMARY:
==16203== definitely lost: 0 bytes in 0 blocks
==16203== indirectly lost: 0 bytes in 0 blocks
==16203== possibly lost: 90,112 bytes in 1 blocks
==16203== still reachable: 0 bytes in 0 blocks
==16203== suppressed: 0 bytes in 0 blocks
==16203== Rerun with --leak-check=full to see details of leaked memory
==16203==
==16203== For counts of detected and suppressed errors, rerun with: -v
==16203== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)
[...]
So it has caught the memory leak! Rerunning with the suggests
--leak-check=full goes even further and tells me where the leaked memory was allocated:
[...]
==16204== 90,112 bytes in 1 blocks are possibly lost in loss record 1 of 1
==16204== at 0x4024480: calloc (vg_replace_malloc.c:593)
==16204== by 0x4050CE4: libxl_list_domain (libxl.c:548)
==16204== by 0x805EEC7: main_list (xl_cmdimpl.c:3931)
==16204== by 0x804D6DD: main (xl.c:285)
[...]
To ease debugging of domain creation I find it useful to use the
-F option to
xl create to stop
xl from forking and daemonising. Most
xl subcommands do not daemonize and so need no special treatment.
For more information on running Valgrind see the
Valgrind Documentation.
Conclusion
Valgrind is a powerful tool in the arsenal of the C programmer, and can be used to catch a large number of hard to debug and common issues before they happen can save many hours of tedious debugging. The ability to use tools such as this on Xen toolstack processes is of enormous benefit and has already
found several bugs in the XL toolstack.
Just back in from an amazing concert at the
Chicago Symphony Center. Vocalists
Angelique Kidjo,
Dianne Reeves and
Lizz Wright supported by all-star
band of
Geri Allen on piano,
Romero Lubambo on electric and acoustic guitar,
James Genus on electric bass guitar,
Munyungo Jackson on percussion and
Terri Lyne Carrington on
drums (and musical director). The concert alternates between solos and
joint pieces with more joy and soul than I heard in a long time. Great evening.
I finally manage to create an 
As usual, I've changed my 
What I have been up to over the last couple of years
Over the last few days, I ve made some incremental improvements to Hy, but I ve been mostly preparing it for a small talk later on this next week.
I am, however, already considering a bit of a rewrite there is a lot of code sitting in the Hy tree that was intended to use the old modfaker method of Python module generation (which sucked)
I m most interested (I think) in bootstrapping Hy in Hy (no small task) and getting macros in early (really really needed), so, dear lazyweb:
Has anyone split rpython out of PyPy yet? Are there any sort of docs on this?
The idea of rHy is pretty intriguing, I d like to make it happen.
More to come soon (including the slides!)
I've been lucky in life. I've known several people who've killed
themselves, but I've been friendly with them rather than friends with them.
Their deaths have saddened me, but haven't left giant gaping holes that
I have trouble imagining filling. I've also been lucky in that I'm not
especially predisposed towards depression, and even luckier that the one
serious episode I had ended well. That luck means I'm not qualified to
speak authoritatively on the topic, but this seems like a good
time to share.
Most of 2002 wasn't great for me. Many of my friends had graduated and
moved away. My PhD, which was what I'd spent the past few years of my life working towards,
was going horribly wrong. The girl I liked didn't like me. Lots of individual small
things that didn't fundamentally matter, but which build up into overall
feelings of isolation and failure at life. I was spending an increasing
number of days not leaving bed and not talking to people. It wasn't that
I couldn't have fun - socialising was still pleasurable, and I didn't
actively avoid people, but it was always tempered with the knowledge
that this was temporary and I'd be returning to the unhappiness
afterwards. I couldn't see any sequence of events that would turn my
life around and restore my happiness. This was how life was, and this
was how life was going to be. I'd irrevocably fucked up, and this was
my future.
Looking back, what strikes me is how reasonable this seemed. I could
point at specific things that were making me unhappy, and nobody could
argue that it was unreasonable to be unhappy about them. There wasn't any
point in speaking to people about it, because what would they be able to
do except agree that I was justifiably unhappy? I thought about suicide.
Not seriously, because overall life still seemed worthwhile, but I could
conceive of a level of further unhappiness that would make it seem like
the best choice. I don't think it would ever have occured to me to speak
to someone about it first. It seemed like it would be the same argument
- I'm justifiably unhappy, I already feel like I'm letting my friends
down, what could they do other than tell me that my feelings are wrong
or make me feel even more guilty? So, when I saw exhortions for people
to speak to someone if they felt suicidal, it seemed like they were
talking to people who hadn't thought this through as well as me. It felt
like I'd thought this all through carefully and rationally and come to a
completely logical decision. If changes in circumstances and further
consideration made it seem like suicide was the better choice, that
would be because it was the better choice. Maybe other people weren't
thinking about this as logically as I was. Maybe they'd have their minds
changed by speaking to a friend or a professional. I wouldn't. Of course, with hindsight I was rationalising the way I already felt rather than making entirely rational decisions. I could have rationalised myself to death even though there were (in my case) straightforward ways to make my life better.
In any case, I've no idea how close I ever got to that point. Things
were at their worst in August - by September I had a new job and new
house, and things just got better from there. In the end, the friends I
was convinced could do nothing for me ended up giving me the opportunity
to find gainful employment and made sure I had somewhere to live.
Without them, things might have been different. As it was, I spent less
than nine months depressed and it was still the most hellish experience
of my life. The thought of returning to that state is terrifying. I was
lucky. I might not be again.
There's no terribly good moral here. If I'd known more about depression
beforehand, I might have been able to identify what was happening to me
and seek professional help. Other than that, I didn't learn anything
about how to avoid or deal with depression. The experience didn't make
me a better person. I've no advice for people in the same situation. The
only thing I gained from it all was the realisation that if I'd
felt any worse and knew that this was what I faced for the rest of my
life, death might not have been the worst choice I had.
Depression is a huge social problem and we deal with it badly. We refuse
to talk about it, and when we do talk about it we mostly limit ourselves
to platitudes about how things will get better or placing the blame on
depressed people for not wanting to talk to those around them. Sometimes
it doesn't get better. Sometimes talking to those around you will make
things worse. People need to be aware of what the effects of depression
are and get better at identifying it in others, rather than assuming
that they'll be able to ask for help themselves. Society as a whole
needs to be better at ensuring that professional support is there for
people who need it. And, unless we can make massive improvements in our
understanding of the causes of depression and effective mechanisms for
countering it, we need to accept that it will cost us friends. Let's
redirect the anger we feel at their choice to avoid a lifetime of misery
into anger at the society that still hasn't done everything it can to
help them.
I just returned home from