Ian Wienand: Debugging __thead variables from coredumps
I think the old-fashioned coredump is a little under-appreciated
these days. I'm not sure when it changed, but I even had to add
myself to /etc/security/limits.conf to raise my
ulimit to even create one.
Anyway, debugging __thread variables from coredumps is a
bit of a pain. For the uninitiated, the __thread identifier
specifies a thread-local variable, i.e. every thread gets its
own copy of the variable automatically.
The implementation of this is highly architecture specific. The
reason is that TLS entries need to be accessed via a register kept as
part of the thread state, and thus every architecture chooses their
own register and builds their own ABI. On x86-32, which is very
register-limited, you certainly don't want to dedicate a register to a
pointer to TLS variables and take it out of operation. Luckily there
is the hang-over from the 70's (60's? 50's?) segmentation.
Without going into real detail, segment registers can be used to
offset into a region of memory based on a look-up of a region
descriptor stored in a table.
Above, you see a simplified example of the %gs register
loaded with the index value 2, and thus when you access
%gs:20 what you are saying is "find entry 2 in the global
descriptor table (GDT), follow it and offset 20 into that
region.
The kernel gives each thread its own GDT (i.e. the GDT register is
part of the thread-state). Thus __thread variables are
stored based on segment offsets and voila thread-local
storage. Now, there's a few tricks here. For various reasons, a
process can not setup entries in the GDT; this is a privileged
operation that must be done by the kernel. There is actually a
special system call for threads to setup their TLS areas in the GDT
set_thread_area. When a new thread starts, the
thread-library and dynamic linker conspire to allocate and load any
static TLS data (i.e. if you have a global __thread variable
initialised to some value, then every thread must see that value when
it starts) and then calls this to make sure the variables are ready to
go. After that, the gs register is filled with the index of
that GDT entry, and all TLS access goes via it. That, in a nut-shell,
is TLS for x86-32.
Now, to the problem. Take, for example, the following short
program:
We start a few threads and then abort to make it dump core. But,
if you try and examine foo:
It seems that gdb doesn't know how to find the value of
foo because its not a variable in the usual sense ("this
target", in this case, means a coredump). It relies on accessing via
the gs register, which relies on the current processes' GDT
state, which has since been destroyed. If you care to consult the canonical source of
TLS info, you can find out exactly why this is so hard to figure out
generically. However, with some work, we can start to figure out the
value by hand.
A coredump is really a ELF file full of just two things: a bunch of
LOAD segments that are just dumps of the process memory
regions, and a NOTE section that includes a bunch of notes
that the kernel dumps out for us such as the current register state,
the process id, the signal that killed us, etc. Here's an example of
a core file under readelf
You can examine the notes; with readelf we see:
This shows 5 notes of type NT_PRSTATUS; it should be
little surprise that each of these notes describes the process status
of one running thread. When gdb pops up [New Thread 4973]
that's because it just hit another note that describes the new
thread.
To actually make sense of the note, however, we need some other
tools that look deeper. The elfutils based tools give us a better
description of the various notes in the coredump; more akin to what
GDB is interpreting them as. Below I've extracted one thread's
info:
What's particularly interesting here is that the note type that was
previously unknown (Unknown note type: (0x00000200)) has been
resolved for us it is in fact of type NT_386_TLS and
is a dump of the GDT entries for the thread.
So, if we examine how our function is accessing our TLS variable by
disassembling the thread function:
Examining the disassembly we can see that foo is accessed
via an offset of -4 from %gs (this is OK, as our limit value
is maxed out. See the TLS ABI doc for more info). Now, we can
examine gs and see which selector it is telling us to
use:
Above we shift out the last 3 bits, as these refer to the privilege
level (bits 0 and 1) and if this is a GDT or LDT reference (bit 2).
Thus, looking at the GDT descriptor for index 6:
we can finally do some maths from the base-address to figure out
the value:
Thus we have found our TLS value; in this case for thread 3 the
value is indeed 3. I caution this is the simplest possible case;
other "models" (see the TLS doc, again) may not be so simple to work
out by hand, but this would certainly be how you would start.
Above, you see a simplified example of the %gs register
loaded with the index value 2, and thus when you access
%gs:20 what you are saying is "find entry 2 in the global
descriptor table (GDT), follow it and offset 20 into that
region.
The kernel gives each thread its own GDT (i.e. the GDT register is
part of the thread-state). Thus __thread variables are
stored based on segment offsets and voila thread-local
storage. Now, there's a few tricks here. For various reasons, a
process can not setup entries in the GDT; this is a privileged
operation that must be done by the kernel. There is actually a
special system call for threads to setup their TLS areas in the GDT
set_thread_area. When a new thread starts, the
thread-library and dynamic linker conspire to allocate and load any
static TLS data (i.e. if you have a global __thread variable
initialised to some value, then every thread must see that value when
it starts) and then calls this to make sure the variables are ready to
go. After that, the gs register is filled with the index of
that GDT entry, and all TLS access goes via it. That, in a nut-shell,
is TLS for x86-32.
Now, to the problem. Take, for example, the following short
program:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
int __thread foo;
void* thread(void *in)
foo = (int)in;
printf("foo is %d\n", foo);
while (1)
sleep(10);
int main(void)
pthread_t threads[5];
int i;
for (i=0; i<5; i++)
pthread_create(&threads[i], NULL,
thread, (void*)i);
sleep(5);
abort();
$ gdb ./thread core GNU gdb (GDB) 7.2-debian Reading symbols from /home/ianw/tmp/thread/thread...done. [New Thread 4970] [New Thread 4975] [New Thread 4974] [New Thread 4973] [New Thread 4972] [New Thread 4971] Core was generated by ./thread'. Program terminated with signal 6, Aborted. #0 0xffffe424 in __kernel_vsyscall () (gdb) print foo Cannot find thread-local variables on this target
$ readelf --headers ./core ELF Header: Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 Class: ELF32 ... OS/ABI: UNIX - System V ... Type: CORE (Core file) ... There are no sections in this file. Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align NOTE 0x0003d4 0x00000000 0x00000000 0x006b4 0x00000 0 LOAD 0x001000 0x08048000 0x00000000 0x00000 0x01000 R E 0x1000 LOAD 0x001000 0x08049000 0x00000000 0x01000 0x01000 RW 0x1000 LOAD 0x002000 0x0804a000 0x00000000 0x21000 0x21000 RW 0x1000 LOAD 0x023000 0x46015000 0x00000000 0x00000 0x1b000 R E 0x1000 LOAD 0x023000 0x46030000 0x00000000 0x01000 0x01000 R 0x1000 LOAD 0x024000 0x46031000 0x00000000 0x01000 0x01000 RW 0x1000 ...
$ readelf --notes ./core Notes at offset 0x000003d4 with length 0x000006b4: Owner Data size Description CORE 0x00000090 NT_PRSTATUS (prstatus structure) CORE 0x0000007c NT_PRPSINFO (prpsinfo structure) CORE 0x000000a0 NT_AUXV (auxiliary vector) LINUX 0x00000030 Unknown note type: (0x00000200) CORE 0x00000090 NT_PRSTATUS (prstatus structure) LINUX 0x00000030 Unknown note type: (0x00000200) CORE 0x00000090 NT_PRSTATUS (prstatus structure) LINUX 0x00000030 Unknown note type: (0x00000200) CORE 0x00000090 NT_PRSTATUS (prstatus structure) LINUX 0x00000030 Unknown note type: (0x00000200) CORE 0x00000090 NT_PRSTATUS (prstatus structure) LINUX 0x00000030 Unknown note type: (0x00000200) CORE 0x00000090 NT_PRSTATUS (prstatus structure) LINUX 0x00000030 Unknown note type: (0x00000200)
$ eu-readelf --notes core
...
CORE 144 PRSTATUS
info.si_signo: 6, info.si_code: 0, info.si_errno: 0, cursig: 6
sigpend: <>
sighold: <>
pid: 4970, ppid: 4960, pgrp: 4970, sid: 4960
utime: 0.000000, stime: 0.000000, cutime: 0.000000, cstime: 0.000000
orig_eax: 270, fpvalid: 0
ebx: 4970 ecx: 4970 edx: 6
esi: 0 edi: 1176018932 ebp: 0xbfa5b710
eax: 0 eip: 0xffffe424 eflags: 0x00000206
esp: 0xbfa5b6f8
ds: 0x007b es: 0x007b fs: 0x0000 gs: 0x0033 cs: 0x0073 ss: 0x007b
LINUX 48 386_TLS
index: 6, base: 0xb6043b70, limit: 0x000fffff, flags: 0x00000051
index: 7, base: 0x00000000, limit: 0x00000000, flags: 0x00000028
index: 8, base: 0x00000000, limit: 0x00000000, flags: 0x00000028
(gdb) disassemble thread Dump of assembler code for function thread: 0x08048514 <+0>: push %ebp 0x08048515 <+1>: mov %esp,%ebp 0x08048517 <+3>: sub $0x18,%esp 0x0804851a <+6>: mov 0x8(%ebp),%eax 0x0804851d <+9>: mov %eax,%gs:0xfffffffc 0x08048523 <+15>: mov %gs:0xfffffffc,%edx 0x0804852a <+22>: mov $0x8048670,%eax 0x0804852f <+27>: mov %edx,0x4(%esp) 0x08048533 <+31>: mov %eax,(%esp) 0x08048536 <+34>: call 0x80483f4 <printf@plt> 0x0804853b <+39>: movl $0xa,(%esp) 0x08048542 <+46>: call 0x8048404 <sleep@plt> 0x08048547 <+51>: jmp 0x804853b <thread+39>
(gdb) print $gs >> 3 $31 = 6
LINUX 48 386_TLS
index: 6, base: 0xb6043b70, limit: 0x000fffff, flags: 0x00000051
(gdb) print *(int*)(0xb6043b70 - 4) $34 = 3
My 
Year ago, when 
Following a series of pre-releases, 
I ve taken the plunge, and converted my blog from a venerable but largely
unmaintain(ed able) blosxom setup that I ve had since 2005 (OMG!) into a new
and shiny Jekyll-backed system. The main benefits are that I m more
comfortable hacking away at Jekyll (being written in Ruby), and my blog post
submission no longer requires me to e-mail my blog posts to myself to put
them up (that was just an artifact of how I set it up all those years ago,
but it had worked so I wasn t willing to change it just for the sake of it).
Jekyll s superior (for me) hackability has reaped dividends already I ve
put together a home-grown comments system, which fulfils my primary
requirements of being as close to completely statically-generated as it s
possible to be (about 10 lines of PHP is the only dynamic content), and
doesn t involve JavaScript or putting my blog comments in the cloud .
Tagging words.
I tried calculating TF/IDF for tokyodebian monthly report.
I found that mecab isn't very suited for this task.
A while ago I registered with 
Last week, the 
So as Alpha 2 approaches, I find myself working towards porting the alternate installer/d-i to the pandaboard to support the netboot installer. There's not a lot of documentation that describes the internals of d-i, nor what bits are platform specific.
Unfortunately Debian cannot sponsor my trip to Banja Luka this year and it s too expensive to pay it from my own budget. Thus I just cancelled my attendance for the conference. I m very sad, since my first DebConf last year in NY was a lot of fun and I was really looking forward to see you guys again this year.
I wish you a lot of fun anyways. Have a productive and fun conference!
You re a volunteer, you re not doing stuff for Debian because someone told you so. You must have some intrinsic motivation and the ultimate motivation is usually that you re enjoying what you re doing.
So what are you enjoying and/or what are your motivations ?
Find something to do
At this point you have a generic idea of what you d like to do but you re still missing a concrete objective. Let s try to find one, we ll explore several ways to do this.
Scratch your itch
The best objectives are those that satisfy your own needs. Here are some examples:
If you don t have a specific itch to scratch, you might want to focus your work on a specific team. Head over to 
You can concentrate your work on a specific package even if you re in a team, it s often a good idea. But I list it separately because not all packages are team maintained and you might want to help maintain a package where there s a single maintainer currently. I leave it up to you to find a way to select a package that interests you
Then head over to the package tracking system: 
Today I was made aware of bug
* Alas. Not going to DebConf 11 
However, these RAID volumes are in a pretty awkward spot for installers and bootloaders: inside a running Linux environment they look like software RAID which cannot be depended on for booting, but at boot time they look like hard disks which cannot be looked under the hood.
I recently got a new desktop machine which has one of these motherboards, and fortuitously my old desktop I was replacing had the same size disks so I had 4 disks and the option of using a RAID setup. Apparently I m a sucker for punishment because I went for a RAID 10 (that is two RAID volumes made up of two-disk mirrors (the RAID 1 component), and then those two volumes are combined via striping (the RAID 0 component). This has the potential for pretty nice performance: in principle any read can come from one of 2 disks, and every 64KB (the stripe size) of linear data will switch to the other mirror set, giving a nice boost. Writes need to write to 2 disks always, but every 64KB worth of data will alternate mirror sets, also giving a boost.
Sadly we (Ubuntu) aren t ready for this yet: there are two key bugs that make this layout almost impossible to install into. This blog post is for my exo-memory, I want to be able to figure out what I did next time around 
After testing Emacs, Eclipse, KDevelop and CodeBlocks for writing C++ code, I decided to stick with Netbeans. It seems to be the most simple to configure and yet full of features and plugins. I was having one problem with it that when I focused out the window, and then focused on it again, it would not really grab the focus, in the sense that I would not be able to type without first clicking with the mouse. At first I thought the problem was with Netbeans, then with the JRE. I tried using sun-java6-jre, I tried upgrading my openjdk-6-jre, and nothing worked. I searched a little bit more and got to 
Trying the BitTorrent DHT
A month ago, noticing that there was a lack of a good BitTorrent tracker
software (good meaning complying with my random requirements ), I
I'm coming as well! Really looking forward to meet the people from Debconf9 again. Also people from the Games Team and the buildd and kfreebsd folks. And ideally there will be some more people interested in (Common) Lisp as well, we'll see