Ian Haken: Trials and Tribulations of ARM Programming
I spent a rather long time today trying to figure out a rather obscure bug. I few months ago I bought a gp2x, an open portable gaming system (and media player, etc.). I've really enjoyed it so far. The open platform for development has lent itself it many fun apps, console emulators the most conspicuous I think, but it's also fun to experiment and develop with it. I recently ported an asteroids clone I wrote and the whole process was actually fairly simple. As such, I decided to take on a larger task of porting something like FreeCNC, which as far as I know, no one has done yet.
After some simple trouble with library version conflicts, I got the program compiling correctly. However, if wasn't reading files quite right. I spent a little while investigating and discovered the problem was in a function which essentially hashes filenames. In the end, it was this snippet of code which was giving the trouble:
char buffer[13];
...
calc = ROL(calc)+*(long *)(buffer+i)
In particular, this is trying to read 4 bytes from this character array as an integer. At first I though the trouble might be endian-ness, but both the gp2x (an ARM processor) and x86 use big-endian*. I finally resorted to running gdb and sifting through assembly level instructions.
$r2 = 0xbffffd7c
$r3 = 0x41434f4c
0x0000a310 _ZN8MIXFiles7calc_idEPc+52>: str r3, [r2]
Here, the register $r2 points to buffer and $r3 is the value we're trying to save into the buffer. After this we would expect
*0xbffffd7c = 0x41434f4c
but instead see
*0xbffffd7c = 0x08db9841.
If I purposely set *0xbffffd7c = 0xffffffff and repeat this instruction we see
*0xbffffd7c = 0xffffff41.
Clearly the problem lies in this single str instruction, and was resident to the ARM architecture. So of course it was time to go to the processor documentation. Reading the fine print finally reveals what I was looking for.
A word store (STR) should generate a word aligned address. The word presented to the data bus is not affected if the address is not word aligned. That is, bit 31 of the register being stored always appears on data bus output 31.
Yep, since the buffer was an odd-sized array and stored on the stack, it started on an odd number in memory. Hence its address was not word aligned, and so as per the documentation, the store did not have the expected result. Here's a short program which shows what I'm talking about.
Perhaps the lesson to learn here is that when casting pointers, it's buyer-beware, and there are safer bitwise manipulations to use, though certainly less efficient.
*Edit: Both x86 and the ARM processor in question are little-endian, not big-endian. Typo on my part, but notwithstanding the point is that they both had the same endian-ness, so it wasn't where the issue lay.
After some simple trouble with library version conflicts, I got the program compiling correctly. However, if wasn't reading files quite right. I spent a little while investigating and discovered the problem was in a function which essentially hashes filenames. In the end, it was this snippet of code which was giving the trouble:
char buffer[13];
...
calc = ROL(calc)+*(long *)(buffer+i)
In particular, this is trying to read 4 bytes from this character array as an integer. At first I though the trouble might be endian-ness, but both the gp2x (an ARM processor) and x86 use big-endian*. I finally resorted to running gdb and sifting through assembly level instructions.
$r2 = 0xbffffd7c
$r3 = 0x41434f4c
0x0000a310 _ZN8MIXFiles7calc_idEPc+52>: str r3, [r2]
Here, the register $r2 points to buffer and $r3 is the value we're trying to save into the buffer. After this we would expect
*0xbffffd7c = 0x41434f4c
but instead see
*0xbffffd7c = 0x08db9841.
If I purposely set *0xbffffd7c = 0xffffffff and repeat this instruction we see
*0xbffffd7c = 0xffffff41.
Clearly the problem lies in this single str instruction, and was resident to the ARM architecture. So of course it was time to go to the processor documentation. Reading the fine print finally reveals what I was looking for.
A word store (STR) should generate a word aligned address. The word presented to the data bus is not affected if the address is not word aligned. That is, bit 31 of the register being stored always appears on data bus output 31.
Yep, since the buffer was an odd-sized array and stored on the stack, it started on an odd number in memory. Hence its address was not word aligned, and so as per the documentation, the store did not have the expected result. Here's a short program which shows what I'm talking about.
| x86 | ARM |
| buf=BFBF3471, buf%4=1 Num: 4C4F4341 Buf+0: 4C4F4341 Buf+1: 4C4F4341 Buf+2: 4C4F4341 Buf+3: 4C4F4341 | buf=BFFFFD75, buf%4=1 Num: 4C4F4341 Buf+0: 4F434100 Buf+1: 43410000 Buf+2: 41000000 Buf+3: 4C4F4341 |
Perhaps the lesson to learn here is that when casting pointers, it's buyer-beware, and there are safer bitwise manipulations to use, though certainly less efficient.
*Edit: Both x86 and the ARM processor in question are little-endian, not big-endian. Typo on my part, but notwithstanding the point is that they both had the same endian-ness, so it wasn't where the issue lay.
DeBaBaReTools seems to work good enough that writing an early-stage HOW TO seems to worth the time :)
When cloning a Linux/Debian system 1:1 you should have a checklist of what’s important to change on the cloned system.
For cloning I use 
That’s my personal checklist:
Following Lucas[
This kind of smoothing actually eliminates high frequencies, hence we see what a usual behavior each step has. What is clear to me is that FD situation is clearly fixed for quite a long time. FD was definitely an issue one year ago, but it has been really really fixed: for more than a year, even on the 6-months smoothing graph that has quite a long memory, the average queue is around 3 people, I can't see how it could even be reasonably improved.
Of course, given that the Keyring step is just after the DAM, and that people at those steps works in batches, the data is a bit biased, though it gives a pretty good idea of the bottlenecks of the whole process.
Like most things, the scale of decisions that have to be made by a person
or organisation fall across a wide spectrum. At one end, you've got the
fine-detail work: lines of code, individual config settings, IP addresses,
that sort of thing. This is crucial work, because it's this detail that
ultimately produces saleable results. Travelling all the way to the other
end, there's the "big picture" decisions: the "visionary" stuff. What
market segment will we target? What will we do (in large-scale terms)
to get there?
I am convinced that individuals naturally gravitate towards one end of this
spectrum or the other, depending on their psychology. While people usually
can work at all scales, they have a "most comfortable" point where they
work the most naturally. The further away from this point they are pushed,
the less comfortable and effective they are.
No level of thinking is better or worse than the other; they're just
different. While "large scale thinker" and "small scale thinker"
could be taken as being insulting, I certainly don't mean it that way.
I should point out that "Strategy" isn't (just) about setting goals. A
strategist isn't someone who comes out with grand pronouncements like "By
the end of next year this company should be providing telecommunications
services to 20% of the Fortune 500" without knowing how to get there.
That's not a strategist, that's a poor-quality management consultant.
People whose only skill is making vague pronouncements should just be taken
out and shot. They're not doing anything useful for anyone.
A strategist is someone who is good at planning the goal of "20%
of the Fortune 500". They're thinking about the big picture issues,
without getting bogged down in the minutiae. A tactician, given the job of
planning something at the macro scale, will chase some good idea down an
intricately detailed rabbit hole, and miss the other dozen or so large
issues that need to be considered. The strategist's planning needs to stop
at a certain level and let other people take on the detail work.
Conversely, a good tactician can focus on the individual details
without getting distracted by the big picture. A strategist put into a
tactical role will miss all the important minutiae because they're working
out how their little part fits into the big picture, instead of working out
how to make their little part work, in and of itself.
It's important, for individual happiness, to try and work out what level
you're most comfortable thinking at, and then trying to find a job that
puts you there. It's almost as important to make sure that the people who
are working at the more and less detail levels around you are equally
comfortable.
Take, for example, the micromanager. We've all either been there ourselves
or have heard multiple stories on the subject. I think the micromanager is
nothing more than a person who wants to work at a smaller scale than they're
being forced to by their organisational rank. Since they can't do
the detail work any more, their only outlet is working vicariously through
their subordinates. Voila! Instant micromanagement.
An even more painful case of micromanagement would be when the subordinate
wants to think at a larger scale than the superior. In that case, the
subordinate will naturally be thinking large scale, and will miss the
details. To the small scale thinking superior, that is incompetence, and
will naturally be dealt with by more micromanagement. In reality, both
parties are incompetent at their jobs, and they should really swap roles,
so that everyone is in their element.
It would be nice if companies stopped thinking hierarchically in this way
-- that a more "senior" person has to think at larger levels. Just because
you've been somewhere longer (or have been working in an industry longer, or
got higher marks, or whatever), it doesn't mean that you're more of a
strategist. This is probably the Peter Principle restated: people get
promoted to a level where they just can't comfortably think at the necessary
scale any more. If instead people were put into a planning hierarchy based
on their natural scale of thinking, we'd all be a lot better off. There'd
be a lot less power games, though, which makes me think it's not going to
happen any time soon.
A business, at whatever size it might be, needs an appropriate mix of
strategists and tacticians. A really small business (1-10 people) can get
away without needing large scale thinkers, in the same way that a really
small boat can get away without needing a rudder or pilot. If a really
small boat gets itself into trouble, everyone on board can grab an oar or
bucket and help get the boat out it's predicament -- it's small and nimble
enough to be able to do that.
As the company expands, though, it's important to make sure you've got some
strategists around. If you don't have anyone to keep an eye out for
problems and plan around them, your small-to-medium sized boat is likely to
have a hole punched in it by a submerged rock. Even if somebody on board
happens to notice the rock before you actually hit it, once you get to a
certain size you can't physically stop or turn the boat in time.
If you don't understand the growth issues in the company, you might not
even realise that you're no longer the nimble and low-inertia dinghy until
you're up to your knees in sea water.
Although an obvious solution to this problem might be to say that all
businesses should be small and nimble ones, it's as impractical as saying
that all boats should be dinghies. A fleet of dinghies isn't going to be
able to support the worldwide sea freight market, and a fleet of small
companies doesn't have the necessary resources on hand to take on the big
projects that are so often needed. Also, a dinghy is at risk of being
swamped by one freak wave, while a larger ship can weather significant
storms without going under.
Regardless of the size of the business, there always needs to be a good
supply of tacticians. A boat (of whatever size) full of people who are all
looking out for rocks isn't going to go anywhere, because nobody's making
the thing move. Similarly, a company full of strategists isn't
going to be doing any of the day-to-day work that actually earns money, and
no money means no company.
The single most important thing, though, is be self-aware enough to know
what sort of person you are, and try to find something that lets you work at
the level you're most comfortable at. Otherwise you're just setting
yourself up for unhappiness.
