Search Results: "reg"

23 December 2007

Russell Coker: Contract Pay Rates

In a comment on my post about Bad Project Management [1] Don Marti [2] says “the more money you charge, the less of your time people waste, and the more seriously they take you” and “you can affect the client more, whether you’re peddling idiocy or wisdom“. That’s a nice theory, however I don’t recall seeing an example of it anywhere that I have worked. There are two broad categories of employers, large and small. Large companies have HR departments etc who decide on rough pay scales. Once the budget is approved (by senior management and HR) for someone’s contract fees or salary the amount of money seems to be of little concern to the line manager. They have a group of N people reporting to them who are all paid by money that comes from some mysterious place (with no direct connection to the quality or quantity of work done) and therefore the amount of money paid is not very relevant. Often team-leaders don’t even know how much the people who report to them are being paid! Note that this applies to people who have a similar job title, if I am paid twice as much as another sys-admin in a corporate environment I don’t expect to get more attention, but someone who has a more senior job title (or in the case of the UK has been to a more prestigious school or has a better accent) may get more attention. Small companies are rarely in a position to be silly in this way. This doesn’t mean that the most highly paid person gets the most attention, it’s more a matter of making a logical case. Most of the small companies I have worked for have been run in a sensible manner, and my ideas have often received more attention than those of others because I have had a better technical background than my colleagues. However in some cases my suggestions have received less attention because I was receiving a higher pay rate. The reason is that I was being paid a rate that the company could not afford in the long-term and therefore the people who would work there in future years would have to maintain the systems in question which meant that their ability to do such maintenance was often considered to be more important than my ideas about the best way to design the system. I have worked for a small company that had no financial pressure to do sensible things (they had some sweet deals going with major corporations which essentially gave them money for nothing). They had chaotic management which they could get away with because there were no immediate consequences for doing silly things (such as not forcing their clients to have a method of backing up servers that they installed). I can’t claim that Don’s advice in this regard never applies, but my observations of situations where it didn’t apply lead me to believe that it’s applicable to some situations in limited areas and is not a general rule. Don also refers to “a high number that you would quote to a client you didn’t want in order to get them to go away“. This is an issue that causes a lot of disagreement. One consultant I respect advises people to never make such quotes, if you don’t want a job then say that you are not available - one reason is that if a position is so bad that you want the potential client to go away then you don’t want to be tempted to reconsider if they agree to pay you a silly amount of money, another is that it may be considered ethically wrong to make an offer is not genuine. Another consultant I respect claims that his greatest career success was when he turned up to an interview in casual clothes, explained to the potential client that he would never wear a suit and demanded an excessive amount of money. The client agreed to such terms and he then went on to charge other clients similar amounts of money and continued to refuse to wear a suit. I won’t claim that it’s wrong to demand excessive amounts of money to make an unwanted potential client go away and I will not rule out the possibility of doing so at some future time. But I think it should be noted that the practice is controversial and not referred to as something that anyone might do. It is something that I don’t recall doing at any previous time in my career (so I haven’t done it in a formal manner at least - I may have done so at the informal discussion phase and forgotten). Please note that I encourage you to read my previous post with Don’s comment in it’s original context to avoid any risk of misinterpretation. The purpose of this post is not to disagree with Don (I agree with him almost all of the time) but to disagree with some very commonly held beliefs which Don represented in his comment.

Share This

Benjamin Mako Hill: Computing in the Cloud

On January 15th, I'm going to be giving a talk on a panel at the Computing in the Cloud conference held by Ed Felton's Center for Information Technology Policy at Princeton University. The conference description says:
Computing in the cloud is one name for services that run in a Web browser and store information in a provider s data center ranging from adaptations of familiar tools such as email and personal finance to new offerings such as virtual worlds and social networks. This workshop will bring together experts from computer science, law, politics and industry to explore the social and policy implications of this trend.
I'm going to talk about the AGPL 3.0, why it's important that we put effort into figuring out what freedom for different technologies means, and what the components of freedom for web services might be. Registration is free and bags you a name-tag and lunch. Worth noting perhaps, the conference is sponsored by Microsoft.

Christoph Berg: Datamining

I'm just digging through OFTC's nickserv database to do some cleaning. We have a bit over 20k nicknames in the database on 18k accounts which means about 10% of registered nicks are linked to other master nicks. By the power of sql, here's some statistics on the domain names of the email addresses our users: 
com  9144
net  2241
org  1778
de   1016
uk    392
nl    288
fr    223
edu   217
au    203
it    193
br    190
ru    174
ca    165
se     88
dk     74
at     70
fi     62
cx     59
info   58

gmail.com             3997
hotmail.com           1405
yahoo.com              843
gmx.de                 221
gmx.net                201
web.de                 156
debian.org             148
free.fr                 94
aol.com                 90
msn.com                 73
comcast.net             72
gentoo.org              71
mail.ru                 65
xs4all.nl               59
linuxmail.org           58
verizon.net             57
yahoo.co.uk             54
yahoo.com.br            45
googlemail.com          45
student.uq.edu.au       44
sbcglobal.net           36
earthlink.net           33
users.sourceforge.net   32
The numbers could use some aggregation as some providers use zillions of TLDs (yahoo, gmx). My personal favorite in there is root@localhorst :-)

22 December 2007

Petr Rockai: on blogging

The last time I have blogged, few people told me that it sounds somewhat patronizing or such (especially that it may sound so to people that do not know me). Well, I have to admit that it is hard to write in a decent tone — the intention and motivation behind the text is oftentimes hard to express. Certainly, better writing skills would probably help — or choosing easier topics. But that is how I am — and afterall, you never learn much when you only do things you already know how to do. So please, dear reader, take no offense in my blog… If you know me, you can probably see through my writing and understand what I mean. If you do not and still enjoy it, the better. And if you do not, please accept my apology — but in that case, it probably was not meant for you. Yes, I know the blog is aggregated in a few places, and I will probably have to do something about this — aggregate only subsets of the blog, appropriate for each of the aggregators. But pretty please, bear with me, I am lazy and it always took me ages to fix even trivial problems.

Ondřej Čertík: People still (and will) use windows

I sometimes look at the download statistics of SymPy and the funny thing is that people download more the Windows installer, rather than the multiplatform source tarball. Part of this can be due to the fact, that SymPy is in Debian, Ubuntu and Sage, but still.

So here is how to make a windows installer in Debian:

$ ./setup.py bdist_wininst
[...]

and a file dist/sympy-0.5.9-hg.win32.exe is created. I only have regular python tools installed. So we just upload the exe to our site and that's it -- kind of scary, since I don't have means to test it.

But one time I tried that on one of brother's computers with windows and not only it worked, but also the 3D plotting using pyglet worked out of the box! If you are curious how it looks like:



So it has it's advantages to develop pure Python programs - they really run everywhere.

I think installing things like Python and playing with SymPy on windows must be pain (maybe people just download SymPy for windows and then run away with disgust). But just the fact that they try means, that people doing science do use windows a lot. And I don't think this is going to change any time soon. I also fully agree with Michael Abshoff's recent post about this issue.

21 December 2007

Russell Coker: A Better Design for Child Seats

The current method of carrying young children (less than 4-6 years old) in cars is to have a special car seat fitted in the back seat. This has several significant problems:
Some car companies are offering child “booster seats” that are an optional attachment to the rear seat (I first noticed this when reviewing the specs of the latest version of the car I drive - the VW Passat [1]). This is a good idea, but it doesn’t go far enough. The best thing to do would be to provide a selection of back-seat assemblies as factory fitted options which have built-in baby and child seats. The combinations that would be most desired are:

  1. Standard car back-seat for three adults (or two adults for a small car).

  2. A regular seat (for an adult) at the road side of the car combined with a baby (backward facing) seat at the kerb side.

  3. A regular seat (for an adult) at the road side with a young child (forward facing) seat at the kerb side.

  4. A baby seat at the road side with a young child seat at the kerb side.

  5. Two young child seats.

It would be quite possible to have all five of these options available from the factory. Of course there are corner cases that this doesn’t cover such as twins or parents who have two children so close together that they need two baby seats. For those cases option 2 combined with one of the current off-the-shelf baby seats would do. The number of different supported options would need to be kept reasonably small to reduce manufacture cost and to allow a reasonable market for second-hand seats. One thing to note is that it’s recommended that the first forward-facing seat a child uses is smaller than the later one. Having options for three different built-in baby/child seats (rear-facing and two sizes of forward-facing) would significantly expand the number of combinations (and thus the expense). I suspect that the safety benefits of having an ideal method of securing a forward-facing child seat would compensate for the disadvantage of having it be too large for the child when they are first placed in it. Another possibility would be to replace the rear seat with a more solid bench with bolt holes for baby and child seats. Securing a child or baby seat to a hard surface with bolts would be a much less technically demanding task than using a seat belt (and thus could be done correctly without expert assistance). Child and baby seats would have to be redesigned for this (I suspect that the safety of them relies on being attached to a soft surface), but after that I expect that safety would improve. For this option the rear seat could bold on to a hard surface that’s suitable for attaching child/baby seats so it would simply be a matter of removing the rear seat and installing the child/baby seat(s). The most common car design in Australia includes a 60/40 split rear seat (meaning that if you have a large item to store in the boot/trunk then you can fold down 40% or 60% of the back of the rear seat to allow the luggage to extend into the passenger compartment). This split could be extended to allow removing the base of the rear seat for 60% or 40% to bolt on child/baby seats. Once a car model had been designed for replacing the rear seat there would be other options available. For example replacing the rear seat with luggage storage space. While almost all cars allow folding down the backs of the rear seats to store extra luggage the option of removing seats that you don’t need to give even more space is not common at all (I’ve only seen it advertised as a feature in vehicles with 6 or more seats). I expect that if this idea was implemented it would allow a small car such as a Toyota Corolla to give an equal or greater amount of usable space for children in the rear as a larger vehicle such as a Toyota Camry. While better options for luggage storage would allow people who don’t have children to use a small car while still being able to carry the luggage that they desire. This would allow considerable savings on car purchase prices and fuel use. I expect that a reduction in fuel use world-wide could be achieved by removing the pressure on parents to buy large cars! The poor support for child seats in cars is really surprising. One of the features that could be introduced is both top and bottom mounts for such seats. There is apparently a standard for this, some (not all) cars support it, but most baby seats apparently don’t. So baby and child seats are secured at the top (to a hook that’s bolted securely to the car frame and which was designed specifically for the purpose) and at the bottom to the seat-belt which was never designed for such things. It’s a pity that some of the money spent on supposedly protecting children from drugs couldn’t be spent on making cars safer for them. The government is in the best position to force car manufacturers to improve their safety features while parents are in the best position to teach children about the dangers of drugs.

Share This

Andrew Pollock: [tech] Waiting for a software RAID-1 to resync is like watching paint dry

I'm so annoyed with myself. So today was the day I was going to upgrade the disks in daedalus. I'd finally managed to convince Dell to sell me two new 140Gb disks to replace to two 70Gb disks that were fast approaching capacity. I'd organised for access to the datacentre from 10am. I arrived early. I didn't manage to get into the datacentre until about 11am, because they were having problems remotely unlocking the door, and I had to wait until an air conditioning repair guy turned up, who was fixing some air conditioning problem. I hadn't had a lot of time to plan how I was going to do the upgrade, but I figured it was going to be pretty straightforward. Each disk is partitioned into three partitions, all of which are mirrored using software RAID. The first partition is the root filesystem, which is just an ext3 filesystem, mirrored. The second partition is swap, also mirrored, and the third partition is an LVM physical volume. The first two partitions are only a couple of gigabytes. So the basic plan was to remove the second disk (disk 1, aka /dev/sdb), put in the new bigger disk, load the partition table from the original disk 1, resync the three mirrors, and then boot off the new disk 1, and rinse and repeat with disk 0. This was all going swimmingly, when right at the end (in fact after the third partition finished syncing according to the kernel), /dev/sda decided it had a few unreadable sectors. This made the software mirroring have a dummy-spit, and decide it needed to restart the syncing from scratch. I'd seen this before with a bad disk. So I wishfully hoped it was something to do with the new disk being bigger than the old disk, so tried to roll back to the original disk 1. As was pretty much expected, disk 0 exhibited the same problems at the same point. So the problem was disk 0, not the new disk 1. So it seemed like I just managed to pick the wrong half of the mirror to work from. I thought I'd try manually dd'ing the third partition from /dev/sda to /dev/sdb, ignoring the errors, but that didn't result in a usable half of a mirror either. Interestingly, these bad sectors hadn't made themselves known until this point. The RAID array had been otherwise healthy, and SMART hadn't uttered a peep. I think at this point, after spending about 3 hours fighting with it, I decided to move up a layer. I could successfully sync /dev/md0 (the root filesystem) and /dev/md1 (the swap device), and the only reason I could think of these bad sectors not giving me grief already, was because they were right at the end of the disk, and I hadn't quite used all the free physical extents in LVM yet. Lucky I hadn't decided to grow things any further. So I decided that rather than trying to convince this existing /dev/md2 to sync with the new /dev/sdb3, I'd just build a new degraded RAID-1 on /dev/sdb3 and move all the physical extents from the degraded RAID-1 on /dev/sda3 to the new degraded RAID-1 on /dev/sdb3. I tried the logical volume with /tmp on it first, since it was fairly sacrificial. It worked fine. So I tried the largest logical volume, /srv, which was 30Gb. That went fine as well, so I moved the rest. All of them went without complaint. So then I removed the now unused /dev/md2 from the volume group, and finally managed to boot with just the new disk 1. Then I was able to put in the new disk 0, and just do a standard RAID-1 rebuild onto it. Hooray for LVM saving the day yet again. And this is where I should have stopped. At this point it was about 3pm. I'd been aiming to leave by 4pm to pick up a rental car that I had to get by 5pm. I'd wanted to also upgrade to Etch while I was physically in front of the box, in case there were any nasty kernel/udev issues that made the box unbootable. The catch was, I needed to grow /var, which was quite full. The intention had been that once I'd got the two new disks in and all synced up, I'd create a fourth mirrored partition, and add this to LVM as a new physical volume, then I could just keep merrily growing things. It felt a little bit dirty having to make a new partition to get at the extra space, and I'd reading about mdadm's "grow" mode, whilst I was sitting around waiting for things to sync. So I thought to myself, I'd just delete the third partition, and recreate it using the remaining additional cylinders I now had at my disposal, then use this grow feature of mdadm to tell Linux the array was now bigger, then resize the physical volume, and go from there. This is where things went a bit pear-shaped. I deleted and recreated (larger) the third partition on each disk, and rebooted. I was then messing around with mdadm trying to update /etc/mdadm/mdadm.conf, since the UUID of /dev/md2 had changed, when I discovered that /dev/md2 had completely disappeared. It seems that recreating that partition made Linux completely fail to see /dev/sda3 and /dev/sdb3 as RAID-1 members, and so because of the fairly transparent nature of Linux's RAID-1, LVM had seen the underlying physical volumes, and just decided to run with /dev/sda3 as the physical volume, and everything had continued to work. I just had no redundancy any more. So it was about 3:20pm at this stage, and I was starting to panic, thinking I'd never get this sorted out before I had to leave, and I'd end up returning to the US with a non-redundant mess on my hands. Then I remembered that I had the backups of the partition tables of the old disks, from before I removed them. So I thought I'd go back to that state, and just have the additional cylinders unallocated. I hadn't yet attempted to grow the RAID-1 (partly because it had vanished), so in my haste, I didn't see this being a problem. The problem was, I'd booted into multi-user mode, and things had merrily been operating on just half the mirror. I restored the partitions, and rebooted, and lo and behold, /dev/md2 magically reappeared, already 100% consistent according to /proc/mdstat. I was somewhat surprised by that, but I booted back into multi-user mode again, only to discover that the /var filesystem was panicking and remounting itself into read-only mode. This was pretty weird, given that during boot, it was considered clean by e2fsck. So I was really getting bothered by this stage, so I rebooted into emergency mode (single-user mode in Debian starts way too much), and manually started up LVM, and forcibly fscked everything. There was a bit of filesystem damage to /var, /home and /srv. Thinking I'd now fixed the problem, I rebooted into multi-user mode again, but again /var, which was considered clean during the fscking/mounting phase of booting, panicked shortly after mounting, and remounted itself read-only. So I rebooted again, broke the mirror, and forcibly fscked the filesystems again (only operating on what was on /dev/sda3. Then I rebooted without incident into multi-user mode and resynced the mirror. At this point I ran away. So I'm really annoyed with myself. This should have been a blog post about how crap Linux's software RAID is at error recovery, and how LVM saved the day yet again, but it was overshadowed by me overreaching in my haste, and managing to cause silent data corruption. The morals of this story: I think that is all. I'm just so pissed off that what could have been a perfect (enough) upgrade with zero data-loss, even considering I had a bum disk, wasn't.

Phil Hands: Virgin Media (ex. NTL) support Linux just fine, despite their claims not to.

Today I helped a friend sign up with Virgin Media's broadband service. He was having trouble, as their front-page at http://activation.virginmedia.com/ does a brain-damaged test, and tells you that they only support Windows and MacOS X and that if you have anything else you'll need to phone them at a pound a minute so you can talk to some idiot until you lose the will to live (or some such). After a brief rummage, I noticed that if instead of doing what they tell you to do, you go to https://autoreg.autoregister.net/ you get prompted for whether you're signing up for Broadband or Dialup, and on clicking Broadband, you're lead through the sign-up procedure with no complaints about compatibility, and within 5 minutes he was online with his GNU/Linux (Ubuntu) system and Firefox browser. This makes me wonder why Virgin Media bother putting extra effort into turning users away when in fact the sign-up procedure would work perfectly if they hadn't bothered with the stupid browser test.

Ian Wienand: Playing with the x86 PMU

A discussion about commas lead to an excuse to have a play with the IA-32 processor performance managment unit (PMU). To start, take two versions of a program to count the number of commas in a text file — one in C and one in Python. The C one runs faster on the input data set of ~60MiB of random data, but why? The CPU performance monitors give are the key to getting some idea of where the programs spend their time. I like to use perfmon2 because it's what I know, but Oprofile can do it too. All the available events for IA-32 are described in the manual; I currently of no better way of finding out about them than just reading it. On Itanium I reccommend Caliper which, for the most common situations, does most of the work for you and presents it in a nice report. Intel's Vtune also does a similar thing. The first thing to investigate is if the CPU is getting enough instructions to keep busy. The IFU_MEM_STALL metric is a good place to start as it is triggered when the instruction fetch pipeline is stalled, presumably waiting on either the ITLB or the trace buffer (Icache). 
$ pfmon -e CPU_CLK_UNHALTED,IFU_MEM_STALL ./comma < ./randomcommas
340559375 CPU_CLK_UNHALTED
   192115 IFU_MEM_STALL
$ pfmon -e CPU_CLK_UNHALTED,IFU_MEM_STALL python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
1287100
4571257047 CPU_CLK_UNHALTED
 71981750 IFU_MEM_STALL
That works out to 0.05% of total cycles for the C version and 1.5% for the Python version, neither of which sets off immediate warning bells. If it did, we could start drilling down to things like the L2_IFETCH and ITLB_MISS events, or the BR_* branch events to try and see why the CPU is having to wait to get its next instruction. Next it is useful to find the CPI (cycles per instruction). This is calculated by the ratio of retired instructions against the number of CPU cycles; since a superscalar machine can issue more than one instruction per cycle this should ideally be much greater than 1 (for example, an Itanium can execute up to 6 instructions each cycle). 
$ pfmon -e INST_RETIRED,CPU_CLK_UNHALTED ./comma < ./randomcommas
542953593 INST_RETIRED
340612036 CPU_CLK_UNHALTED
$ pfmon -e INST_RETIRED,CPU_CLK_UNHALTED python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
1194455205 INST_RETIRED
4569931735 CPU_CLK_UNHALTED
This works out at a CPI of 1.59 for the C version and 0.26 for the Python version. The Python version is clearly spending a lot of time waiting, because it isn't even able to issue one instruction every cycle. At this point it seems the CPU has enough instructions to do, but it is sitting around waiting to get through those instructions. This suggests the waiting is related to getting data from the cache. The load and store requests from the L2 cache are accounted to the L2_LD and L2_ST events respectively. These have the ability to mask out cache lines in different states of the MESI protocol, but for this we don't care so just ask pfmon to show us everything. 
$ pfmon -e L2_LD:M:E:S:I,L2_ST:M:E:S:I ./comma < randomcommas 
102505 L2_LD:M:E:S:I
   167 L2_ST:M:E:S:I
$ pfmon -e L2_LD:M:E:S:I,L2_ST:M:E:S:I python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
3278774 L2_LD:M:E:S:I
  10457 L2_ST:M:E:S:I
This shows us that the Python version does quite a few more stores than the C counterpart. Considering this program should simply be reading the input stream and counting the number of commas, we do not expect much store traffic at all. This suggests the Python version is doing some extra copying, for whatever reason (maybe some Python expert can pinpoint it?). We can drill down a bit more into the memory related latencies. The DCU_LINES_IN event gives the total number of lines allocated in the cache. Another event, DCU_MISS_OUTSTANDING, gives a weighted measure of the cycles spent waiting for a cache line to be brought in. Each cycle spent waiting is weighted by the number of outstanding cache misses (I think the Pentium M I'm using can have up to 4 cache miss requests outstanding at once) and has some caveats, but can be considered a rough estimate of the time spent waiting for a cache line to be brought in. Therefore divding DCU_MISS_OUTSTANDING by DCU_LINES_IN gives us an approximate metric of how long a cache miss takes. 
$ pfmon -e DCU_MISS_OUTSTANDING,DCU_LINES_IN ./comma < randomcommas 
769736 DCU_MISS_OUTSTANDING
102387 DCU_LINES_IN
$ pfmon -e DCU_MISS_OUTSTANDING,DCU_LINES_IN python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
99810150 DCU_MISS_OUTSTANDING
 4240179 DCU_LINES_IN
So that works out to 7.5 cycles for the C version and 23 cycles for the Python version. This seems to strongly suggest that it is memory traffic that is weighing the Python version down. That is only the initial phase; the results give a high level idea of why one program is running slower than the other. The initial analysis phase generally consists of taking the ratios of certain events to try and get some idea in your head of what the program is doing. Then comes the really hard work; drilling down to figure out how to fix it! Some useful references:

20 December 2007

Andreas Schuldei: Helping Debian

Regarding Lucas' points that he makes about helping debian, I think there is an important factor for people that was only mentioned between the lines: "Instant Gratification". It is much more gratifying for a contributor if his effords have immediate effects. (I guess that is one reason for the success of Wikis.) Things that slow down the effect usually dimish this gratification.

So what can people who want to help Debian do to achive more imminent gratification? Pick projects that let you help directly with direct svn/git/whatnot access (Security team, Debian Edu, Debian-Installer...).

On the other hand we should try to come up with ways to cut down on delays as much as possible and make it easier to rollback and repair if an error occures. (That is almost allways a good thing anyways.)

19 December 2007

Ian Haken: Trials and Tribulations of ARM Programming

I spent a rather long time today trying to figure out a rather obscure bug. I few months ago I bought a gp2x, an open portable gaming system (and media player, etc.). I've really enjoyed it so far. The open platform for development has lent itself it many fun apps, console emulators the most conspicuous I think, but it's also fun to experiment and develop with it. I recently ported an asteroids clone I wrote and the whole process was actually fairly simple. As such, I decided to take on a larger task of porting something like FreeCNC, which as far as I know, no one has done yet.

After some simple trouble with library version conflicts, I got the program compiling correctly. However, if wasn't reading files quite right. I spent a little while investigating and discovered the problem was in a function which essentially hashes filenames. In the end, it was this snippet of code which was giving the trouble:

char buffer[13];
...
calc = ROL(calc)+*(long *)(buffer+i)

In particular, this is trying to read 4 bytes from this character array as an integer. At first I though the trouble might be endian-ness, but both the gp2x (an ARM processor) and x86 use big-endian*. I finally resorted to running gdb and sifting through assembly level instructions.

$r2 = 0xbffffd7c
$r3 = 0x41434f4c
0x0000a310 _ZN8MIXFiles7calc_idEPc+52>: str r3, [r2]

Here, the register $r2 points to buffer and $r3 is the value we're trying to save into the buffer. After this we would expect
*0xbffffd7c = 0x41434f4c
but instead see
*0xbffffd7c = 0x08db9841.
If I purposely set *0xbffffd7c = 0xffffffff and repeat this instruction we see
*0xbffffd7c = 0xffffff41.

Clearly the problem lies in this single str instruction, and was resident to the ARM architecture. So of course it was time to go to the processor documentation. Reading the fine print finally reveals what I was looking for.
A word store (STR) should generate a word aligned address. The word presented to the data bus is not affected if the address is not word aligned. That is, bit 31 of the register being stored always appears on data bus output 31.
Yep, since the buffer was an odd-sized array and stored on the stack, it started on an odd number in memory. Hence its address was not word aligned, and so as per the documentation, the store did not have the expected result. Here's a short program which shows what I'm talking about.

x86ARM
buf=BFBF3471, buf%4=1
Num: 4C4F4341
Buf+0: 4C4F4341
Buf+1: 4C4F4341
Buf+2: 4C4F4341
Buf+3: 4C4F4341		buf=BFFFFD75, buf%4=1
Num: 4C4F4341
Buf+0: 4F434100
Buf+1: 43410000
Buf+2: 41000000
Buf+3: 4C4F4341

Perhaps the lesson to learn here is that when casting pointers, it's buyer-beware, and there are safer bitwise manipulations to use, though certainly less efficient.


*Edit: Both x86 and the ARM processor in question are little-endian, not big-endian. Typo on my part, but notwithstanding the point is that they both had the same endian-ness, so it wasn't where the issue lay.

18 December 2007

Norbert Tretkowski: MySQL 5.0.51 uploaded to Debian

Yesterday in the evening I uploaded MySQL 5.0.51-1 to Debian unstable. Unfortunately it took a bit longer to prepare the updated package than expected, mainly because 5.0.51 wasn't MySQL's best relase, and I had to backport and include some fixes from 5.0.52 and 5.0.54 to get it working (see the changelog for details). Then I had to wait a few more days until 5.0.45-5 got into testing before I could finally upload 5.0.51-1 to unstable.

The most important changes are:Regarding the Community vs. Enterprise decision I had a discussion with Kaj Arn , Lenz Grimmer, Daniel Fischer and Monty Taylor on IRC last week and we decided to stay with the Community Edition in Debian for now, and see what the future will bring us.

15 December 2007

Jos Parrella: glibc 2.7 in Virtuozzo VPSs

If you’re using a Virtuozzo-based VPS running Debian Testing or Unstable (and probably other modern Linux distributions), you will probably run into problems regarding the patches used in the default kernel and glibc 2.7, which were properly described in the debian-glibc mailing list. Tektonic, my current provider, will upgrade the kernel this weekend but if you’re really in a hurry, you can set your VPS on repair mode, download a 2.6 version of libc6, install it from the repair VPS using dpkg –root=/repair –force-all –install … and restart your VPS. It will boot up, but lots of things that depend on versioned packages of libc6 won’t work correctly, most notably OpenSSH and PAM. The problem is due to non-standard TUX optimization patches implemented over Red Hat’s standard kernels, which seem to be quite popular in Virtuozzo-based VPSs (at least it seems that Strato, Tektonic and Unixshell use them?)

Benjamin Mako Hill: Annual Free Software Foundation Fundraiser

It's an end of year tradition for non-profit organizations to do big fund-raising and membership pushes. As I mentioned several days ago, I am personally giving to two organizations this year: the Wikimedia Foundation and the Free Software Foundation. The FSF has a goal of 500 associate members by year-end and it's an important goal that will sustain the foundation's activities. While membership dues keep the lights on, the fact that the foundation has a robust and growing membership is equally, if not more, important. FSF executive director Peter Brown put an appeal online in both video and text versions. In it, he lays out some of the most important issues for the next year. You should watch the video version in OGG Theora or this YouTube version (requires Gnash or non-free Flash). The appeal briefly lays out the FSF's plans for next year. My partner Mika Matsuzaki and my friend Oliver Day shot and edited the video. Please pass the link around to those you feel might be interested. Here's my appeal:
Now is the time to join and give to Free Software Foundation. 2008 is going to be extraordinarily important year for free software. Eben Moglen likes to quote Gandhi's "first they ignore you, then they laugh at you, then they fight you, then you win" progression when describing the free software movement. As I pointed out when I joined the FSF board, we're beginning to see powerful interests fighting free software. It's going to increase in the next few years. Things will probably get a lot uglier for free software before they get better. We can win but things are far from settled. The FSF is the front-line organization in this fight and we need a robust and proactive foundation, and an active and involved membership, if we're going to win. Here are the issues that I'm going to pushing the FSF to pursue in the next year. Expanding activism outside our traditional technologist communities: In part through the work of projects like Defective By Design, we've seen the tide turn for DRM on music in what what may be the FSF's greatest success last year. I'm going to push the FSF to continue the campaign to attack DRM for video, eBooks, and the other places it is cropping up. The most remarkable thing to me about Defective By Design is that its participants and supporters are not, for the most part, people who develop or use GNU/Linux or even know what GNU is! If advocacy for software freedom involves a conversation we can only have with people who understand what POSIX is and how one uses it, we've already lost. Through DbD, BadVista, and other projects, the FSF has made major strides in the last year. It need to do much more and needs your support to do so. Get proactive about software patents: As a community, we've had our head in the sand about software patents for far too long. There are companies and patent trolls sitting on massive, growing piles of software patents. They are not our friends and they do not mean us well. One cannot write non-trivial software today without running a serious risk of infringing patents. The software patents minefield we've found ourselves in is a very fundamental threat to the success of free software and we've already begun to see the first casualties and costs. We must eliminate software patents. Now. The US is very important in this fight (much patent law is "exported" from the US) and almost no organization is working on software patent elimination there. Not enough people are thinking and acting strategically on this issue. The FSF is planning to make major steps in this fight in the coming year and we need your support to do so. Web services and the changing face of software: This last year, I worked to help launch the new version the AGPLv3. The license addresses the role of copyleft for software like web-services which, due to the legal particulars of the GPL, did not extend to the purveyors of web services. Of course, access to source code does not make the users of all web-services free (e.g., the GMails and the Facebooks). Nobody seems to know what freedom for webserver entails. There might not even be good answers. In the next year, I'm going to push the FSF to help start several conversation and to begin to follow up on what I think was an important first step with the AGPLv3. While this is not a major organizational priority yet, it's a major action item that I will be pursuing through the FSF. If you feel strongly about this issue, whatever your position, become a member, stay involved as these projects develop, and have your voice be heard. We don't know the answers yet and we need your input as much as we need your action.
Please, support the FSF in the efforts listed above, and in others, by giving generously. If you're not a member, please join the FSF as an associate member. If you are already a member, please consider making a tax-deductible donation. The FSF is a very lean, very humble organization of passionate and dedicated individuals working tirelessly for software freedom. Every little bit helps. Members pays USD $120 ($10/month) and student members pay half that. FSF has members across the world -- where a weak dollar often makes it even cheaper. Member support and participation builds capacity and credibility for the foundation and keep the organization responsible, responsive, and in tune with our community.

14 December 2007

Axel Beckert: No more NDA for events hosted at Google Zurich?

I first heard about the Open Source Jam Zurich somewhere at BlogCampSwitzerland 2.0 (which was more a TechCrunch7 than a BlogCamp — why did the organisators call it BlogCamp?) and subscribed to its Google Group. At Linuxday.at, hansmi (who seems to be assimilated bywork for Google) gave me a flyer about Open Source Jam Zurich. And while reading it, I noticed that it will be held at Google’s Zurich office. Remembering the need for early registration for one of the recent Webtuesdays because of signing an NDA being necessary to get into Google’s office, I asked him, if I need to sign an NDA if I want to take part at Open Source Jam Zurich. He acknowledge it and so I returned the flyer and forgot about the Open Source Jam Zurich. Today Gürkan told me, he was at Open Source Jam Zurich at Google and he didn’t need to sign any NDA. He also told me that he knows other people which didn’t take part either because of the expected the need to sign an NDA. I was puzzled. Did Google really started to realize that “Open Source” and “Free Software” doesn’t fit together with “Non-Disclosure Agreements”? I hope so, because this would make it possible to come to all future Webtuesdays — my favourite local geek event — and not only to those not taking place at Google.

MJ Ray: Spam: The Leading Company

Today I was cold-called by someone announcing themselves as "UK Bank Claim", a service of "the leading company", asking me questions about my bank charges. Now, that put my back up right away. I co-own most of my banking service providers (aren't cooperatives and mutuals great?), and you can't get much more confusing service and company names if you tried. Then they claimed to be registered with the Ministry of Justice, but I can find no mention of this registration service on the MoJ web site. I've emailed the ministry to let them know. The phone number they called is registered with the Telephone Preference Service. The call did not send any Caller Line ID and refused to give a telephone number when asked, so I suspect it may have been from overseas. I registered a TPS complaint nonetheless. The caller gave me the domain theleadingcompany.co.uk which is registered to The Leading Company ltd - UK Limited Company, (Company number: 6081225), 28-32 church street, slough. Strangely, that is not the Registered Address: Excelsior House, 3-5 Balfour Road, Ilford Essex shown on their web site, nor the one listed at Companies House:- ROPE WALK SHOPPING CENTRE, ROPE WALK, RYE EAST SUSSEX - are companies allowed to use multiple registered addresses? I thought that was forbidden. In short, who is this company, why are they claiming government support and why are they allowed to behave like this this? How far should I go with this? Do you think I should contact hedleycreative.co.uk (whose email address is in their web source) and ask them why they're working for phone-spammers? I guess I shouldn't waste any more time, but I think it's worth publicising this behaviour a bit, to try to avoid these people finding unwary victims.

John Goerzen: A Clarification

You may remember my post Monday in which I told Mother Nature to "bring it on".

I would like to clarify that remark at this time. I would have preferred Mother Nature to interpret this in the President Bush sense of "please don't hurt us", rather than the "let's just knock out power to the entire plains region". A nice little romantic power outage Tuesday evening would have been nice.

We lost power at 3:10 on Tuesday according to my workstation's logs. It's still out, and our rural electric cooperative is estimating it will be another week until it's back, though that's better than the Dec. 22 date the state's much larger for-profit utility is giving people. We were lucky enough to be able to find a generator to rent on Wednesday morning. $35 a day isn't cheap. But on the other hand, we don't have a flooded basement due to not pumping out the sump pump, our food isn't spoiled, and our pipes aren't frozen.

At work, our Internet connection went down Tuesday morning. We use a fixed wireless connection, and this was the first time our ISP's radios have been tested in ice. Some antenna designs performed flawlessly. Then there were the antennas we had. 80% packet loss or worse. It wasn't safe to get up on the roof to deice them until Wednesday afternoon, and even then we couldn't get all the ice off. Finally today the connection was back to normal.

That two weeks after someone -- yes -- dug through Sprint/Embarq's apparent only fiber-optic link into town. Took down all long-distance calling into or out of the town, most cell towers, as well as all access to 911 (since dispatch is in another town) for about 8 hours. If we had still be using wired T1s, we would have been out of service with Internet as well.

All told, 200,000 households or business customers of the state's largest utility have been without power. Dozens of other utilities have also had significant outages. Ours alone lost 700 power poles and there are substations in need of repair all over the eastern part of the state. And we weren't even the hardest hit.

On the generator, we can generally pick any three of refrigerator, server, workstation, or furnace to run. So sometime warm, I'll upload the pictures I've taken so far. We have one nice CFL-powered electric lamp we're using, flashlights, and a kerosene lamp. It's amazing how after just a few days of this, we're starting to want to go to bed earlier. Or perhaps that was because I haven't had the chance to get a lot of sleep lately...

When we look out the house to the east, we see pitch black. Normally we can see some yard lights in the distance. Not this week. Nothing at all to the east. To the west, we can see yard lights tantalizingly close -- only 2 miles away, fed from a different location. Ah, the joys of being at the very end of a utility's network.

All this despite the fact that this storm wasn't really all that bad. The roads were never really terrible, and there weren't as many trees down as there were in the ice storm of 2005.

I heard a comment on the news today from an Iowa Republican who had followed the GOP debate yesterday. She mentioned that Romney made a comment like "Well, guess there's no global warming in Iowa with an ice storm like this, huh?" She pointed out that if Iowa was still as chilly as it used to, it would have been a snow storm instead of an ice storm. I agree.

13 December 2007

Martin F. Krafft: Delaying mail delivery

My current mailfilter has two features which increase my day-to-day productivity:
  1. a "tickler," which is a reminder system inspired by the tickler file component of David Allen's Getting Things Done action management method: I can submit emails and notes to the tickler along with a timestamp in the future, and the tickler delivers the mail (or note) to my inbox when the timestamp has passed.
  2. the ability to delay certain types of mail (e.g. Debian mail is held until the weekend, while news items and the like are only delivered at night).
My current implementations work alright, but they're brittle and crap. What's worse is that the two features could be combined and handled by one and the same tool, but I implemented them differently for now: The tickler consists of a Maildir, to which I can submit mails, either by mailing a note to a specific email address (currently broken, thus not linked from here), or by adding a tickle stamp (X-Tickle header) to a message with this script, saving it to the local tickler mailbox and asking offlineimap to shove it to the server. On this server, the tickler queue is regularly scanned by a script, which resubmits mail whose timestamp has expired to my mailfilter, where it is treated somewhat specially as a resubmitted mail. After almost three months with this setup, I can identify the following shortcomings: When I implemented the delay queue, I knew of these problems and went a different way: delayed mail is stored in a Maildir and a (msgid, timestamp, filename) tuple is inserted into an sqlite3 database on the mail server. This script regularly processes the queue. If I just sent you the chills, at least we have the same taste. There are numerous problems with this approach, the foremost being that Maildir filenames are not guaranteed to be constant: mails jump between the new and cur directories, and tags, such as seen are encoded in the filename (thus, symlinks also cannot be used). My script now uses an ugly heuristic, which at least makes it work. I should investigate whether inodes could be used instead as I think those wouldn't change throughout the lifetime of a mail, at least while it's not moved between folders. I initially considered just dumping messages to files and encoding the timestamp in the files' mtime, but then I would not be able to access the queue with mutt in case I needed to fetch a delayed mail prematurely, or if I wanted to synchronise the queue with offlineimap as well. The past few days, I've been condensing experiences from both approaches and am working out a new technique to combine both features. In essence, I think the database/index approach is the best, if I can figure out a way to uniquely identify mail message files, ideally across folders. Assuming I can use inodes for that, delayed mail would then be stored into the delayed Maildir and an (inode, timestamp) tuple saved into the database. Tickler mail would be stored along with all other mail in my store Maildir and would get a similar entry in the database. This approach solves some problems and leaves others. Assuming I synchronise the store Maildir remotely (which I do), then I can easily fathom making modifications via IMAP which causes orphan records in the database (if only IMAP would allow me to store key=value pairs for mails ). Furthermore, I'd have to submit mails to the tickler by bouncing them to an email address, and deleting the local copy, unless I want duplicates. If now the mail is somehow dropped, I've lost mail. Still unhappy about all of this, still searching for a better implementation, I'd appreciate any feedback! NP: Luminous Flesh Giants: Duma I Upadek

Joey Hess: if all you have is a hammer

David's post about programming languages and hand tools reminded me about the old saw about the hammer and the nail. The funny thing about that saying is that the hammer is actually one of the more versatile tools there is. A few things I've done or seen done with hammers: So don't knock the hammer. Indeed it doesn't have the complexity of a programming language, but the variety of ways it can be used, number of varients that can be chosen for different purposes, and beauty of a well-designed one makes it at least comprable to a hash table, regexp engine, or the like.

12 December 2007

David Welton: Programming languages are not like hand tools

I have seen the "right tool for the job" comment one too many times, and felt like writing something about it. Real tools are pretty simple, really. Things like saws and hammers have been around for thousands of years, and don't require a great deal of thinking to understand. Some people are masters with hand tools and create wonderful works of art with them, but the tools themselves are not complex. My programming language, Hecl, is a pretty simple one in a lot of ways (it has to be, to fit on cell phones), and at last count, the core has 7670 lines in a rough count that includes comments. That entails quite a few "moving parts" compared to even something like an electric drill. Programming languages are complex. Certainly to create, but even to use well; to understand the nooks and crannies and ramifications of the language's design is not something that is accomplished in a few days for most people. Your average programmer really doesn't know all that many languages, and even those who know many probably don't use most of them on a day to day basis. I like learning new languages, yet still wouldn't relish the idea of going through the mental task-switch required to move between too many in one day. What this means, is that when there is a job to be done, most programmers will reach for what they already know, rather than the best possible system, and that makes sense. If you know Python, why not write web software in Python, if you get asked to create something for the web? Or if you know Tcl because you create gui's with Tk, you're still likely to pick it when you need to create a little server. There are reasons why something like Erlang might make for a better server platform than Tcl, but is it that much better to justify dropping everything, learning a new language, committing to a new set of infrastructure requirements, and adding another thing to ask of people joining your organization? Most likely, it is not, unless the server you're writing is for a telephony switch that has to be on all the time, in which case, yes, maybe you do need Erlang. If, as is often the case, it wasn't something that had to be the best in the world, all that extra time to learn a new language would have been mostly "wasted" (ignoring, of course, the fact that for individuals, learning new things is almost always good, but that's not what we're talking about). Even the smart folks who hack away in languages like Haskell are fond of using what it has to offer, rather than jumping ship to something else at the first hint of the "something else" being better at their language of choice for a given task. So both Tcl and Erlang were up to the task of creating a little server. You could have used Ruby or Python as well. Java might prove a bit more verbose, but it's certainly up to the task. Any number of languages would be, of course. So it's not like there is a "right tool" and a "wrong tool". Maybe a better analogy is languages are like toolboxes, or even whole workshops, full of tools. Most of them have most of the common tools like hammers and saws, some sort of web libraries or the means to create tcp/ip servers. Where they differ is in things like the amount of tools available, the specialized tools that you get for free, the craftsmanship and ease of use of the tools in the hands of both experts and beginners... or in some cases (lisp, say), some fancy tools and supplies that make it possible to build yourself some new tools if you need to (that only you know how to use, but that's another topic). Some sets of tools really aren't very good for some jobs - you don't want to build blazing fast 3-d shooter games in PHP, just as you wouldn't want to use a regular set of screwdrivers on a delicate watch. However, even though you'd want to use a toolset that also has some specialized tools for working on the watch, you'd rather own a toolbox that came with the watch tools as well as the regular Phillips screwdrivers, because those are pretty handy around the house, and it wouldn't do to have only the watch tools for fixing common household items. The point being, languages are big investments, so those that do more, even if they're not the absolute greatest or first choice for a given task, are generally going to be more useful to more people, especially those just starting out who want a "safe" choice, a language they know will let them do a variety of things, rather than only one or two things (pound nails, say) really well.

Next.

Previous.