Search Results: "ag"

27 November 2008

Christian Perrier: How to get work done without doing anything

It's interesting to be here in Casar de Caceres for the joined i18n+QA+FTPmasters Extremadura meeting. Today, I've been sitting all day long in the biblioteca, which is used as hacklab, and just watched all great people around playing around with the concept of "tdebs". That weird beast can be described in two words as a concept of deb packages that would be able to distribute translation data in a more independent way then "regular" deb packages. Don't smash me as the concept is currently being twisted around in all directions by the nice folks here, to bring it to a great concept well suited for "side" material of our packages. To get an idea, you can read the original concept that popped up during the 2006 i18n Extremadura meeting, but got transformed drastically by ideas exchanged at Debconf 8 All this is made possible because the concept is now a very important issue for Emdebian and is strongly pushed by Neil Williams, whose nickname shall now be "MisterT", by general agreement of the folks here. What's really interesting right now is to see live discussions happening all the way round and see the concept slowly emerging from this. Something we could probably never do in regular mailing lists discussions or on IRC. We should really grant tons of rewards to the Extremadura region, and particularly out tireless C sar who again organised that meeting. It's still hard to say what will emerge from this and we willprobably need some more such live meetings to have real implementations of these tdeb beasts, but this is really how I like to see free software development going on. And now people begin to think about coding stuff and prepare patches, which is a good sign, in general.

Russell Coker: Slicehost vs Linode

Six months ago I investigated the options for Xen virtual servers [1]. I ended up receiving an offer of free hosting and not needing that, but the research was useful. There is a good range of options for Xen servers with different amounts of CPU power, RAM, bandwidth, and disk space. There are a couple of things that seem to be missing, options to upgrade from virtual servers to physical servers, and information on dedicated disk and database performance - but I’ll explain that later after some history. About a week ago a client needed a Xen virtual server in a hurry, their main server (a Xen system that I run on hardware that they rent) was getting a bit overloaded and needed to have one of the largest DomUs moved off. I ended up recommending Linode [2] based on my research and comments I received. The Linode server is working quite well and the client is happy, one nice feature of Linode is the choice of server rooms that they offer. I was able to choose a room in the same region as the other servers that the client owns and thus get ping times that are sometimes less than 2ms! Due to a missing feature in a program that I’m maintaining for the client a large number of MySQL queries are being made. Due to a problem I’m having with MySQL it won’t let me create a slave database server so all the queries go over the VPN and use a large amount of data. This combined with the other traffic that should be going over that link means that about 600G per month is being used, fortunately that is rather cheap. Linode staff handled this very well, after the server had exceeded it’s quota by 120G they asked my client to confirm that the traffic was legitimate and then suggested an upgrade to a plan that could handle the traffic (which went smoothly). Now I have another week to add the feature in question before I meet the quota again. Shortly after getting the new virtual server running at full capacity David Welton wrote a detailed review of Linode and Slicehost for the issues that matter to his use [3]. His conclusion seems strongly in favor of Linode. But now I am looking at getting a Slicehost [4] virtual server for the same client (for a different project) because Slicehost is owned by Rackspace [5], and the new project if successful will need a set of powerful servers and Rackspace seems like a reasonable company to host that. The problem with Rackspace is that they (and every other ISP I’ve researched so far) seems to offer little in regard to customers who need serious disk IO. I am planning some servers that will have a write bottleneck on a MySQL database (or maybe multiple shards), so serious disk capacity is needed. At least I would like to be able to get disk storage by the tray (12-14 disks) with the controllers having RAID-6 support. Rackspace only offers RAID-5 (according to the “livechat” person), and we didn’t get as far as discussing how to add more trays. What would be ideal is if there was an ISP that had both virtual servers and physical servers (so I could start with a virtual server and move to a physical server when things are working well), and also serious storage options. They would offer internal disks, external RAID arrays, and NetApp Filers [6] (or some equivalent device). It would be really nice if I could just instruct the ISP to add another NetApp Filer to my back-end network and have it done for me (I’m certainly not going to visit the US to install new hardware). It’s been over a day since I submitted a sales request to NetApp asking whether they partner with any ISPs and I haven’t received a response. OpenSolaris with ZFS also sounds good for disk IO performance (they have similar features to NetApp). Unfortunately the support for OpenSolaris among ISPs is not that great (while everyone offers Linux and Windows), and I haven’t used any recent version of Solaris. So using OpenSolaris would require finding someone with the skills to manage it who can work for my client - as opposed to a NetApp device that would be just like any other NFS server, SAN, or iSCSI server. But I’m not ruling OpenSolaris out, if someone knows of a good ISP that hosts OpenSolaris machines and supports adding dozens of disks and decent amounts of NVRAM for ZFS then I would be interested to investigate it. Joyent has some interesting OpenSolaris virtual server plans [7], they are a little pricey but offer large amounts of data transfer. They don’t provide any information on disk IO capacity (other than saying that they use ZFS for good performance). I’ve just downloaded Nexenta (Debian with the OpenSolaris kernel) [8] and will test it out over the next few days. One of the reasons I’m tending towards Rackspace at the moment (with Slicehost as the entry point) is that they seem cooperative to customer requests. My discussions with them (on a web based “livechat” and on the phone) have indicated that they may be able to do something special for me.

Eric Warmenhoven: cado, cadere, cecidi, ?

The other day I was looking at the statistics collected from my web server logs, and saw that easily more than 80% of people who find my website are looking for my Latin vocab learning tools. They’re pretty basic, but I found them to be somewhat helpful when I was learning Latin. I’m seriously considering expanding it a bit, adding conjugation/declension drills, making it a little prettier, etc. Anyway, tonight, I got a nice email from a woman who uses the vocab tools, and noticed that there was a problem with fourth principal part of the Latin verb ‘cado’, and it sent me on a somewhat fruitless search for what the right word is. (A quick primer for those not familiar with Latin verbs: most verbs have four principal parts, which can be used for conjugation. The fourth principal part is the perfect passive participle, e.g. “to have been heard”.) So then, what is the perfect passive participle of ‘cado’, which means “to fall”? Before I get into that, consider for a moment what the translation to English would be. “To fall” is usually used as an intransitive verb (the transitive form is mostly obsolete), so “to have been fallen” doesn’t really sound natural. But the translation can be approximated to something like “to have been dropped” or “to have been struck down”. But what is the actual word that is used? On this there appears to be some disagreement. Wheelock’s Latin says that it’s ‘casurum‘, as does Wiktionary. However, Tyro’s Dictionary says that it’s ‘casum‘, as does All Verbs. Meanwhile, Whitaker’s Words says that ‘casurum‘ is “very rare”, from “later” Latin, and used in the Vulgate Bible, and says it should actually be ‘casusus‘. I still don’t know which one is correct, and suspect that all three are correct depending on the text that’s used. I can somewhat see how ‘casum‘ might have been correct at some point, but changed as it overlaps with the noun ‘casus‘, which is the noun “fall”, as in “that was a nasty fall”. Or vice versa. I tried to go searching through prose to see which was most commonly used, but most of what I found was from the Vulgate Bible and used ‘casurum‘. So, I’m still not sure which is actually the “best” word. I’ve changed the site so that it uses ‘casurum‘ to be consistent with Wheelock’s. So, does anyone definitively know, which is actually best, and what the etymology is?

26 November 2008

Uwe Hermann: jhead - List and modify EXIF fields in JPEG photos

jhead is a very nice and very powerful command line utility to mess with JPEG headers (esp. EXIF fields).
  $ apt-get install jhead
It can display/extract a great amount of metadata fields from JPEG files and also extract the thumbnails stored in JPEG files (if any). The following will list all known metadata fields from a sample photo:
  $ wget http://farm4.static.flickr.com/3173/3061542361_60acb0904b_o.jpg
  $ jhead *.jpg
  File name    : 3061542361_60acb0904b_o.jpg
  File size    : 1074172 bytes
  File date    : 2008:11:26 23:38:04
  Camera make  : Panasonic
  Camera model : DMC-FZ18
  Date/Time    : 2008:03:05 15:45:52
  Resolution   : 3264 x 2448
  Flash used   : No
  Focal length : 4.6mm  (35mm equivalent: 28mm)
  Exposure time: 0.0100 s  (1/100)
  Aperture     : f/3.6
  ISO equiv.   : 100
  Whitebalance : Auto
  Metering Mode: matrix
  Exposure     : program (auto)
  GPS Latitude : N %:.7fd %;.8fm %;.8fs
  GPS Longitude: E %;.8fd %:.7fm %;.8fs
  GPS Altitude : 174.00m
  Comment      : Aufgenommen auf dem <a href="http://www.froutes.de/TT00000014_Ars_Natura">Kunstweg Ars Natura</a>.
  ======= IPTC data: =======
  Record vers.  : 4
  Headline      : Felsburg auf dem Felsberg
  (C)Notice     : www.froutes.de
  Caption       : Aufgenommen auf dem <a href="http://www.froutes.de/TT00000014_Ars_Natura">Kunstweg Ars Natura</a>.
As you can see there's a huge amount of potentially privacy-sensitive metadata in your typical JPEG as generated by your camera (including camera type, settings, date/time, maybe even GPS coordinates of your location, etc). You can extract the thumbnail stored in all JPEGs in the current directory with:
  $ jhead -st "&i_t.jpg" *.jpg
  Created: '3061542361_60acb0904b_o.jpg_t.jpg'
Random flickr image and its differing thumbnail Note that the JPEG thumbnail does not necessarily show the same picture as the JPEG itself. Depending on the image manipulation software that was used to create the edited/fixed/cropped JPEG, the thumbnail may still reflect the original JPEG contents (see sample image on the right-hand side). This is a huge potential privacy issue. There have been a number of articles about this some years ago, in case you missed them: Thus, an important jhead command line to know is the following, which removes all metadata (including any thumbnails) from all JPEG images in the current directory:
  $ jhead -purejpg *.jpg
  Modified: 3061542361_60acb0904b_o.jpg
As you can see the result is that only very basic information can be gathered from the file afterwards:
  $ jhead *.jpg
  File name    : 3061542361_60acb0904b_o.jpg
  File size    : 1052506 bytes
  File date    : 2008:11:26 23:38:04
  Resolution   : 3264 x 2448
  $ jhead -st "&i_t.jpg" *.jpg
  Image contains no thumbnail
I recommend doing this for most photos you make publically available on sites like flickr etc. (unless you have a good reason not to). Finally, see the jhead(1) manpage for lots more options that the tool supports.

Steve Kemp: Have you been following that man?

meta-hacking I've had a lot of fun over the past few years detecting and fixing XSS attacks - a few months ago compromising several thousand user-accounts belonging to a particular niche social networking site and then more recently experimenting with XSS issues upon a popular software developer's advocate blog. One thing I've been wondering about recently is meta-XSS attacks. Consider the LKML (linux kernel mailing list). This list receives lots of long patches, submitted by email, which are copied verbatum to various sites. For example if I mailed an interesting patch to LKML chances are it would get posted to: (Obviously the challenge here is to make a patch sufficiently interesting that it received more than usual coverage.) Do each of those sites HTML-encode patches? In general they do, certainly the ones I looked at had code like this:
#include &lt;linux.h&gt;
...
...
But I'm certain that not all sites do so. I'm also pretty sure there are interesting avenues to explore here, and the general idea of indirectly attacking a specific target is ripe for exploration. Anyway I'm probably not the person to go playing in the field these days; I don't have the time. But it is certainly interesting to think about. ObFilm: Dirty Harry

Russell Coker: Bill Joy

Some time ago Bill Joy (who is famous among other things for being a co-founder of Sun) [1] wrote an article for Wired magazine titled “Why the future doesn’t need us” [2]. He wrote many sensible things but unfortunately focussed on the negative issues and didn’t receive a good response. On reading it today I thought more highly of the article than I did in 2000 when it was printed, largely due to having done some background research on the topic. I’ve recently been reading Accelerating Future [3] which has a more positive approach. Now a talk by Bill Joy from 2006 has been published on the TED.com web site [4]. He starts by talking about “super empowered individuals” re-creating the 1918 flu. He also claims that “more technology super-empowers people more“. Such claims seem to be excessively hyped, I would be interested to see comments from someone who has a good knowledge of current bio-technology as to the merits of those claims. He talks briefly about politics and has some good points such as “the bargain that gives us civilisation is the bargain to not use power” and “we can’t give up the rule of law to fight an asymmetric threat - which is what we seem to be doing“. He mentions Moore’s law and suggests that a computer costing $1000 then (2006) might cost $10 in 2020. He seems to be forgetting the cost of the keyboard and other mechanical parts. I can imagine a high-end CPU which cost about $800 a couple of years ago being replaced by a $2 CPU in 2020, but I don’t expect a decent keyboard to be that cheap any time before we get fully automated nano-factories (which is an entirely separate issue). Even the PSU (which has a significant amount of government regulations for safety reasons) will have a floor cost that is a good portion of $10. Incidentally the keyboard on my EeePC 701 sucks badly, I guess I’m spoiled by the series of Thinkpad keyboards that I keep getting under warranty (which would cost me a moderate amount of money if I had to pay every time I wore one out). I will make a specific prediction, that by 2015 one of the better keyboards will comprise a significant portion of the entire cost of a computer system (more than a low end computer unit) - such that in some reasonable configurations the keyboard will be the most expensive part. It would be good if PCs could be designed to use external PSUs (such as the Compaq Evo models that took laptop PSUs). Then the PSU, keyboard, and monitor would be optional extras thus giving a small base price. Given that well built PSUs and keyboards tend not to wear out as fast as people want to replace computers, it seems that financial savings could be provided to most customers by allowing them to purchase the computer unit without the extra parts. People like me who type enough to regularly wear out keyboards and who keep using computers for more than 5 years because they still work are in a small minority and would of course be able to buy the same bundle of computer, PSU, and keyboard that new users would get. In 2020 a device the size of an iPaQ H39xx with USB (for keyboard and mouse) and the new DisplayPort [5] digital display interface (that is used in recent Lenovo laptops [6]) would make a great PDA/desktop. You could dock a PDA the way some people dock laptops now and carry all your data around with you. Bill cites an example of running the Mac interface on an Apple ][ (does anyone know of a reference for this) as an example of older hardware being more effective with newer software. It's a pity that often new software needs such powerful hardware. EG it's only recently that hardware developments have overtaken the developments of OpenOffice to make it deliver decent performance. He has an interesting idea of using insurance companies to replace government regulation of food and drugs. The general concept is that if you can convince an insurance company that your new drug is not a great risk to them so that they charge a premium you can afford then you could sell it without any further testing. Normally I'm in favor of government regulation of such things, but given the abject failures of the US government this idea has some merit. Of course there's nothing stopping insurance companies from just taking a chance and running up debts that they could never hope to repay (in a similar manner to so many banks). Finally I think it's interesting to note the camera work that the TED people used. My experience of being in the audience for many lectures (including one of Bill Joy's lectures in the late 90's) is that a speaker who consults their notes as often as Bill does gives a negative impression to the audience. Note that I'm not criticising Bill in this regard, often a great talk requires some significant notes - very few people can deliver a talk in a convincing manner entirely from memory. It seems to me that the choices of camera angle are designed to give a better impression than someone who was seated in the audience might receive - there's no reason why a video of a talk should be spoiled by seeing the top of the speaker's head while they consult their notes!

Gunnar Wolf: Familar poetry

I love it when a lack-of-humor and lack-of-appropriateness-originated flamewar causes somebody to point me towards a very nice display of intelligent humor. Specially when it is so close to me, to my roots, to my family and my personal history. FWIW, for several years, while I was a BBS user, I used WereWolf as my nickname. Great thanks to Frank K ster - and, of course, to Christian Morgenstern.
The Werewolf - English translation by Alexander Gross A Werewolf, troubled by his name,
Left wife and brood one night and came
To a hidden graveyard to enlist
The aid of a long-dead philologist. "Oh sage, wake up, please don't berate me,"
He howled sadly, "Just conjugate me."
The seer arose a bit unsteady
Yawned twice, wheezed once, and then was ready. "Well, 'Werewolf' is your plural past,
While 'Waswolf' is singularly cast:
There's 'Amwolf' too, the present tense,
And 'Iswolf,' 'Arewolf' in this same sense." "I know that--I'm no mental cripple--
The future form and participle
Are what I crave," the beast replied.
The scholar paused--again he tried: "A 'Will-be-wolf?' It's just too long:
'Shall-be-wolf?' 'Has-been-wolf?' Utterly wrong!
Such words are wounds beyond all suture--
I'm sorry, but you have no future." The Werewolf knew better--his sons still slept
At home, and homewards now he crept,
Happy, humble, without apology
For such folly of philology.

Biella Coleman: Hacking Spaces, the Spaces of Hacking

Space and place has long been important to hackers. Whether it was/is the university lab, the workplace, the hacker con, or the particularly high-tech city, hackers congregate and meet face to face, often and everyday. One semi-new development has been the explosion and proliferation of hacker labs and spaces, such as Noisebridge located in the Mission district of San Francisco Foulab in Montreal. I recently got back from San Francisco and was able to spend a few nights at Noisebridge and was jaw droppingly impressed. The space is, well, spacious and nice (and located right by the Bart, a real +++++), but more important is that it is a thriving collective with all sorts of geeky participants and they have just souped up the space with all sorts of equipment, from the usual suspects (lots of computers) to lots of electronic gear such as oscilloscopes. What I also was impressed by was not only the blizzard of events but the open and accessible nature of the organization, which seemed to sit in some contrast to NYC’s hacker collective, NYC Resistor. Like so many organizations in this metropolis, they apparently are lacking in space to grow and the word on the streets (which I cannot confirm or deny as I have had very little contact with them but have heard this repeatedly) is that the organization has had a tough time letting in new members. Some folks are understanding of this given their space limitation, others have been less kind, and have referred to the group as a clique. There are already a few other initiatives under way to find a larger space so as to accommodate a more open, participatory atmosphere for a hacker space but I imagine that over time the culture and developments of NYC Resistor will also change as new spaces develops and do hope that this creates the conditions for more access rather than less (and again, I know next to nothing of the situation though I suspect space plays a real limiting role as it does with nearly everything else in NYC). And thankfully Rose White, a NYC-based sociology graduate student at CUNY, is paying close attention to the rise and development of these hacker spaces. She is well underway doing her dissertation dissertation on these hack spaces and I really look forward to her work.

Steve Kemp: If there isn't a movie about it, it's not worth knowing, is it?

So, I've just got a portable machine. I've configured it to be a pretty minimal installation of Debian Lenny, but one thing that makes me unhappy is mail handling. By default it came with several exim4- packages. Now in general Exim4 rocks. But it is a deamon running all the time, and overhead I could live without. I looked around to find a mail transport agent that would be more suited to the machine and was suprised to find nothing suitable. Basically I figure the machine will never generate "real" emails. Instead it will only receive mails from cron, etc. The machine will never have a real fixed IP, and so relaying mail externally is a waste of time. The mail should just go somewhere predictable and local. There are a couple of lightweight agents which will forward to another system, but nothing seems to exist which will queue mail locally only. So I've hacked a simple script which will do the job. Given the spool director /var/spool/skxmail the following command:
skxmail root < /etc/passwd
Produces this:
/var/spool/skxmail/
 -- root
     -- cur
     -- new
         -- 1227702470.P8218M243303Q22.vain.my.flat
     -- tmp
4 directories, 1 file
That seems to be sufficient for my needs. (I support the flag which says "read the receipient from the body). Of course to do this properly I'd be setgid(mailgroup). Instead I assume that local means everybody can see it and /var/spool/skxmail is mode 777. Ooops. Still happy to share if it sounds interesting. ObFilm: Dogma

Russell Coker: Flash Storage and Servers

In the comments on my post about the Dell PowerEdge T105 server [1] there is some discussion of the internal USB port (which allows the use of a USB flash device for booting which is connected inside the case). This is a really nice feature of the Dell server and something that would be useful if it was in more machines. However I believe that it would be better to have flash storage with a SATA interface on the motherboard. The cost of medium size flash storage (eg 4G) in the USB format is not overly great, if soldered to the motherboard or connected to a daughter-board the incremental price for the server would be very small. Dell servers shipped with a minimum of an 80G SATA disk last time I checked, it seems quite likely to me that Dell could reduce the prices of their servers by providing flash storage on the motherboard and having no hard disk. It seems likely to me that there is a significant number of people who don’t want the default hard drive that ships with a Dell server. The 80G disk that came with my PowerEdge is currently gathering dust on a shelf, it was far to small to be of any use in that machine and Dell’s prices for bigger disks were outrageous so I replaced the default disk with a pair of big disks as soon as the server had passed some basic burn-in tests. Most servers that I run fall into one of two categories, machines which primarily do computation tasks and need little storage space or IO capacity (in which case 4G of flash would do nicely) and machines which have databases, backups, virtual machine images, and other big things (in which case anything less than 160G is silly and less than 500G makes no economic sense in today’s market). Note that for a machine with small storage requirements I would rather have a 4G flash device than an 80G disk, I am inclined to trust flash to not die but not trust a single disk, two 80G disks means more noise, heat dissipation, and expense. According to comments on my previous post VMWare ESX requires a USB boot device, so if VMWare could be booted with a motherboard based flash device then that would be an ideal configuration for VMWare. In some mailing list discussions I’ve seen concern raised about the reliability of permanently connected USB devices, while I’ve only encountered USB problems related to buggy hardware and drivers other people have had problems with the electrical connection. So it seems that motherboard based flash could be expected to increase the reliability of VMWare servers. The down-side to having flash permanently attached to the motherboard is of course the impossibility of moving the boot device to different hardware. In terms of recovering from failure restoring a few gig of flash storage from backup is easy enough. The common debugging option of connecting a hard drive to another machine to fix boot problems would be missed, but I think that the positive aspects of this idea outweigh the negative - and it would of course be an option to not boot from flash. If anyone knows of a tower server that is reasonably quiet, has ECC RAM and a usable amount of flash storage on the motherboard (2G would be a bare minimum, 4G or 8G would be preferred) then please let me know.

25 November 2008

John Goerzen: Why Do Web Applications Stink So Badly?

So today, I happen to be looking at wikis for two small to mid-sized public proojects (MoinMoin and DokuWiki look like frontrunners right now — any suggestions?) Recently, I’ve also looked at blog and CMS software, and a host of other web apps. It’s as if these people have learned nothing about good software practices over the last 20 years. Warning: Rant Ahead So how many of you have been here before? You download WebApp X. It tells you to cd to your DocumentRoot and unzip/untar it there. At this point, most of them will tell you to chmod -R 777 the install directory. Some of the better ones, such as WordPress, will tell you to chmod it 777, or if that makes you nervous, to instead chown it to the user that your webserver runs as. It is at this point that you realize that the Java-based programs ship with their own webserver that takes 2 minutes to load and uses 2GB of RAM, while the PHP-based programs want you to give them 32MB RAM per process, and probably modify your global PHP settings in a way that breaks some other PHP web app you’re already using. As if that isn’t enough to scare you off, generally speaking, config files — including passwords to databases — are stored in the same directory, along with .htaccess files. Many of these programs are also downloading and updating plugins over the Internet, usually without any kind of cryptographic authentication, and overwriting their own program files in the process. Oh, and this is a class of app that is notorious for security problems to start with, and makes your server known to billions of people via search engines. Absolutely no opportunity for trouble here, of course! That sentence was dripping with sarcasm, in case you didn’t get it. It also makes it almost impossible for people such as Debian maintainers to package up some webapps (such as just about every single one that uses Ruby on Rails) because there is just no sane way to make it behave with respect to the Filesystem Hierarchy Standard. I’d love to see web app developers do a few simple things:
  1. Separate code from data
  2. Separate code from configuration
  3. Separate all of the above from the DocumentRoot to the greatest extent possible
I realize that some of this is purportedly to make things easier to install when you have FTP access only. But to me it seems just really poor design. I’ve written webapps, and it’s not that hard to do this part right. Plus, doing the above right means that I no longer have to do something like use git on my WordPress installations because it’s too much of a hassle to apply security and plugin updates on all three separate ones otherwise.

Ian Wienand: Position Independent Code and x86-64 libraries

If you've ever tried to link non-position independent code into a shared library on x86-64, you should have seen a fairly cryptic error about invalid relocations and missing symbols. Hopefully this will clear it up a little! Let's start with a small program to illustrate.
$ cat function.c
int global = 100;
int function(int i)  
	return i + global;
 
$ gcc -c function.c
Firstly, inspect the disassembley of this function:
0000000000000000 <function>:
   0:	55                   	push   %rbp
   1:	48 89 e5             	mov    %rsp,%rbp
   4:	89 7d fc             	mov    %edi,-0x4(%rbp)
   7:	8b 05 00 00 00 00    	mov    0x0(%rip),%eax        # d <function+0xd>
   d:	03 45 fc             	add    -0x4(%rbp),%eax
  10:	c9                   	leaveq
  11:	c3                   	retq
Lets just go through that for clarity: The IP relative move is really the trick here. We know from the code that it has to move the value of the global variable here. The zero value is simply a place holder - the compiler currently does not determine the address (i.e. how far away from the instruction pointer the memory holding the global variable is. It leaves behind a relocation -- a note that says to the linker "you should determine the correct address of foo, and then patch this address of the code to point to that addresss (i.e. foo)." Relocations with addend The image above gives some idea of how it works. We can examine relocations with the readelf tool.
$ readelf --relocs ./function.o
Relocation section '.rela.text' at offset 0x518 contains 1 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000009  000800000002 R_X86_64_PC32     0000000000000000 global + fffffffffffffffc
If you try and build a shared object (dynamic library) with this function, you should get something like
$ gcc -shared function.c
/usr/bin/ld: /tmp/ccQ2ttcT.o: relocation R_X86_64_32 against  a local symbol' can not be used when making a shared object; recompile with -fPIC
/tmp/ccQ2ttcT.o: could not read symbols: Bad value
collect2: ld returned 1 exit status
Position Independent Code (PIC, enabled with -fPIC) just means that the output binary does not expect to be loaded at a particular base address, but is happy being put anywhere in memory (compare the output of readelf --segments on a binary such as /bin/ls to that of any shared library). This is obviously critical for implementing shared-libraries, where you may have many, many libraries loaded in essentially any order, and trying to pre-allocate where in memory they would all live just does not work. However, PIC has slightly wider implications than simply movable base addresses. So, back to relocations. The exact rules for different relocation types are described in the ABI for the architecture. The R_X86_64_PC32 relocation is defined as "the base of the section the symbol is within, plus the symbol value, plus the addend". The addend makes it look more tricky than it is; remember that when an instruction is executing the instruction pointer points to the next instruction to be executed. Therefore, to correctly find the data relative to the instruction pointer, we need to subtract the extra. This can be seen more clearly when layed out in a linear fashion (as in the bottom of the above diagram). What's the problem with this in a shared library? In a shared library situation, we can not depend on the local value of global actually being the one we want. Consider the following example, where we override the value of global with a LD_PRELOAD library.
$ cat function.c
int global = 100;
int function(int i)  
	return i + global;
 
$ gcc -fPIC -shared -o libfunction.so function.c
$ cat preload.c
int global = 200;
$ gcc -shared preload.c -o libpreload.so
$ cat program.c
#include <stdio.h>
int function(int i);
int main(void)  
   printf("%d\n", function(10));
 
$ gcc -L. -lfunction program.c -o program
$ LD_LIBRARY_PATH=. ./program
110
$ LD_PRELOAD=libpreload.so LD_LIBRARY_PATH=. ./program
210
If the code in libfunction.so has a fixed offset into its own data section, it will not be able to see the overridden value provided by libpreload.so. This is not the case when building a stand-alone executable, where references are satisfied internally. Of course, any problem in computer science can be solved with a layer of abstraction, and that is what is done when compiling with -fPIC. To examine this case, let's see what happens with PIC turned on.
$ gcc -fPIC -shared -c  function.c
$ objdump --disassemble ./function.o
./function.o:     file format elf64-x86-64
Disassembly of section .text:
0000000000000000 <function>:
   0:	55                   	push   %rbp
   1:	48 89 e5             	mov    %rsp,%rbp
   4:	89 7d fc             	mov    %edi,-0x4(%rbp)
   7:	48 8b 05 00 00 00 00 	mov    0x0(%rip),%rax        # e <function+0xe>
   e:	8b 00                	mov    (%rax),%eax
  10:	03 45 fc             	add    -0x4(%rbp),%eax
  13:	c9                   	leaveq
  14:	c3                   	retq
It's almost the same! We setup the frame pointer with the first two instructions as before. We push the first argument into memory in the pre-allocated "red-zone" as before. Then, however, we do an IP relative load of an address into rax. Next we de-reference this into eax (e.g. eax = *rax in C) before adding the incoming argument to it and returning.
$ readelf --relocs ./function.o
Relocation section '.rela.text' at offset 0x550 contains 1 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000000000a  000800000009 R_X86_64_GOTPCREL 0000000000000000 global + fffffffffffffffc
The magic here is again in the relocations. Notice this time we have a P_X86_64_GOTPCREL relocation. This says "replace the data at offset 0xa with the global offset table (GOT) entry of global. Global Offset Table operation with data variables As shown above, the GOT ensures the abstraction required so symbols can be diverted as expected. Each entry is essentially a pointer to the real data (hence the extra dereference in the code above). Since the GOT is at a fixed offset from the program code, it can use an IP relative address to gain access to the table entries. This extra reference is obviously slower; however for the most part I imagine the overhead would be essentially immeasurable and is required for "generic" operation. If you have figured the cost of indirection through the GOT is the major bottleneck of your program, I imagine you wouldn't be reading this and would already be considering strategies to remove it! The next question is why this works on plain old x86-32. Inspecting the code reveals why:
$ objdump --disassemble ./function.o
00000000 <function>:
   0:	55                   	push   %ebp
   1:	89 e5                	mov    %esp,%ebp
   3:	a1 00 00 00 00       	mov    0x0,%eax
   8:	03 45 08             	add    0x8(%ebp),%eax
   b:	5d                   	pop    %ebp
   c:	c3                   	ret
$ readelf --relocs ./function.o
Relocation section '.rel.text' at offset 0x2ec contains 1 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00000004  00000701 R_386_32          00000000   global
We start out the same, with the first two instructions setting up the frame pointer. However, next we load a memory value into eax -- as we can see from the relocation information, the address of global. Next we add the incoming argument from the stack (0x8(%ebp)) to the value in this memory location; implicitly dereferencing it. This provides the abstraction we need -- if the relocation makes the patched address at 0x4 the address of the GOT entry, it will be correctly dereferenced. It is the inability of the x86-32 architecture to try and optimise by doing instruction-pointer relative offseting which means it always needs to do slower memory references, which turns out to be just what you want when you're making a shared library! So, the executive summary: the ability of x86-64 to use instruction-pointer relative offsetting to data addresses is a nice optimisation, but in a shared-library situation assumptions about the relative location of data are invalid and can not be used. In this case, access to global data (i.e. anything that might be changed around on you) must go through a layer of abstraction, namely the global offset table.

Holger Levsen: awesome bts and snow

Right now, I'm a happy hacker: I just added a keybinding and shell oneliner consisting of four piped commands finally invoking the bts command to my awesome configuration. So now I can press Mod+F4 (anywhere, anytime) and then either enter a bug number or get a tab completion list of bugs I already looked at. Totally awesome. And I guess there is room for improvements once I have used /usr/bin/bts more :)

In related news I'm currently flying from Z rich (which made me think about snow...) to Madrid, in anticipation of the Debian QA/i18n/ftpteam meeting, which will start tomorrow evening with a bus drive to Caceres, the town where DebConf9 will be held in 2009. And somehow I didnt feel like going through my mails, I rather wanted to do some interesting new stuff. So I started to read the bts manpage...

To bring this post to and end, I share Gunnars feeling about all these snow blog posts, you insensitive colds! Last friday morning, Hamburg was among the first cities in Germany getting snow (the cold was coming from Greenland, thus from north west and Hamburg is pretty much north-north-west in Germany). Unfortunatly, the snow was almost gone in the city by friday noon, so when all these "got lots of snow" blog posts popped up on planet, I felt a bit like Gunnar...

Joost Yervante Damad: first snow (meme)

Since everybody is posting pictures of the first snow of the year, I can't stay behind. Quickly popped out this morning to make this snap of my favorite habitat, the valley of the "Grote Nete":


(click on the image to enlarge)

nikon d300 tamron 17-50 f/2.8 2008 Joost Yervante Damad



Isaac Clerencia: The coolest IP on the net

My workmate Josh just discovered this :P isaac@cooper:~
% host 69.69.69.69
69.69.69.69.in-addr.arpa domain name pointer the-coolest-ip-on-the-net.com.

Joachim Breitner: Linux Journal on the Neo FreeRunner

A fellow lodger of my student dormatory just popped in and gave me the December issue of Linux Journal as a gift from his trip to the US (thanks for that!). When I skimmed the front page, I immediatelly noticed the line Reviewed: OpenMoko s Neo FreeRunner. The four page article by Cory Wright contains some general notes on the FreeRunner, a list of things he doesn t like and to my surprise installation instructions for Debian on the FreeRunner! It s nice to see one s work used in such public places. I especially liked the line I must say, the Debian installers sure have improved a lot since the Potato days. (Of course, the installer for the FreeRunner is mostly a dumb script that can not be compared to the great official debian-installer that you can use on a PC.)

Bastian Venthur: Lenovo T500 (shiny new laptop)

Yesterday I exchanged my “old” Thinkpad T60 with a shiny new T500. This time I ordered a Thinkpad with integrated Intel graphics since it’s supposed to be way better supported than the graphic chips from ATI or NVIDIA under Linux. And of course it is supposed to be much more silent. I don’t regret this decision so far, the new T500 is barely hearable even under heavy load. Debian I installed Debian unstable (aka. frozen for waaayy to long) without a problem. Sound worked out of the box, graphics worked somewhat. The kernel automatically loaded the correct modules but I had to tell Xorg that I want the “intel” driver instead of Vesa. Unfortunately Xorg’s intel driver in unstable is already too old to allow me to use an external monitor via the docking station’s DVI port, so I had to to upgrade to Xorg from experimental, which is not very cool since this is the machine I depend on at work. Wifi also doesn’t work, since the the driver and firmware available in unstable are outdated. Suspend to RAM works sometimes, but most of the time the laptop refuses to wake up properly so I have to shut it down. I’ve not yet tested if bluetooth and Wimax works. So currently the T500 is usable under unstable but without external monitor and without Wifi. If you are brave enough to use packages from experimental, you get your external monitor to work but still don’t have Wifi. Suspend to RAM does not work reliably. The T500 should probably work fine with Lenny but of course without external Monitor and Wifi support. Vista After I’ve installed Debian and moved my /home from the old laptop to my new baby, I noticed that silly “Windows Vista” sticker on my laptop. Of course I tried to remove that thing, which I regretted painfully. After I’ve successfully removed the sticker — which was surprisingly hard — some glue remained on the surface of the laptop. No problem I thought, nothing some nail polish remover can’t handle. Unfortunately it only smeared the glue but did not resolve it as I had expected. So I’ve tried perfume, water, soap, rubber, tape and solvent for several hours, but was not able to remove the glue at all. All I achieved was to smear the compact but thick chunk of glue to a very thin but much more distributed layer of this nasty glue on my laptop. Now it looks like a 5×5cm square of dirt on top of my laptop — not sure if removing the Vista sticker was worth it.

Russell Coker: SE Linux and Decrypted Data

There is currently a discussion on the Debian-security mailing list about how to protect data which came from an encrypted file. I was going to skip that one until someone summoned me by mentioning SE Linux. The issue which was raised is that data from an encrypted file can be read from /dev/mem (for all memory of the machine) or /proc/<pid>/mem (for the memory of the process). It was suggested that SE Linux can prevent such attacks, however it’s not that simple. In most SE Linux configurations there will be a domain with ultimate privileges, examples are unconfined_t for a default configuration (also known as targeted) and sysadm_t for a “strict” configuration. SE Linux differs from many security models (such as the Unix permissions model that we are familiar with) in that there is no inherent requirement for a domain with ultimate privileges. I have in the past written policy for machines in which no domain could read /dev/mem or modify the running policy. This meant that booting from installation media might be required to modify the configuration of the system - some people desire systems like this and have good reasons for doing so! As the vast majority of SE Linux users will have a “targeted” configuration, and the majority of the rest will have a “strict” configuration with minimal changes (IE sysadm_t will work as expected) there will always be a domain that can access /dev/mem (I’m not certain that sysadm_t can directly access /dev/mem on all variants of the policy - but given that it can put SE Linux in permissive mode it has ultimate access to work around that). This however doesn’t mean that SE Linux provides no benefit. A typical Linux system will have many daemons running as root, while a typical SE Linux system will have few processes running as root while also having a SE Linux context that grants ultimate privileges. The root owned processes that SE Linux constrains are generally the network facing daemons and other processes which provide a risk, while the root owned processes that are not constrained by SE Linux policy are generally processes related to early stages of the system boot and programs run as part of the system administrator’s login session - processes that are inherently trusted. Now regarding the memory of the process (and the ptrace API), in most cases a process that is likely to be accessing such a file will run in a domain that permits such access to itself. If the process in question is run from a login session (ssh login etc) then other processes in the same session will have access to the data. It is quite possible to write policy to create a new domain for the program which accesses the decrypted data and to deny the domain the ability to ptrace itself, but it will require at least 10 minutes of work (it’s not a default configuration). So while SE Linux will generally help you to improve your security no matter what your exact requirements may be, it’s not something that can be used to make such corner cases entirely disappear. In a more general sense the idea that only a single program can access certain data is almost unattainable in a typical system. Among other things to consider is the fact that the Linux kernel is one of the larger programs on a typical Linux system…

Wouter Verhelst: New hackergochi

... again. This must be my third, in about as many years. The picture from which I made it was taken by Andrew, with my camera. I kindof like it; and I think the hackergochi came out well, too. No, I'm not planning on changing my hackergochi every year, even though it's turning out to be the case. Oh well. (Speaking of hackergochi: If people on Planet Grep have a hackergochi or something similar that they'd like to get associated with their posts, please send it to me. Now, preferably :-)

24 November 2008

Tollef Fog Heen: How to handle reference material in a VCS?

I tend to have a bunch of reference material stored in my home directory. Everything from RFCs, which is trivial to get at again using a quick rsync command (but immensely useful when I want to look up something and am not online) to requirements specifications for systems I made years and years ago. If I didn't use a VCS, I would just store those in a directory off my home directory, to be perused whenever I felt the need. Now, with a VCS controlling my ~, it feels like I should be able to get rid of those, and just look them up again if I ever need them. However, this poses a few problems, such as "how do you keep track of not only what is in a VCS, but also what has ever been there". Tools like grep doesn't work so well across time, even though git has a grep command too, it still doesn't cut it for non-ASCII data formats. Does anybody have a good solution to this problem? I can't think I'm the only one who have the need for good tools here.

Next.

Previous.