Search Results: "xam"

25 February 2009

Russ Allbery: Expectation and obligation

Some necessary background: George R.R. Martin is a science fiction and fantasy author who's been writing (and editing) in the field for quite a while. In 1996, he started a new epic fantasy series entitled The Song of Ice and Fire, a hugely complex and sprawling series telling the story of an entire medieval world, with four volumes published and something on the order of 19 viewpoint characters to date. The fourth and last book to date in the series was published in 2005. It contained only half the viewpoints expected (and in my opinion and that of at least some other readers, the least interesting half). There was an afterword saying basically that the book was cut somewhat artificially in half for length reasons (it was 753 hardcover pages for what was there), the next book would have the other viewpoints, and it was basically done. Three years and some later, and after a few other optimistic updates, it's not yet done. Some of Martin's fans are increasingly getting antsy and sometimes abusive about this. This has increasingly been getting to Martin, and he's responded in various ways in his personal blog, most recently February 19th. See also John Scalzi and Charlie Stross reacting to his post. Now, I'm not one of the people who is rabid about this series. I'm a little disappointed that I haven't gotten to read the next bits with the characters I really like in the series, and I didn't much care for A Feast for Crows because of how the book was split. But I have a thousand other books I want to read (literally I've read less than half the books I own) and if I don't get to see more of the series, oh well. I am, however, very interested in the fan reaction to this delay and Martin's reaction to that, because it has started reminding me of similar things in my own life. Some of the fans are just batshit insane, which is one of the things that happens to a best-selling author; I feel for Martin there but there isn't much anyone can really do about that. But among the sane people, there are a bunch who say something like "I understand why the constant nagging doesn't help, but on the other hand, when you start a big story with unresolved plots, you have an obligation to your readers to finish it." I call bullshit on that. This same effect happens in the free software world. If you write some program, you're perceived to have an obligation to support it, fix it, release new versions, and so forth. I've also had this happen in other volunteer contexts, such as Usenet. Often that obligation is based on some comments that one makes about future plans, things that will go into the next release, e-mail messages where you say that you'll look at something, and so forth. It's not entirely groundless, and it's not groundless with Martin either. However. As any software developer knows, time estimates for tasks are horribly difficult and almost always too short. This is clearly what happened to Martin, and bitching about the time estimates doesn't help at all. Some problems turn out to be so much harder than you expect that time estimates fail entirely. With creative work (including some programming) it's particularly hard to know how much work is remaining. You may know roughly how much you have left to write, but you don't know how hard each chunk will be until you start it, and sometimes until you finish it. With some projects, the only thing you can do is say "sorry, my estimates were completely off and I don't know how to estimate this" and then stop promising dates. Which Martin has already done. He was "wrong" in some sense for setting bad expectations, but estimation is insanely hard and the world usually doesn't let you get away with just refusing to estimate anything. So much for time. What about whether Martin has an obligation to write the next book at all? The equivalent question in the free software world is whether one can just walk away from a project, or away from a volunteer position, leaving a bunch of stuff undone that some people who were using the software were counting on having finished. It's a bit easier to deal with in free software, since usually just promises are at stake, whereas with Martin he's sold a commercial product and people get weird about money. There is a reasonable expectation that a series of this type will have a conclusion (although I'll point out that the SFF world is filled with unfinished series). Maybe you wouldn't have bought any of the series if you had known that it wouldn't end, so now you're out $100 for the hardcovers that you feel like you bought on false pretenses. Let's examine that. First, I have a serious problem with people who use the economic transaction as leverage in this sort of argument. This is a really piddly amount of money for each individual person, generally less than I get paid for one hour of work. Writing a new novel is way, way more than one hour of work. Martin's fans are demanding that he do work that's worth many orders of magnitude more than they're conceivably out monetarily. (And the reactions are individual reactions of anger; the level of upsetness of each individual shouldn't change based on the total size of the fan base.) Furthermore, their economic transaction was with the publisher, and they had no control and no say over how Martin got paid. From multiple different directions, they're not his employer, have no contractual control over how he spends his time, and have no legitimate economic basis for asserting any such control. They can make a decision to not buy his future work because they're disappointed, and that's it. Second, it is absolutely mandatory to give other people room to be wrong or change their mind. Human beings cannot keep every promise that they make. We fail at things that we fully intend to do, particularly around estimates, and particularly around one's dedication to decade-long major life projects. People are wrong about marriages and children, which involve far more serious consequences than a fiction series. I think it's a good reality check here to note that his fans are placing way more restrictions on what he can do with his life than would be legal for any employer. Third, when someone doesn't meet your expectations, the response should be proportional and rational. This is the one that really gets to me. Choosing not to read the next book in the series because it's been too long is entirely reasonable. Trying to make Martin feel like crap is not. I would go further: nothing that an artist could promise to their general public could warrant intentionally making them feel like crap for failing to do it. That level of emotional consequence should be reserved for things like marriage, parenting, or life-long friendship, not disappointment that you didn't get to hear the end of a story you were enjoying. I speak from personal experience here: this type of punishment by guilt is a nasty, nasty business. It burns people out faster than any other possible reaction, it makes it much less likely that Martin will continue the series at all, and it's probably poisoning his feelings about the entire world. It's burning down the bridge you're standing on. It always hurts and never helps. People already create feelings of obligation around things they've promised, implicitly or explicitly, to do, and generate plenty of their own guilt. That guilt is one of the major obstacles that one has to overcome to be productive; if you don't believe me, read any good productivity book, such as Allen's Getting Things Done, and notice how much it talks about getting past guilt. Intentionally going out of your way to contact someone you do not know personally and pile on more guilt is vicious, cruel, and tactically counterproductive. Trust me, Martin didn't fail to notice that he missed his own deadlines. I think Martin's post and the subsequent discussion is a great opportunity for all of us who create or use creative and technical works to think about how we apply the Golden Rule. There's a natural tendency to hold people tightly to promises that we'll get something that we want. There's a natural tendency for all of us to make those promises, since people are grateful for them, and then pile guilt on ourselves for not meeting them. All of the emotion wrapped up in both sides of that tension of expectation and obligation can be exhausting and poisonous. It makes relationships brittle because we don't let people renegotiate. We don't let people say they were wrong. We don't let people change their minds. And we don't let ourselves change our minds. Here's my promise: the next time someone doesn't meet an expectation I had, I will remember this discussion, give them space to be wrong, and not pile on more guilt. Like all promises, I'll try to keep it, but I might not be able to.

24 February 2009

Russell Coker: Tragedy and Profit

Every time something goes wrong there will be someone who tries to take advantage of the situation. The recent bushfires in Australia that have killed hundreds of people (the count is not known yet) are a good example. Pastor Nalliah of Catch the Fire Ministries [1] claims that it is due to legalising abortion. This is astoundingly wrong. In a more extreme example representatives of the Westboro Baptist Church were planning to visit Australia to launch a protest in support of the bushfires [2]. I have not yet found any news reports about whether they actually visited Australia or protested - it s most likely that they decided not to visit due to the Australian laws being very different to US laws regarding the relative importance of freedom of speech and incitement to violence. Apparently the insane Westboro Baptist Church people (who are best known for GodHatesFags.com and GodHatesAmerica.com) believe that God hates Australia and caused the fires (presumably due to Australia not persecuting homosexuals). Danny Nalliah has permanently damaged his own reputation by acting in a similar way to the Westboro Baptist Church. The reputation of Catch The Fire now depends on how quickly they get a new pastor Please note well that the vast majority of Christians have nothing in common with Westboro or Catch The Fire. I don t recall the last time I met an Australian Christian who was strongly opposed to homosexuality or abortion. Now we do have to try and investigate ways of avoiding future tragedies, and the work to do this needs to begin immediately. John Brumby (the Premier of Victoria) has announced that Victoria will get new strict building codes for fire resistant buildings [3]. There have been many anecdotes of people who claim to have been saved by attaching sprinkler systems to their homes, by building concrete bunkers to hide in while the fire passes, and using other techniques to save their home or save themselves. Some more research on the most effective ways of achieving such goals would be worthwhile, an increase in funding for the CSIRO to investigate the related issues would be a good thing. The article also has an interesting quote As the fallout from the disaster widened, the union representing the nation s 13,000 firefighters warned both the federal and state governments to take global warming seriously to prevent a repeat of last weekend s lethal firestorm . However given that traditionally Australia and the US have been the two nations most opposed to any efforts to mitigate global warming it seems unlikely that anything will change in this regard in a hurry. The attempts to link bushfires to abortion and homosexuality are offensive, but can be ignored in any remotely serious debate about politics. However there are some other groups trying to profit from the tragedy that make claims which are not as ridiculous. On the 9th of February the Australian Green party was compelled to release an official statement from Spokesperson Scott Ludlam, Sarah Hanson-Young, Rachel Siewert, Christine Milne, and Bob Brown following some political discussion about Greens policies [4]. There have been attempts to blame the Greens for the tragedy which were politically motivated, some of which came from groups that traditionally oppose the Greens for other reasons (I m not going to provide the detail - anyone who is really interested can do google searches on the people in question). On the 16th of February Bob Brown (the leader of the Green party) felt obliged to make another media release reiterating the fact that the Greens support prescribed burn-offs to limit the scope of wild fires [5], he also decried the hate mongering that has been occurring in the wake of the disaster. One of the strange memes that seems to be spread by opponents to the Greens is that the Greens are all supposedly from the city and know nothing about the country. To avoid being subject to such attack I feel obliged to note that on one of the bad fire days I visited my parents. I spent the morning with my father and some friends at a park that was not far from the fire area, my friends then returned to their home which was not far from the fire area. I then had lunch with my parents and watched the smoke through the dining room window. After that my friends didn t respond to email for a while and I was concerned that they may have lost their house or maybe suffered injury or death. I didn t know them well enough to feel it appropriate to try a dozen different ways of contacting them (I m sure that many other people were doing so), but I was rather concerned until my wife received an email from them. But I don t base my political beliefs on what I personally observe or my connections to people on the edge of the fire zone. I believe in the Green principles of Peace and Non Violence, Grassroots Democracy, Social and Economic Justice, Ecological Sustainability and the use of science and statistics to determine the best ways of achieving those goals.

Bernd Zeimetz: why you should use gpsd over gypsy

While working on the Debian packages of navit I stumbled upon gypsy, as the libgypsy is used by navit to listen to notifications about location changes via D-Bus. As maintainer of the gpsd package I started to wonder what the opinion of the gpsd developers about gypsy is - especially after reading that "Gypsy was designed to fix the numerous design flaws found in GPSD". Well, here is their answer, which I support fully. I'm blogging about it as I think it is important to know about this details if you plan to do something useful with a gps device, for example on mobile phones like the OpenMoko Freerunner. gpsd works very well on embedded devices, and if there's a feature missing just contact the developers. They're very responsive and like to implement useful features, if they can get a spec to do so (and even better, the hardware to test it). So I'm really wondering if it wouldn't make sense to replace the minimal NMEA/UBX implementation in the FSO Framework Daemon by gpsd, even if that means to compile code instead of writing Python code, which is much faster for rapid prototyping, of course.

22 February 2009

Theodore Ts'o: Should Filesystems Be Optimized for SSD s?

In one of the comments to my last blog entry, an anonymous commenter writes:
You seem to be taking a different perspective to linus on the adapting to the the disk technology front (Linus seems to against having to have the OS know about disk boundaries and having to do levelling itself)
That s an interesting question, and I figure it s worth its own top-level entry, as opposed to a reply in the comment stream. One of the interesting design questions in any OS or Computer Architecture is where the abstraction boundaries should be drawn and which side of an abstraction boundary should various operations be pushed. Linus s arguments is that there a flash controller can do a better job of wear leveling, including detecting how worn a particular flash cell might be (for example, perhaps by looking at the charge levels at an analog level and knowing when the last time the cell was programmed), and so it doesn t make sense to try to do wear leveling in a flash file system. Some responsibilities of flash management, such as coalescing newly written blocks into erase blocks to avoid write amplification can be done either on the SSD or in the file system for example, by using a log-structured file system, or some other copy-on-write file system, instead of a rewrite-in-place style file system, you can essentially solve the write amplification problem. In some cases, it s necessary let additional information leak across the abstraction for example, the ATA TRIM command is a way for the file system to let the disk know that certain blocks no longer need to be used. If too much information needs to be pushed across the abstraction, one way or another, then maybe we need to rethink whether the abstraction barrier is in the right place. In addition, if the abstraction has been around for a long time, changing it also has costs, which has to be taken into account. The 512 byte sector LBA abstraction has been around long time, and therefore dislodging it is difficult and costly. For example, the same argument which says that because the underlying hardware details are changing between different generations of SSD is all of these details should be hidden in hardware, was also used to justify something that has been a complete commercial failure for years if not decades: Object Based Disks. One of the arguments of OBD s was that the hard drive has the best knowledge of how and where to store an contiguous stream of bytes, and so perhaps filesystems should not be trying to decide where on disk an inode should be stored, but instead tell the hard drive, I have this object, which is 134 kilobytes long; please store it somewhere on the disk . At least in theory the HDD or SSD could handle all of the details of knowing the best place to store the object on the spinning magnetic media or flash media, taking into account how worn the flash is and automatically move the object around in the case of an SSD, and in the case of the HDD, the drive could know about (real) cylinder and track boundaries, and store the object in the most efficient way possible, since the drive has intimate knowledge about the low-level details of how data is stored on the disk. This theory makes a huge amount of sense; but there s only one problem. Object Based Disks have been proposed in academia and advanced R&D shops of companies like Seagate et.al. have been proposing them for over a decade, with absolutely nothing to show for it. Why? There have been two reasons proposed. One is that OBD vendors were too greedy, and tried to charge too much money for OBD s. Another explanation is that the interface abstraction for OBD s was too different, and so there wasn t enough software or file systems or OS s that could take advantage of OBD s. Both explanations undoubtedly contributed to the commercial failure of OBD s, but the question is which is the bigger reason. And the reason why it is particularly important here is because at least as far as Intel s SSD strategy is concerned, its advantage is that (modulo implementation shortcomings such as the reported internal LBA remapping table fragmentation problem and the lack of ATA TRIM support) filesystems don t need to change (much) in order to take advantage of the Intel SSD and get at least decent performance. However, if the price delta is a stronger reason for its failure, then the X25-M may be in trouble. Currently the 80GB Intel X25-M has a street price of $400, so it costs roughly $5 per gigabyte. Dumb MLC SATA SSD s are available for roughly half the cost/gigabyte (64 GB for $164). So what does the market look like 12-18 months from now? If dumb SSD s are still available at 50% of the cost of smart SSD s, it would probably be worth it to make a copy-on-write style filesystem that attempts to do the wear leveling and write amplification reduction in software. Sure, it s probably more efficient to do it in hardware, but a 2x price differential might cause people will settle for a cheaper solution even if isn t the absolutely best technical choice. On the hand, if prices drop significantly, and/or dumb SSD s completely disappear from the market, then time spent now optimizing for dumb SSD s will be completely wasted. So for Linus to make the proclamation that it s completely stupid to optimize for dumb SSD s seems to be a bit premature. Market externalities for example, does Intel have patents that will prevent competing smart SSD s from entering the market and thus forcing price drops? could radically change the picture. It s not just a pure technological choice, which is what makes projections and prognostications difficult. As another example, I don t know whether or not Intel will issue a firmware update that adds ATA TRIM support to the X25-M, or how long it will take before such SSD s become available. Until ATA TRIM support becomes available, it will be advantageous to add support in ext4 for a block allocator option that aggressively reuses blocks above all else, and avoids using blocks that have never been allocated or used before, even if it causes more in-file system fragmentation and deeper extent allocation trees. The reason for this is at the moment, once a block is used by the file system, at least today, the X25-M has absolutely no idea whether we still care about the contents of that block, or whether the block has since been released when the file was deleted. However, if 20% of the SSD s blocks have never been used, the X25-M can use 20% of the flash for better garbage collection and defragmentation algorithms. And if Intel never releases a firmware update to add ATA TRIM support, then I will be out $400 out of my own pocket for an SSD that lacks this capability, and so adding a block allocator which works around limitations of the X25-M probably makes sense. If it turns out that it takes two years before disks that have ATA TRIM support show up, then it will definitely make sense to add such an optimization. (Hard drive vendors have been historically S-L-O-W to finish standardizing new features and then letting such features enter the market place, so I m not necessarily holding my breath; after all, the Linux block device layer and and file systems have been ready to send ATA TRIM support for about six months; what s taking the ATA committees and SSD vendors so long? <grin> On the other hand, if Intel releases ATA TRIM support next month, then it might not be worth my effort to add such a mount option to ext4. Or maybe Sandisk will make an ATA TRIM capable SSD available soon, and which is otherwise competitive with Intel, and I get a free sample, but it turns out another optimization on Sandisk SSD s will give me an extra 10% performance gain under some workloads. Is it worth it in that case? Hard to tell, unless I know whether such a tweak addresses an optimization problem which is fundamental, and whether or not such a tweak will either be unnecessary, or perhaps actively unhelpful in the next generation. As long as SSD manufacturers force us treat these devices as black boxes, there may be a certain amount of cargo cult science which may be forced upon us file system designers or I guess I should say, in order to be more academically respectable, we will be forced to rely more on empirical measurements leading to educated engineering estimations about what the SSD is doing inside the black box . Heh. Related posts (automatically generated):
  1. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...

Theodore Ts'o: Should Filesystems Be Optimized for SSD s?

In one of the comments to my last blog entry, an anonymous commenter writes:
You seem to be taking a different perspective to linus on the adapting to the the disk technology front (Linus seems to against having to have the OS know about disk boundaries and having to do levelling itself)
That s an interesting question, and I figure it s worth its own top-level entry, as opposed to a reply in the comment stream. One of the interesting design questions in any OS or Computer Architecture is where the abstraction boundaries should be drawn and which side of an abstraction boundary should various operations be pushed. Linus s arguments is that there a flash controller can do a better job of wear leveling, including detecting how worn a particular flash cell might be (for example, perhaps by looking at the charge levels at an analog level and knowing when the last time the cell was programmed), and so it doesn t make sense to try to do wear leveling in a flash file system. Some responsibilities of flash management, such as coalescing newly written blocks into erase blocks to avoid write amplification can be done either on the SSD or in the file system for example, by using a log-structured file system, or some other copy-on-write file system, instead of a rewrite-in-place style file system, you can essentially solve the write amplification problem. In some cases, it s necessary let additional information leak across the abstraction for example, the ATA TRIM command is a way for the file system to let the disk know that certain blocks no longer need to be used. If too much information needs to be pushed across the abstraction, one way or another, then maybe we need to rethink whether the abstraction barrier is in the right place. In addition, if the abstraction has been around for a long time, changing it also has costs, which has to be taken into account. The 512 byte sector LBA abstraction has been around long time, and therefore dislodging it is difficult and costly. For example, the same argument which says that because the underlying hardware details are changing between different generations of SSD is all of these details should be hidden in hardware, was also used to justify something that has been a complete commercial failure for years if not decades: Object Based Disks. One of the arguments of OBD s was that the hard drive has the best knowledge of how and where to store an contiguous stream of bytes, and so perhaps filesystems should not be trying to decide where on disk an inode should be stored, but instead tell the hard drive, I have this object, which is 134 kilobytes long; please store it somewhere on the disk . At least in theory the HDD or SSD could handle all of the details of knowing the best place to store the object on the spinning magnetic media or flash media, taking into account how worn the flash is and automatically move the object around in the case of an SSD, and in the case of the HDD, the drive could know about (real) cylinder and track boundaries, and store the object in the most efficient way possible, since the drive has intimate knowledge about the low-level details of how data is stored on the disk. This theory makes a huge amount of sense; but there s only one problem. Object Based Disks have been proposed in academia and advanced R&D shops of companies like Seagate et.al. have been proposing them for over a decade, with absolutely nothing to show for it. Why? There have been two reasons proposed. One is that OBD vendors were too greedy, and tried to charge too much money for OBD s. Another explanation is that the interface abstraction for OBD s was too different, and so there wasn t enough software or file systems or OS s that could take advantage of OBD s. Both explanations undoubtedly contributed to the commercial failure of OBD s, but the question is which is the bigger reason. And the reason why it is particularly important here is because at least as far as Intel s SSD strategy is concerned, its advantage is that (modulo implementation shortcomings such as the reported internal LBA remapping table fragmentation problem and the lack of ATA TRIM support) filesystems don t need to change (much) in order to take advantage of the Intel SSD and get at least decent performance. However, if the price delta is a stronger reason for its failure, then the X25-M may be in trouble. Currently the 80GB Intel X25-M has a street price of $400, so it costs roughly $5 per gigabyte. Dumb MLC SATA SSD s are available for roughly half the cost/gigabyte (64 GB for $164). So what does the market look like 12-18 months from now? If dumb SSD s are still available at 50% of the cost of smart SSD s, it would probably be worth it to make a copy-on-write style filesystem that attempts to do the wear leveling and write amplification reduction in software. Sure, it s probably more efficient to do it in hardware, but a 2x price differential might cause people will settle for a cheaper solution even if isn t the absolutely best technical choice. On the hand, if prices drop significantly, and/or dumb SSD s completely disappear from the market, then time spent now optimizing for dumb SSD s will be completely wasted. So for Linus to make the proclamation that it s completely stupid to optimize for dumb SSD s seems to be a bit premature. Market externalities for example, does Intel have patents that will prevent competing smart SSD s from entering the market and thus forcing price drops? could radically change the picture. It s not just a pure technological choice, which is what makes projections and prognostications difficult. As another example, I don t know whether or not Intel will issue a firmware update that adds ATA TRIM support to the X25-M, or how long it will take before such SSD s become available. Until ATA TRIM support becomes available, it will be advantageous to add support in ext4 for a block allocator option that aggressively reuses blocks above all else, and avoids using blocks that have never been allocated or used before, even if it causes more in-file system fragmentation and deeper extent allocation trees. The reason for this is at the moment, once a block is used by the file system, at least today, the X25-M has absolutely no idea whether we still care about the contents of that block, or whether the block has since been released when the file was deleted. However, if 20% of the SSD s blocks have never been used, the X25-M can use 20% of the flash for better garbage collection and defragmentation algorithms. And if Intel never releases a firmware update to add ATA TRIM support, then I will be out $400 out of my own pocket for an SSD that lacks this capability, and so adding a block allocator which works around limitations of the X25-M probably makes sense. If it turns out that it takes two years before disks that have ATA TRIM support show up, then it will definitely make sense to add such an optimization. (Hard drive vendors have been historically S-L-O-W to finish standardizing new features and then letting such features enter the market place, so I m not necessarily holding my breath; after all, the Linux block device layer and and file systems have been ready to send ATA TRIM support for about six months; what s taking the ATA committees and SSD vendors so long? <grin> On the other hand, if Intel releases ATA TRIM support next month, then it might not be worth my effort to add such a mount option to ext4. Or maybe Sandisk will make an ATA TRIM capable SSD available soon, and which is otherwise competitive with Intel, and I get a free sample, but it turns out another optimization on Sandisk SSD s will give me an extra 10% performance gain under some workloads. Is it worth it in that case? Hard to tell, unless I know whether such a tweak addresses an optimization problem which is fundamental, and whether or not such a tweak will either be unnecessary, or perhaps actively unhelpful in the next generation. As long as SSD manufacturers force us treat these devices as black boxes, there may be a certain amount of cargo cult science which may be forced upon us file system designers or I guess I should say, in order to be more academically respectable, we will be forced to rely more on empirical measurements leading to educated engineering estimations about what the SSD is doing inside the black box . Heh. Related posts (automatically generated):
  1. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...

Matthew Palmer: Information Leaks from Vacation Replies

As if there weren't already enough reasons not to use out-of-office replies, here's a new one I haven't seen before (from a web design company, I think):
I am out of the office until Monday, 9 March. For LargeCorpA, please contact Person A at person.a@example.com. For LargeCorpB, SmallCompany1, or SmallCompany2, please contact Person B at person.b@example.com.
Well, I now know several companies who are clients of this web design company that I didn't know before. So, there's a nice corporate info gathering trick -- send e-mail to known addresses at your competitors on a regular basis (especially during holiday-prone times, like school holidays and Christmas/New Year) and see if they'll give you lists of their clients for free, like this helpful person has.

21 February 2009

Pablo Lorenzzoni: Git basics: reversing the 'git sucks' effect

I ve been using git this last few days and I am still working on a workflow for my projects. Unfortunately, as others have noticed, git violates POLS is so many ways, it ends up being hard to get. Creating a remote repository seems to be the first thing to bite a developer switching to git (mainly if coming from a centralized SCM). I have not decided which is the best way of doing it, but I ve been using git-daemon via inetd and a path in my remote hosting holding all my repositories for public pulling, and ssh for pushing. Here is how to create it: 

local$ ssh spectra@remotehost
remotehost$ mkdir /var/git/myproject.git
remotehost$ cd /var/git/myproject.git
remotehost$ git --bare init 
remotehost$ logout
local$ cd myproject
local$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
local$ git push remotehost master
...
otherlocal$ git clone git://remotelocal/myproject.git
otherlocal$ cd myproject
otherlocal$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
(hack hack hack)
otherlocal$ git push remotehost master
Now everybody can pull your repository at git://remotehost/myproject.git and you can push and pull to it via ssh. Note that you have to setup git-daemon, which is pretty straight forward. I am using it as an inetd daemon, but you can use it as a standalone one. Debian has a package which does just that. Now, some people think logging in a remote server just to create an empty repository is too much. Well repositories are just .git directories. It happens that you can push for the first time by rsyncing your .git with a remote host: 

local$ cd myproject
local$ rsync -a .git/ spectra@remotehost:/var/git/myproject.git
local$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
(hack hack hack)
local$ git push remotehost master
(Apparently you can push using rsync every time, but it s regarded as wiser to keep your probably crappy local repository commits separated from the public repository otherwise commit messages like Please, don t use this code are likely to pop up everywhere :) ). Now, I don t know if this have side effects, but it works :-) Another thing to notice is that git is more directed at pulling than pushing. This may be because of its designer: the way Linus works is by pulling changes from others repositories and not by letting others push em into his one. And this is another violation of POLS for most of the people, who is used to commit their changes into some remote repository. Rather than that, people using git would expose their own repositories in order to have it pulled by others. I also agree with most of the git critics wrt git commands There s a lot of examples and I am not going deeper in this but I think it was a bad choice calling checkout what git does when told to checkout , for instance. Yes, I know different tools, different ways of seeing it but everyone was already used to what centralized SCMs call their operations, and I think it would only help git adopting same names for the same operations, and inventing new ones for those proper of decentralized operations. Anyway, once you get it (and I have not completely got it yet), it seems all flow as expected. If this adaptation fails, there s still Easy Git to the rescue! One last thing that I think contribute to the git sucks effect: git-svn. This is a great tool, but it was built from git s point of view Given it s intended as a glue for Subversion newcomers, it would benefit more from being built from svn s point of view. This was mentioned by a colleague developer in my company, when he just couldn t understand why one have to git svn rebase instead of a simple update. So git-svn also suffers of the bad command naming habit git do. Of course, that given that you came from this environment (I am sure git and git-svn makes perfect sense for Linus & cia :-) ). I have not tried yet, but yap seems to be targeted on providing an alternative to git-svn.

20 February 2009

Theodore Ts'o: Aligning filesystems to an SSD s erase block size

I recently purchased a new toy, an Intel X25-M SSD, and when I was setting it up initially, I decided I wanted to make sure the file system was aligned on an erase block boundary. This is a generally considered to be a Very Good Thing to do for most SSD s available today, although there s some question about how important this really is for Intel SSD s more on that in a moment. It turns out this is much more difficult than you might first think most of Linux s storage stack is not set up well to worry about alignment of partitions and logical volumes. This is surprising, because it s useful for many things other than just SSD s. This kind of alignment is important if you are using any kind of hardware or software RAID, for example, especially RAID 5, because if writes are done on stripe boundaries, it can avoid a read-modify-write overhead. In addition, the hard drive industry is planning on moving to 4096 byte sectors instead of the way-too-small 512 byte sectors at some point in the future. Linux s default partition geometry of 255 heads and 63 sectors/track means that there are 16065 (512 byte) sectors per cylinder. The initial round of 4k sector disks will emulate 512 byte disks, but if the partitions are not 4k aligned, then the disk will end up doing a read/modify/write on two internal 4k sectors for each singleton 4k file system write, and that would be unfortunate. Vista has already started working around this problem, since it uses a default partitioning geometry of 240 heads and 63 sectors/track. This results in a cylinder boundary which is divisible by 8, and so the partitions (with the exception of the first, which is still misaligned unless you play some additional tricks) are 4k aligned. So this is one place where Vista is ahead of Linux . unfortunately the default 255 heads and 63 sectors is hard coded in many places in the kernel, in the SCSI stack, and in various partitioning programs; so fixing this will require changes in many places. However, with SSD s (remember SSD s? This is a blog post about SSD s ) you need to align partitions on at least 128k boundaries for maximum efficiency. The best way to do this that I ve found is to use 224 (32*7) heads and 56 (8*7) sectors/track. This results in 12544 (or 256*49) sectors/cylinder, so that each cylinder is 49*128k. You can do this by doing starting fdisk with the following options when first partitioning the SSD: # fdisk -H 224 -S 56 /dev/sdb The first partition will only be aligned on a 4k boundary, since in order to be compatible with MS-DOS, the first partition starts on track 1 instead of track 0, but I didn t worry too much about that since I tend to use the first partition for /boot, which tends not to get modified much. You can go into expert mode with fdisk and force the partition to begin on an 128k alignment, but many Linux partition tools will complain about potential compatibility problems (which are obsolete warnings, since the systems that would have booting systems with these issues haven t been made in about ten years), but I didn t needed to do that, so I didn t worry about it. So I created a 1 gigabyte /boot partition as /dev/sdb1, and allocated the rest of the SSD for use by LVM as /dev/sdb2. And that s where I ran into my next problem. LVM likes to allocate 192k for its header information, and 192k is not a multiple of 128k. So if you are creating file systems as logical volumes, and you want those volume to be properly aligned you have to tell LVM that it should reserve slightly more space for its meta-data, so that the physical extents that it allocates for its logical volumes are properly aligned. Unfortunately, the way this is done is slightly baroque: # pvcreate metadatasize 250k /dev/sdb2
Physical volume /dev/sdb2 successfully created Why 250k and not 256k? I can t tell you sometimes the LVM tools aren t terribly intuitive. However, you can test to make sure that physical extents start at the proper offset by using: # pvs /dev/sdb2 -o+pe_start
PV VG Fmt Attr PSize PFree 1st PE
/dev/sdb2 lvm2 73.52G 73.52G 256.00K If you use a metadata size of 256k, the first PE will be at 320k instead of 256k. There really ought to be an pe-align option to pvcreate, which would be far more user-friendly, but, we have to work with the tools that we have. Maybe in the next version of the LVM support tools . Once you do this, we re almost done. The last thing to do is to create the file system. As it turns out, if you are using ext4, there is a way to tell the file system that it should try to align files so they match up with the RAID stripe width. (These techniques can be used for RAID disks as well). If your SSD has an 128k erase block size, and you are creating the file system with the default 4k block size, you just have to specify a strip width when you create the file system, like so: # mke2fs -t ext4 -E stripe-width=32,resize=500G /dev/ssd/root (The resize=500G limits the number of blocks reserved for resizing this file system so that the guaranteed number size that the file system can be grown via online resize is 500G. The default is 1000 times the initial file system size, which is often far too big to be reasonable. Realistically, the file system I am creating is going to be used for a desktop device, and I don t foresee needing to resize it beyond 500G, so this saves about a 50 megabytes or so. Not a huge deal, but waste not, want not , as the saying goes.) With e2fsprogs 1.41.4, the journal will be 128k aligned, as will the start of the file system, and with the stripe-width specified, the ext4 allocator will try to align block writes to the stripe width where that makes sense. So this is as good as it gets without kernel changes to make the block and inode allocators more SSD aware, something which I hope to have a chance to look at.

horizontal separator All of this being said, it s time to revisit this question is all of this needed for a smart , better by design next-generation SSD such as Intel s? Aligning your file system on an erase block boundary is critical on first generation SSD s, but the Intel X25-M is supposed to have smarter algorithms that allow it to reduce the effect of write-amplification. The details are a little bit vague, but presumably there is a mapping table which maps sectors (at some internal sector size we don t know for sure whether it s 512 bytes or some larger size) to individual erase blocks. If the file system sends a series of 4k writes for file system blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, 99, followed by a barrier operation, a traditional SSD might do read/modify/write on four 128k erase blocks one covering the blocks 0-31, another for the blocks 32-63, and so on. However, the Intel SSD will simply write a single 128k block that indicates where the latest versions of blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, and 99 can be found. This technique tends to work very well. However, over time, the table will get terribly fragmented, and depending on whether the internal block sector size is 512 or 4k (or something in between), there could be a situation where all but one or two of the internal sectors on the disks have been mapped away to other erase blocks, leading to fragmentation of the erase blocks. This is not just a theoretical problem; there are reports from the field that this happens relatively easy. For example, see Allyn Malventano s Long-term performance analysis of Intel Mainstream SSDs and Marc Prieur s report from BeHardware.com which includes an official response from Intel regarding this phenomenon. Laurent Gilson posted on the Linux-Thinkpad mailing list that when he tried using the X25-M to store commit journals for a database, that after writing 170% of the capacity of an Intel SSD, the small writes caused the write performance to go through the floor. More troubling, Allyn Malventano indicated that if the drive is abused for too long with a mixture of small and large writes, it can get into a state where the performance degredation is permanent, and even a series of large writes apparently does not restore the drive s function only an ATA SECURITY ERASE command to completely reset the mapping table seems to help. So, what can be done to prevent this? Allyn s review speculates that aligning writes to erase write boundaries can help I m not 100% sure this is true, but without detailed knowledge of what is going on under the covers in Intel s SSD, we won t know for sure. It certainly can t hurt, though, and there is a distinct possibility that the internal sector size is larger than 512 bytes, which means the default partitioning scheme of 255 heads/63 sectors is probably not a good idea. (Even Vista has moved to a 240/63 scheme, which gives you 8k alignment of partitions; I prefer 224/56 partitioning, since the days when BIOS s used C/H/S I/O are long gone.) The Ext3 and Ext4 file system tend to defer meta-data writes by pinning them until a transaction commit; this definitely helps, and ext4 allows you to configure an erase block boundary, which should also be helpful. Enabling laptop mode will discourage writing to the disk except in large blocks, which probably helps significantly as well. And avoiding fsync() in applications will also be helpful, since a cache flush operation will force the SSD to write to an erase block even if it isn t completely filled. Beyond that, clearly some experimentation will be needed. My current thinking is to use a standard file system aging workload, and then performing an I/O benchmark to see if there has been any performance degradation. I can then vary various file system tuning parameters and algorithms, confirm whether or not a heavy fsync workload makes the performance worse. In the long term, hopefully Intel will release a firmware update which adds support for ATA TRIM/DISCARD commands, which will allow the file system to inform the SSD that various blocks have been deleted and no longer need to be preserved by the SSD. I suspect this will be a big help, if the SSD knows that certain sectors no longer need to be preserved, it can avoid copying them when trying to defragment the SSD. Given how expensive the X25-M SSD s are, I hope that there will be a firmware update to support this, and that Intel won t leave its early adopters high and dry by only offering this functionality in newer models of the SSD. If they were to do that, it would leave many of these early adopters, especially your humble writer (who paid for his SSD out of his own pocket), to be quite grumpy indeed. Hopefully, though, it won t come to that. Update: I ve since penned a follow-up post Should Filesystems Be Optimized for SSD s? Related posts (automatically generated):

  1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
  2. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

Theodore Ts'o: Aligning filesystems to an SSD s erase block size

I recently purchased a new toy, an Intel X25-M SSD, and when I was setting it up initially, I decided I wanted to make sure the file system was aligned on an erase block boundary. This is a generally considered to be a Very Good Thing to do for most SSD s available today, although there s some question about how important this really is for Intel SSD s more on that in a moment. It turns out this is much more difficult than you might first think most of Linux s storage stack is not set up well to worry about alignment of partitions and logical volumes. This is surprising, because it s useful for many things other than just SSD s. This kind of alignment is important if you are using any kind of hardware or software RAID, for example, especially RAID 5, because if writes are done on stripe boundaries, it can avoid a read-modify-write overhead. In addition, the hard drive industry is planning on moving to 4096 byte sectors instead of the way-too-small 512 byte sectors at some point in the future. Linux s default partition geometry of 255 heads and 63 sectors/track means that there are 16065 (512 byte) sectors per cylinder. The initial round of 4k sector disks will emulate 512 byte disks, but if the partitions are not 4k aligned, then the disk will end up doing a read/modify/write on two internal 4k sectors for each singleton 4k file system write, and that would be unfortunate. Vista has already started working around this problem, since it uses a default partitioning geometry of 240 heads and 63 sectors/track. This results in a cylinder boundary which is divisible by 8, and so the partitions (with the exception of the first, which is still misaligned unless you play some additional tricks) are 4k aligned. So this is one place where Vista is ahead of Linux . unfortunately the default 255 heads and 63 sectors is hard coded in many places in the kernel, in the SCSI stack, and in various partitioning programs; so fixing this will require changes in many places. However, with SSD s (remember SSD s? This is a blog post about SSD s ) you need to align partitions on at least 128k boundaries for maximum efficiency. The best way to do this that I ve found is to use 224 (32*7) heads and 56 (8*7) sectors/track. This results in 12544 (or 256*49) sectors/cylinder, so that each cylinder is 49*128k. You can do this by doing starting fdisk with the following options when first partitioning the SSD: # fdisk -H 224 -S 56 /dev/sdb The first partition will only be aligned on a 4k boundary, since in order to be compatible with MS-DOS, the first partition starts on track 1 instead of track 0, but I didn t worry too much about that since I tend to use the first partition for /boot, which tends not to get modified much. You can go into expert mode with fdisk and force the partition to begin on an 128k alignment, but many Linux partition tools will complain about potential compatibility problems (which are obsolete warnings, since the systems that would have booting systems with these issues haven t been made in about ten years), but I didn t needed to do that, so I didn t worry about it. So I created a 1 gigabyte /boot partition as /dev/sdb1, and allocated the rest of the SSD for use by LVM as /dev/sdb2. And that s where I ran into my next problem. LVM likes to allocate 192k for its header information, and 192k is not a multiple of 128k. So if you are creating file systems as logical volumes, and you want those volume to be properly aligned you have to tell LVM that it should reserve slightly more space for its meta-data, so that the physical extents that it allocates for its logical volumes are properly aligned. Unfortunately, the way this is done is slightly baroque: # pvcreate metadatasize 250k /dev/sdb2
Physical volume /dev/sdb2 successfully created Why 250k and not 256k? I can t tell you sometimes the LVM tools aren t terribly intuitive. However, you can test to make sure that physical extents start at the proper offset by using: # pvs /dev/sdb2 -o+pe_start
PV VG Fmt Attr PSize PFree 1st PE
/dev/sdb2 lvm2 73.52G 73.52G 256.00K If you use a metadata size of 256k, the first PE will be at 320k instead of 256k. There really ought to be an pe-align option to pvcreate, which would be far more user-friendly, but, we have to work with the tools that we have. Maybe in the next version of the LVM support tools . Once you do this, we re almost done. The last thing to do is to create the file system. As it turns out, if you are using ext4, there is a way to tell the file system that it should try to align files so they match up with the RAID stripe width. (These techniques can be used for RAID disks as well). If your SSD has an 128k erase block size, and you are creating the file system with the default 4k block size, you just have to specify a strip width when you create the file system, like so: # mke2fs -t ext4 -E stripe-width=32,resize=500G /dev/ssd/root (The resize=500G limits the number of blocks reserved for resizing this file system so that the guaranteed number size that the file system can be grown via online resize is 500G. The default is 1000 times the initial file system size, which is often far too big to be reasonable. Realistically, the file system I am creating is going to be used for a desktop device, and I don t foresee needing to resize it beyond 500G, so this saves about a 50 megabytes or so. Not a huge deal, but waste not, want not , as the saying goes.) With e2fsprogs 1.41.4, the journal will be 128k aligned, as will the start of the file system, and with the stripe-width specified, the ext4 allocator will try to align block writes to the stripe width where that makes sense. So this is as good as it gets without kernel changes to make the block and inode allocators more SSD aware, something which I hope to have a chance to look at.

horizontal separator All of this being said, it s time to revisit this question is all of this needed for a smart , better by design next-generation SSD such as Intel s? Aligning your file system on an erase block boundary is critical on first generation SSD s, but the Intel X25-M is supposed to have smarter algorithms that allow it to reduce the effect of write-amplification. The details are a little bit vague, but presumably there is a mapping table which maps sectors (at some internal sector size we don t know for sure whether it s 512 bytes or some larger size) to individual erase blocks. If the file system sends a series of 4k writes for file system blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, 99, followed by a barrier operation, a traditional SSD might do read/modify/write on four 128k erase blocks one covering the blocks 0-31, another for the blocks 32-63, and so on. However, the Intel SSD will simply write a single 128k block that indicates where the latest versions of blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, and 99 can be found. This technique tends to work very well. However, over time, the table will get terribly fragmented, and depending on whether the internal block sector size is 512 or 4k (or something in between), there could be a situation where all but one or two of the internal sectors on the disks have been mapped away to other erase blocks, leading to fragmentation of the erase blocks. This is not just a theoretical problem; there are reports from the field that this happens relatively easy. For example, see Allyn Malventano s Long-term performance analysis of Intel Mainstream SSDs and Marc Prieur s report from BeHardware.com which includes an official response from Intel regarding this phenomenon. Laurent Gilson posted on the Linux-Thinkpad mailing list that when he tried using the X25-M to store commit journals for a database, that after writing 170% of the capacity of an Intel SSD, the small writes caused the write performance to go through the floor. More troubling, Allyn Malventano indicated that if the drive is abused for too long with a mixture of small and large writes, it can get into a state where the performance degredation is permanent, and even a series of large writes apparently does not restore the drive s function only an ATA SECURITY ERASE command to completely reset the mapping table seems to help. So, what can be done to prevent this? Allyn s review speculates that aligning writes to erase write boundaries can help I m not 100% sure this is true, but without detailed knowledge of what is going on under the covers in Intel s SSD, we won t know for sure. It certainly can t hurt, though, and there is a distinct possibility that the internal sector size is larger than 512 bytes, which means the default partitioning scheme of 255 heads/63 sectors is probably not a good idea. (Even Vista has moved to a 240/63 scheme, which gives you 8k alignment of partitions; I prefer 224/56 partitioning, since the days when BIOS s used C/H/S I/O are long gone.) The Ext3 and Ext4 file system tend to defer meta-data writes by pinning them until a transaction commit; this definitely helps, and ext4 allows you to configure an erase block boundary, which should also be helpful. Enabling laptop mode will discourage writing to the disk except in large blocks, which probably helps significantly as well. And avoiding fsync() in applications will also be helpful, since a cache flush operation will force the SSD to write to an erase block even if it isn t completely filled. Beyond that, clearly some experimentation will be needed. My current thinking is to use a standard file system aging workload, and then performing an I/O benchmark to see if there has been any performance degradation. I can then vary various file system tuning parameters and algorithms, confirm whether or not a heavy fsync workload makes the performance worse. In the long term, hopefully Intel will release a firmware update which adds support for ATA TRIM/DISCARD commands, which will allow the file system to inform the SSD that various blocks have been deleted and no longer need to be preserved by the SSD. I suspect this will be a big help, if the SSD knows that certain sectors no longer need to be preserved, it can avoid copying them when trying to defragment the SSD. Given how expensive the X25-M SSD s are, I hope that there will be a firmware update to support this, and that Intel won t leave its early adopters high and dry by only offering this functionality in newer models of the SSD. If they were to do that, it would leave many of these early adopters, especially your humble writer (who paid for his SSD out of his own pocket), to be quite grumpy indeed. Hopefully, though, it won t come to that. Update: I ve since penned a follow-up post Should Filesystems Be Optimized for SSD s? Related posts (automatically generated):

  1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
  2. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

18 February 2009

Julian Andres Klode: Python modules, licenses, and more


Today, I want to present you some things I have asked myself and some ideas about them. You should not expect the information to be correct. Therefore, if you find mistakes, please leave a comment. Copyright statements / Comments MIT license: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. - If you had a python module released under the MIT license, and this is in the comment of the module and you somehow ship only pyc or pyo files, you would be violating the license by not including the copyright notice, because these files do not contain the modules. This is also true for many other licenses, but this seems to be the best example. If you include this in the docstrings, you would only violate such license terms if you distribute bytecode created with the -OO option. This also does not apply if the code is a program which prints the license (eg. via a commandline license option). GPL vs LGPL Is there any difference at all? The LGPL requires you to publish all changes you make to the code, while the GPL also requires you to publish source files you have created. This also means that you can t link a non-free program to a GPL library, but you can link it to a LGPL library. Because Python modules are not linked to each other, everything you do is normally considered a use of the module. Therefore, if there is a module G released under the GPL, and a module X released under a different, incompatible license X, you would still be able to use the facilities provided by the module G. This also effects subclassing classes of G in X. Due to the enormous flexibility provided by Python you can easily break the intented rules of the GPL. Instead of editing the class definition you subclass the class and edit it. You can also replace stuff inside the module G during run-time, simply by setting the relevant attributes. In summary, Python makes it very easy to work around the restrictions of the GPL, therefore, using the LGPL instead of the GPL makes no sense. You can t give others more rights than they already have. You would just make it easier for others in case they want to write new code and want to copy some of yours. What about the AGPL? The AGPL exposes (compared to the GPL3) further restrictions on using the software on eg. websites. It is intended for programs which may be used by SaaS providers. Like with the GPL, the enormous flexibility of Python compensates most of the restrictions the license. BTW, which license to choose? I normally choose to release my programs and modules, etc. under the terms of the GNU General Public License, version 3 (or at your option) any later version. But it also depends on the size of the project. When I work on small scripts, like hardlink, I generally choose the MIT license. This is also somehow related to the fact that I don t want to have a license which takes more than 50% of the size of my project. This is actually a bit different to what Bruce Perens does. Bruce recommends 3 types of licenses. The first one is what he calls the gift license. He recommends the Apache License 2.0, because it provides better protection from software patent lawsuites . The MIT license is another example for this type of license. While not providing the patent protection, this is not that critical for persons like me who live in Germany. Furthermore, the number of patents possibly infringed by the code is proportional to the amount of code. The second type he recommends is a sharing-with-rules license, like the GPL 3. Like him, I mostly use this license for my code. Sometimes I also use the GPL 2, but only when I am required to do so, or because of tradition. In generally, I only upgrade software from GPL-2+ to GPL-3+ when I introduce new features, not for bug fixes or similar. The third type he describes is the in-between license , like the LGPL. As I pointed out above, this type of license is not much different than the GPL, at least if applied to Python modules. Therefore, I never release any Python module under such a license. Things may be different for C libraries (and others), but I never released one. Documentation, etc. Well, I license all my documentation under the same license as the software. This makes it easier for the user because he does not need to read yet another license (at least if he reads all the licenses of the software he uses). If I distribute non-code content independent of code, I generally choose a Creative Commons License (CC-BY-SA 3.0, CC-BY 3.0), Germany. This also has an effect on this blog. From now on, all content (ever) provided by me via this blog is licensed under the terms of the Creative Commons Attribution-Share Alike 3.0 Germany, unless a different license information is included as part of the post. The design and comments from other persons are not included. Why I wrote this Really, I don t know. Maybe I just want to write something, maybe I want to write these things down, so I can read them. Anyway, please tell me if I my conclusions/ideas are wrong. Update 1: There was a mistake should expect the information to be correct , fixed now: should not expect [...] . I may be wrong with the GPL vs. LGPL thing, have not completely checked this. (2009-02-18 19:18 CET) Update 2: Seems the GPL vs LGPL thing is not correct, as written by Anonymous and Bruce. (2009-02-18 19:26 CET) Posted in General

Mark Brown: Buffer overflows ahoy

I may be wrong on this but it looks like Microsoft SMTP clients (at least Windows Mail and Outlook) don t like being sent a large volume of SSL certificate information when opening a TLS connection. They appear to assume that the data they are being sent is malformed and assume that STARTTLS failed, continuing with an unencrypted SMTP dialogue. This can be triggered relatively easily on a Debian system by telling Exim to use all the certificates provided by the ca-certificates package (which is the default configuration). The Windows clients will give an unhelpful the remote end dropped the connection style error, caused by the server getting upset by the unexpected fallback to unencrypted SMTP. The server logs will show something like this: 
2009-02-17 21:32:55 TLS error on connection from client.example.com (Client) [192.168.192.168] (gnutls_handshake): A TLS packet with unexpected length was received.
2009-02-17 21:33:00 SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=client.example.com [192.168.192.168] input="EHLO Client\r\n"
Configuring the MAIN_VERIFY_TLS_CERTIFICATES option in the Debian Exim configuration (which sets the tls_verifiy_certificates option in the actual Exim configuration) to point to something with less certificates in should avoid the issue. On the bright side, at least they re making an effort to avoid overflows.

17 February 2009

Adrian von Bidder: Bowflex machine

After dropping all (pretense at) regular exercise after leaving high/grammar/secondary school (the german word is Gymnasium) and doing more or less nothing for more than 10 years, I've taken up weight training (after a not very successful attempt at volleyball; I like the sport but the team fell apart) two years ago. Since I actually managed to work out regularly (twice a week) in the first year, and since membership in a gym is not cheap, and since I moved and thus had to find a different solution anyway, I decided to buy a Bowflex Sport home gym. I can't compare it to other home equipment, and it apparently it has been discontinued by Nautilus, but nonetheless, a few thoughts: The conclusion first: Would I buy it again? No. Was it a waste of money? No, it's not that bad. In detail: Closing remarks: I'll probably keep it for quite a while. I've thought about spending much less money and just getting a bench and a few weights, but I really like to have a lat tower. Overall I can do what I want to do with the exception of cardio workouts. The other angle is that there is quite a bit of bad press around about the founder of Nautilus, Arhur Jones. I can't comment on how trustworthy these articles are, and I can't comment on Nautilus' business practices today (Mr. Jones seems to be dead), so take this with a large pinch of salt before making a buying decision based on moral/ethical standards.

Sami Haahtinen: Know you upgrades, apt-listchanges

As an obligatory note, Debian Lenny was released earlier today. Which means that sysadmins all over the world are starting to upgrade their servers. There is an oddly little known tool that each and every sysadmin should install on at least one server they maintain, called apt-listchanges. It lists changes made to packages since the currently installed version. Sure that information will be overwhelming on major upgrades, but what is useful even on major upgrades is the capability to parse News files in the same way. News files contain important information about the package in question. For example a maintainer could list known upgrade problems there, like is done in the lighttpd package. Or list changes in package specific default behaviour, like is done in Vim package. Sure, you will notice these in time, but it's nice to get a heads up before a problem bites you.

Sami Haahtinen: Xen domU and grub

I've been bitten by grub upgrades and installations on Debian family domU servers. Apparently there are others out there who have been bitten too. The bug itself is caused by a missing device entry, probably because of udev. Anyway, grub-probe tries to discover the root device so that update-grub can properly generate a menu.lst. In certain scenarios the root device itself doesn't exist. Here is an example from a configuration generated with xen-tools: 
Hydrogen:/etc/xen# grep phy Neon.cfg 
disk    = [ 'phy:Local1/Neon-disk,sda1,w', 'phy:Local1/Neon-swap,sda2,w' ]
While this is a valid configuration, the device sda doesn't exists within the virtual machine. As a workaround the above blog entry suggests manually adding the sda device and the device entry in device.map. This solution does work, but it will fail with the next upgrade. The proper solution is to adjust the Xen configuration so that the root device is created. And since Xen uses different naming scheme for devices we can upgrade to that too. So the above example becomes: 
Hydrogen:/etc/xen# grep phy Neon.cfg 
disk    = [ 'phy:Local1/Neon-disk,xvda,w', 'phy:Local1/Neon-swap,xvdb,w' ]
You also need to adjust the existing grub configuration and fstab within the domU. It's a bit more work and requires an additional reboot, but it gives you a peace of mind that the next upgrade will work without a hitch.

16 February 2009

Jon Dowland: hard sf

"Hard SF" is a sub-genre of Science Fiction where "Science" is key. It's very hard to draw solid lines around stories that are examples of Hard SF, but it's easier to provide examples of things which are not: "Star Wars", for example, where space ships, travel and light sabres are window-dressing around a fantasy story. If you look at the Wikipedia article on Hard SF, in particular it's edit history, you will see people continually argue back and forth. Editor Jonathan Strahan recently asked for some recommendations of Hard SF authors to be included in a new Hard SF anthology. I stumbled across this via sfsignal. There are some good comments at both sites, and a fair amount of disagreement over whether certain authors should be considered Hard SF authors or not. Alastair Reynolds was cited by people several times. I really enjoy much of Reynold's writing, but I wouldn't consider it Hard SF. I argued this point and Reynolds himself joined the discussion.
This page is a draft.

15 February 2009

Michael Prokop: Debian GNU/Linux 5.0 codename Lenny - News for sysadmins

Alright, Debian GNU/Linux 5.0 AKA as Lenny has been released. Time for a Debian unstable unfreeze party! 8-) What does the new stable release bring for system administrators? I ll give an overview what news you might expect when upgrading from Debian GNU/Linux 4.0, codename Etch (released on 8th April 2007) to the current version Debian GNU/Linux 5.0, codename Lenny (released on 14th February 2009). I try to avoid duplicated information so make sure to read the release announcement and the official release notes for Lenny beforehand. Noteworthy Changes Virtualisation Virtualisation related new tools: Desktop oriented packages like virtualbox and qemu are available as well of course. Noteworthy Updates This is a (selective) list of some noteworthy updates: New packages Lenny ships over 7000 new packages. Lists of new/removed/replaced packages are available online. I ll name 238 sysadmin related packages that might be worth a look. (Note: I don t list addon stuff like optional server-modules, docs-only and kernel-source related packages. I plan to present some of the following packages in more detail in separate blog entries.) Further Ressources

Steve Kemp: Imake art, until someone dies

This weekend I have mostly been investigating Common Lisp. If you're new to the environment then you may get started via a simple: 
apt-get install clisp
Once you've done that the following resources will help you decide whether you're wasting your time or not: It has been a long time since I've had so much fun - even though the first thing I did was experiment with implementing a spam filter - although in my defense I was only classifying based upon the links contained in the mails. So for example: (NSFW links) ObFilm: Batman

David Pashley: MySQL silently truncating your data: Update

After my entry yesterday about MySQL truncating data, several people have pointed out that MySQL 4.1 or later gives you a warning. Yes, this is true. You can even see it in the example I gave: 
Query OK, 1 row affected, 1 warning (0.00 sec)
I ignored mentioning this, but perhaps should have said something about it. I reason I didn't mention it was because I didn't feel that a warning really helped anyone. Developers have enough problems remembering to check for errors, let along remembering to check in case there was a warning as well. Plus, they'd then have to work out if the warning was something serious or something they could ignore. There's also the question of how well the language bindings present this information. Take for example, PHP. The mysqli extension gained support for checking for warnings in PHP5 and gives the following code as an example of getting warnings: 
mysqli_query($link, $query);
if (mysqli_warning_count($link))  
   if ($result = mysqli_query($link, "SHOW WARNINGS"))  
      $row = mysqli_fetch_row($result);
      printf("%s (%d): %s\n", $row[0], $row[1], $row[2]);
      mysqli_free_result($result);
Hardly concise code. As of 5.1.0, there is also mysqli_get_warnings(), but is undocumented beyond noting its existence. The MySQL extension does not support getting warning information. The PDO wrapper doesn't provide any way to get this information. In perl, DBD::mysql has a mysql_warning_count() function, but presumably would have to call "SHOW WARNINGS" like in the PHP example. Seems Python's MYSQLdb module will raise an exception on warnings in certain cases. Mostly using the Cursor object. In java, you can set the jdbcCompliantTruncation connection parameter to make the driver throw java.sql.DataTruncation exceptions, as per the JDBC spec, which makes you wonder why this isn't set by default. Unfortunately this setting is usually outside the programmer's control. There is also the java.sql.Statement.getWarnings(), but once again, you need to check this after every statement. Not sure if ORM tools like Hibernate check this or not. So, yes MySQL does give you a warning, but in practice is useless.
Read Comments (0)

Mark Brown: Upgrading crm114

When upgrading from older crm114 releases and trying to retain your existing configuration it is important to check that all the configuration options that the new version expects to be set have been set. While some will cause errors if they re omitted others will appear to work but will cause unwanted behaviour at runtime. For example, omitting good_threshold and spam_threshold will cause everything to be flagged as spam in X-CRM114-Status even though the classifier is working well. In practice there are relatively few configuration options that users are expected to configure so it may be easier to redo the configuration based on the example provided. For safety it s best to delete your existing CSS files too in case they ve been invalidated by a configuration or format change. It s all a bit manual but it s worth it for what it does for my inbox.

14 February 2009

Joerg Jaspert: Lenny Release

I just finished most of my jobs for the Lenny release. I started, together with Mark Hymers, at 10:23 UTC. This was ahead schedule, which said 12:00 UTC, but better early than late. After disabling all our ftpmaster cronjobs, setting up our work environment and whatever else we needed and at 10:43 UTC we got the Release Teams GO. All of what we did got logged from screen, as we intend to write a dak tool doing such releases. Anyway, I won t bore you with a detailed set of actions we had to took, way too much for now, just some few moments (All times in UTC): While writing this entry people are still doing work for the release. For example the cd building will take time, after all there are well over 500 images (and 36 live cds). Current plan seems to be that we ftpmaster push the archive mirrors around 23:00 UTC, so it has time to sync everywhere. Some while later the cd mirror will get a push, and then the official announcement will happen. Thanks go out to everyone involved in preparing this release, be it in the past by fixing bugs, uploading packages, doing whatever was needed, as well as doing the work today.

Next.

Previous.