Search Results: "ed"

23 November 2010

Mike Hommey: Improving libxul startup I/O by hacking the ELF format

If you have been following this blog during the past months, you might have gathered that I ve been focusing on I/O patterns on library initialization, more specifically libxul.so. The very patterns there s not much to do about, except enduring them, or hacking ld.so and/or the toolchain. The latter is not necessarily easy, and won t benefit everyone until all linux distros use an improved glibc and/or toolchain. This could take years before any actual result would reach end users. But with a little creativity, one can overcome these limitations and improve the situation today. Here is a summary of my recent hacks. Backwards static initializers As Taras showed in the past, static initializers are executed in reverse order. It turns out it is an unfortunate design choice in gcc, and it also turns out not to impact all ELF binary files. As far as I know there are currently two different implementations of the static initializers call loop, one provided by gcc, and one provided by ld.so. In the gcc case, the compiler creates a .ctors section, starting with 0xffffffff, ending with 0x00000000, and filled with pointers to the various static initializer functions, in ascending order. The compiler also injects an initializer function, called from .init, going through this .ctors section, starting from the end (__do_global_ctors_aux in crtend.o). A similar .dtors section is created for functions to run when the library is unloaded, and the pointers in it are called in ascending order. There is actually no importance in the order in which .ctors and .dtors are executed. All that matters is that they are executed in reverse order from each other, because that limits the risks of bad ordering. It is only unfortunate that .ctors was chosen to be the one backwards. In the ld.so case, the compiler creates a .init_array section, filled with pointers to the various static initializer functions, in ascending order. It also stores the location and the size of this section in the PT_DYNAMIC ELF header. At runtime, ld.so itself reads this information and executes the static initializers in ascending orders. A similar .fini_array section is created for functions to run when the library in unloaded, and ld.so goes through it in descending order. The .init_array section is newer, and for backwards compatibility reasons, is not used on systems that have been using the .ctors approach forever. On the other hand, some newer ABIs, such as arm EABI, uses the new facility, which means they are not victim of backwards static initialization. I however wonder if backwards compatibility is still needed, considering how long .init_array support has been around (more than 11 years, according to glibc git repository). While we could obviously bug the gcc maintainers to revert their .ctors execution order, or ditch .ctors support on glibc systems, which means we wouldn t get any immediate global benefit, we can also just revert the .ctors section content. Please note the source code out there doesn t take care of reverting the .dtors section accordingly, because it is currently empty in libxul.so. Relocations As I mentioned in an earlier post, relocations are what make it possible to load libraries at arbitrary locations. Unfortunately, a great number of relocations means awful I/O patterns: ld.so reads and applies them one by one, and while the kernel reads ahead by 128 KB chunks, when your relocation section exceeds that amount by more than an order of magnitude, it means I/O goes back and forth a lot between the relocation section and where the relocations take place. There are two classes of relocations: REL and RELA. In both cases, there are several types of relocations, some of which require an addend. The difference between both classes is in REL relocations, the addend is stored at the offset the relocation takes place, and in RELA relocations, the addend is stored in the relocation entry. It means that RELA relocations effectively waste space, as the addend ends up actually being stored twice: it is also stored at the offset the relocation takes place, like with REL relocations, though it could technically be nulled out, there ; in any case, the space is wasted. REL relocations thus take 2 words of storage, and RELA relocations, 3. On x86, the ELF ABI only uses REL relocations, and words are 32 bits, making each relocation take 8 bytes ; on x86-64, the ELF ABI only uses RELA relocations, and words are 64 bits, thus making each relocation take 24 bytes, 3 times that of x86 relocations. On ARM, both may be used, but apparently the compiler and the linker only use REL relocations. These words of storage are used to store 3 or 4 bits of informations: The linker splits relocations in two main categories: those for dynamic linking at startup and those for procedure linkage table (PLT) resolution at runtime. Here is what the repartition look like on libxul.so (taken from the mozilla-central nightly from Nov, the 22nd):
dynamic relocations PLT relocations
Architecture number size (bytes) number size (bytes)
x86 231,836 1,854,688 3,772 30,176
x86-64 235,893 5,661,432 3,773 90,528
So, PLT relocations are quite negligible, and dynamic relocations are just huge, especially on x86-64. Compared to the binary size, it s pretty scary:
Architecture libxul.so size relocations size %
x86 21,869,684 1,884,864 8.61%
x86-64 29,629,040 5,751,984 19.41%
Let me remind you that not only do we need to read these relocations at startup (at least the dynamic ones, but the PLT ones are just negligible), but we only read them by chunks, and need to read other data in between. Of the many existing types of relocations, only a few are actually used in dynamic libraries. Here is a summary of those in use in libxul.so:
x86 relocation type number x86-64 relocation type number
dynamic relocations
R_386_TLS_DTPMOD32 1 R_X86_64_DTPMOD64 1
R_386_GLOB_DAT 213 R_X86_64_GLOB_DAT 238
R_386_32 27,643 R_X86_64_64 27,611
R_386_RELATIVE 203,979 R_X86_64_RELATIVE 208,043
PLT relocations
R_386_JUMP_SLOT 3,772 R_X86_64_JUMP_SLOT 3,773
The above types can be separated in two distinct categories: In the second category, it means we are wasting all the bits for the symbol reference in each entry: 24 on x86 and 32 on x86-64. Unfortunately, the vast majority of the relocations in libxul.so are from that category. At this point, this is how much waste we identified:
description num. entries waste per entry (bytes) total % of relocation size
x86
Relocation without symbol ref. 203,980 3 611,937 32.46%
x86-64
Duplicated addend 239,666 8 1,917,328 33.33%
Relocation type encoded on 32 bits 239,666 3 718,998 12.49%
Relocation without symbol ref. 208,044 4 832,176 14.46%
Total 3,468,502 60.30%
On top of that, one could argue that we don t need 24 or 32 bits for symbol reference (which is the entry number in the symbol table), and that we re really far from requiring 64 bits for r_offset on 64 bits systems, as libxul.so is far from being bigger than 4GB. Anyways, that means we have room for improvement here, and it means that the ELF format, especially on 64 bits systems with RELA relocations, really doesn t help with big programs or libraries with a lot of relocations. (For that matter, the mach-o format used on OS-X has some kind of similar issues ; I don t know for the DLL format) Another interesting data point is looking at how relocations r_offset are following each other, once relocations are reordered (to avoid too much symbol resolution, they are actually grouped by symbol). It turns out 206,739 relocations r_offset are at 4 bytes from the previous one on x86 (i.e. strictly following it), and 210,364 at 8 bytes from the previous on x86-64. The main reason why this happens is that most relocations are happening in pointer tables, and libxul.so has lots of them: And when I say libxul.so has lots of them, that s not an understatement: there are 184,320 relocations due to vtables (function tables you get when using virtual methods in C++) on x86 and 184,098 on x86-64. That s respectively 78.23% and 76.81% of the total number of relocations. Interestingly, while most of these relocations are R_*_RELATIVE, 26,853 are R_386_32, and 26,829 are R_X86_64_64. Except for a very few, these non relative relocations are for the __cxa_pure_virtual symbol, which is a dummy function that does nothing, used for pure virtual methods, such as the following:
class Abstract
public:
virtual void pure_virtual() = 0;
;
As for the remaining relative relocations, there are a few pointer tables responsible for several thousand relocations, such as JSString::length2StringTable, gEntries, GkAtoms_info, Html5Atoms_info, kUnixCharsets, nsIDOMCSS2Properties_properties, etc. Packing relocations While they are not detrimental to most binaries, we saw that relocations can have a bad effect on startup I/O for big binaries. Reducing how much space they take would have a direct impact on these startup I/Os. One way to do so is obviously to reduce the number of relocations. But as we saw above, the vast majority are due to vtables. While reducing vtables would be a nice goal, it s a long way before this can have a significant impact on the relocations number. So another possible way is to try to avoid wasting so much space in relocations. The main problem is that relocations are applied by ld.so, and that, as mentioned in the introduction, while we can hack ld.so to do what we want, it won t work everywhere before a long time. Moreover, it s difficult to test various implementations, and binary compatibility needs to be guaranteed once something is applied in the upstream libc. The idea I started to test a few weeks ago, is to effectively apply relocations by hand instead of relying on ld.so doing it. Considering the first thing ld.so does with a library after applying relocations is to call its DT_INIT function, the idea was to replace the dynamic relocations we can with a new DT_INIT function we would inject that d apply those relocations, while storing them in a more efficient manner. The function would obviously need to call the original DT_INIT function once done. Some parts of the PT_DYNAMIC section also obviously need to be adjusted accordingly. The first iteration I tested a few weeks ago helped validating the idea, but didn t change much of the binary: the dynamic relocation section still was taking all the space it took before, though its content was mostly empty (we ll see further below how efficient even the current dumb format is). Once the theory that an injected DT_INIT function could apply the relocations without breaking everything, I went on to implement a program that would actually strip the ELF binary. I won t get too much into the gory details, but it required PT_LOAD splitting, thus program headers growth, which means moving the following sections, then removing a huge part of the relocations, shrinking the relocations section, injecting our code and the packed relocations as sections, and moving the remaining sections after these new sections, to avoid empty space, but still respecting the proper offsets for both PT_LOAD and the code itself. So far, I only took care of the R_*_RELATIVE relocations. Some but not all other relocations could be taken care of, too, but everything involving symbol resolution from other libraries would be really difficult to handle from the injected code. The relocations are currently stored with a quite dumb format: As we only take care of one type of relocations, we can skip any information about the type. Also, the addend already being stored at r_offset, we just strip it from RELA relocations ; the binary being far from 4GB big, 32 bits is enough on 64 bits systems too. We also saw that a lot of relocations were next to the following ones, so instead of storing r_offset for each one, storing some kind of range is more efficient. As I ve so far been focusing on getting something that actually works more than on getting something really efficient, I only tested this format, but the results are already nice:
x86 x86-64
before
dynamic relocations (bytes) 1,854,688 5,661,432
libxul.so size 21,869,684 29,629,040
after
dynamic relocations (bytes) 222,856 668,400
injected code size 129 84
packed relative relocations (bytes) 224,816 228,568
libxul.so size 20,464,836 (-6.42%) 24,865,520 (-16.07%)
Getting rid of the __cxa_pure_virtual relocations would help getting this even further down, and an even better storage format could be used, too. Gathering new startup coverage data should be interesting as well. The code is available via mercurial. Please note the Makefile deeply sucks, and that the code is in the middle of its third refactoring. Anyways, once you get the code to compile, you just need to run test-x86 or test-x64 depending on your architecture, from the build directory, and give it the libxul.so file as an argument. It will create a libxul.so.new file that you can use to replace libxul.so. You may successfully use the tool on other libraries, but chances are you ll bump into an assert(), or in the worst case, that the resulting library ends up broken. The tool might work on PIE executables, but most probably not on other executables. If you wonder, Chrome is not PIE. It s not even relocatable (which means ASLR won t work with it) ; or at least the x86-64 binaries I just got aren t. More possible ELF hacking The code I wrote to pack relocations allows to do various tricks on ELF files, and it opens various opportunities, such as: Some of the above could probably be done with linker scripts, but using linker scripts is unnecessarily over complicated: as soon as you start using a linker script to fiddle with sections, you need to reimplement the entire default linker script (the one you can see with ld --verbose). And it s not guaranteed to work (Taras told me he had a hard time failing to align the .bss section). And as far as I know, you can t use the same thing depending whether you use ld or gold (AIUI, gold doesn t support the whole ld linker script syntax).

Biella Coleman: An unlikely story about a pit bull attack, free software, and a New York Times reporter

If I told you that in the last two days, I have been caught in a vortex of coincidence, a vortex composed of pit bulls, free software, diaspora (the software), mold, and a New York Times reporter, I bet you would think not likely. So the story started on Jet Blue, which offers snacks, lots of them, and Direct TV. Since I don t have TV I kinda go on a binge, watching all sorts of shows as I make my way home. I watched a pretty distributing but interesting documentary on Jim Jones on CNN and a show on Animal Planet on pit bulls and parolees. When I rolled into my my current digs in northern Manhattan (I am currently banished from my downtown apt due to mold, but that is a whole other story), there was a dinner party well underway. At some point in the evening prompted by me, we talked pit bulls as my friends want to get one but their family has issued a threat of disavowal if they do. The next morning, I was scoping out the website for the Animal Planet show as I was intrigued by it and frankly I kinda like pit bulls (maybe less now although I think they are unfairly maligned). Five minutes into pursuing the site, I hear screeches from hell. It sounds like a woman is being attacked. And she is. A woman right outside of my window was being attacked by a pit bull. So I am staying with a friend, an open source developer, Karl Fogel and good soul that he is, he runs out to help the lady (since I have been subject to 5 weeks of sickness due to mold or so that is what we think it is was enough for me; I could not stomach the idea of getting bit so I played the role of concerned spectator). It took minutes upon minutes, really just too many minutes to get the pit bull off, even a brick pounded against his head failed (apparently, a cigarette or match held to the throat does the trick, which I found out later). Eventually, the dog was extracted, a huge team of cops showed up, the dog was whisked away, the victim taken to the hospital, and life returned to calm and quiet. So the next day, I was being interviewed by a New York Times reporter Jim Dwyer who wrote a story about Diaspora for the New York Times back in the summer, helping to propel it from relative obscurity to near insta-fame (one of the Diaspora developers, Max, was my student). We were running out of time (I had another appointment) so I asked him if he lived in northern Manhattan as that is what his bio page indicates. He confirmed, I explained I was up there and that we could meet up there later to finish up. He inquired what part, I told him roughly where I was, he remarked he was near there, and so naturally I told him about the crazy pit bull attack I witnessed from my window as I can t shut my trap when it comes to things like that. Well yes you know what is coming next next: he was there, helping Karl (and others) deal with the pit bull attack. He lives nearby and heard the shrieks of agony and came out to aid. All and all it was pretty horrific. He also met Karl in so far as Karl gave him his phone number and email just in case he was needed as a witness (Karl had to dash off to catch a plane). Well, the funny thing, or as you also might guess: Jim, who is doing some more writing on tech, free software etc, should really talk to Karl given his key role in the community, so they already met, although under odd and terrible circumstances. I am not sure if I am more wigged out by the fact that I was reading about pit bulls when the attack happened or whether the reporter I was interviewed by was there along side with a free software developer he really needs to interview. Whatever the case, I kinda hope the vortex of coincidence now leaves me to hit someone else (sans any horrible attack). Or else, as Karl noted in the blog comments, I will have to be very careful about what shows I watch:
Amen to that! Enough with the coincidence vortex. As I said to Biella in IRC later: Do us a favor don t watch any shows about nuclear attacks on New York, okay

John Goerzen: Game Suggestions?

I m not an avid gamer, but I do occasionally enjoy playing video games. I sometimes have time to do so over the holidays, so I m looking for suggestions. I have a PS3 and a PC available. Thinking about what I like, most recently it seems to be immersive world games: Oblivion, Dragon Age, Mass Effect 1 & 2, and GTA. I ve also enjoyed simulation games, such as Civ, SimCity, FreeCiv, and Railroad Tycoon in the past and could again. Some of the Star Wars games (Knights of the Old Republic, Jedi Academy, Force Unleashed) I ve enjoyed, though the ones that have essentially no plot I didn t really. What suggestions do you have?

Dominique Dumont: How about a configfs with Config::Model ?

Hello During the Config::Model presentation at Debian mini-conf, I was asked whether Config::Model could be used from other languages than Perl. Of course, config-edit command line can be launched from any program with any language, but we can hardly call it an API. There s also the possibility to use Config::Model::SimpleUI to fork a process and send commands on its STDIN and get results on STDOUT. But using this from another program is awkward. On kernel side, /proc and /sys are widely used to get information from the kernel and set kernel configuration. How about using a similar approach with system configuration data ? Configuration data from config files is already mapped into a tree within Config::Model. This tree can also be mapped to a file system structure. This file system structure could have the following layout:
/conf/
system/
sshd/
PermitRootLogin # contain 0 or 1
ssh/

popcon/
As a bonus, this file system would provide a unified access to all system configuration data available through Config::Model. In any case, people would still be able to tinker configuration with their favorite text editor on a file in /etc. The modification will be propagated to the corresponding entry in the /conf virtual file system. The reverse will also be done. Anyone interested ? All the best PS: Further technical discussion will be held on Config::Model mailing list: config-model-users at lists.sourceforge.net 2010.11.25 edit: specified that config data can still be written in usual config file.

Russell Coker: A First Digital Camera

I ve just been asked for advice on buying a digital camera. I m not an expert on cameras but I have a good general knowledge of technology and I m sure that the readers of my blog will correct me rapidly if I make any serious mistakes. ;)Types of CameraThe Wikipedia Page about Digital Cameras is worth reading [1].Here are the types of camera that are useful as a stand-alone camera (IE not camera phones or industrial cameras):I suggest that people not start out a hobby of digital photography with a DSLR or other interchangeable lens camera. I think that it s best to start out with simple gear that s cheap if you decide that digital photography is not your thing then you have wasted less money, and if you really get into it then you ll be able to make a more informed choice about an expensive camera after getting some experience.A spare cheap compact camera can be useful even if you own a more expensive camera. There are times when you don t want the bulk of a DSLR or Bridge Camera and when you have a risk of theft or accidental damage such that you don t want to take an expensive camera. People who are really serious about photography apparently take a camera everywhere, you could have a compact camera in your front pocket for fast pictures and a bulky camera in your backpack for when you have time to prepare a quality shot. So buying a cheap camera and then buying an expensive one a few months later would not be wasting money!As the person who asked for advice has never owned a digital camera before I ll focus on cheap compact cameras for the rest of this post.Pixels and LensesCameras are typically advertised with the resolution in Megapixels described in bold. Presumably most people search for the camera with the highest resolution. The first thing to beware of is cameras that don t have hardware which is capable of taking a picture of the stated resolution, they use interpolation to generate a higher resolution image. There are probably some cameras with interpolation that are OK and provide a decent picture at a low price, but generally I recommend avoiding all cameras that do interpolation.Optical zoom is a very important feature. Often pictures have to be cropped to remove unwanted background, if you zoom in appropriately you can avoid cropping and make better use of the available resolution. Beware of cameras that advertise things such as advanced zoom , anything that doesn t exactly say optical zoom is using digital zoom IE interpolation. I suggest not considering a compact camera unless it can do at least 4* optical zoom, with 5* or better being preferable.The physical size of the lens is important. A bigger lens allows better pictures in adverse lighting and also allows a faster shutter speed to give better photos of moving objects and better photos when you are moving. Generally you can get a rough idea of the potential that a camera has by just glancing at the lens, if it s the size of a lens in a mobile phone then the pictures won t be that great. If it is on a telescopic mount and it s wide then the result will be better. When buying a lens for a DSLR or other detachable lens camera you should be able to read specifications of the lens which indicate it s size. For compact cameras the specifications of the lens are usually available from the vendor and often available from review sites but generally aren t included in adverts by retailers. If you are comparing cameras in the store looking at them seems like the best option.The ratio of the lens area to the number of pixels determines how much light is received by the sensors for each pixel. So when there are two cameras that are essentially identical apart from the number of megapixels the one with the highest number isn t necessarily better in all situations. A 12MP camera might not give a better result than a 10MP camera, a monitor described as Full HD has a resolution of 1920*1080 which is 2MP. A 5MP camera is useful to allow cropping but if the aim is to display pictures on current monitors then anything much bigger than 5MP is probably wasted at this time and 10MP will provide pictures that can use the capabilities of monitors that are developed for a while.PrintingSome printers that are affordable for home use might require something like a 7MP camera to print a picture at A4 or Letter size at the highest quality. Professional printing will probably require something even greater. But if your intended use of a digital camera doesn t involve printing the pictures, or only rarely involves printing them at A4 or Letter size then you don t need a high resolution for printing. I expect that there are some available printers that can use paper at sizes such as B3 which might require a 20MP camera for best quality. But if the aim of the printing is to put a picture on your wall (the most common case) then even if it s not at the ideal resolution then probably no-one will notice the difference you can t see the pixels from a few meters away.If the aim of the camera is to photograph professional artwork for the purpose of selling it on the Internet then a high resolution camera really isn t desired. You want to offer pictures on the Internet which aren t good enough to be usefully printed.VideoThe capabilities for recording video can vary significantly between different cameras in the same range. If you don t care about video then that s fine, but if video matters to you then you have to read the specs. It s also worth considering digital video cameras, I have previously summarised the available digital video cameras although I m sure that there have been some new models since then [3]. A good (expensive) digital camera will have video functionality that compares well to most digital video cameras but it will cost a lot more.Choosing a Compact CameraThere are many compact cameras in the $100 to $150 price range. Ted s has the Samsung Digimax ES65 for $99.95 which has 10MP and 5* optical zoom. This seems like a good deal. Practice with a cheap camera could easily allow you to make a better choice when buying a more expensive camera that saves you more than $100 so it seems unlikely that you will have any great regrets about buying a $100 camera.Buying from the specs is a bad idea, I recommend testing a camera in the store before buying every store that you want to buy from allows this. But don t expect that quality pictures in the store means much, the people who run the camera stores usually set them up with good lighting so that pictures will tend to turn out well. The aim of an in-store test is not to discover what the camera does well, but to discover what (if anything) it does badly. Also it s useful to test the ergonomics of the camera before buying, discover whether you would be happy to hold the camera for an hour.It s a good idea to ask the staff at a camera store for advice, but don t rely on such advice. My experience is that they tend not to ask what the prospective customer plans to do with the camera and this significantly limits the quality of their recommendations.ConclusionMake your first digital camera a cheap compact camera that doesn t cost much more than $100. Make sure it has at least 4* optical zoom with 5* being preferred. Don t worry too much about the number of megapixels, anything more than 5 will do. Test the camera in the store to look for any obvious reasons that make it unsuitable for your use. If you want video then be prepared to pay a little more as the cheapest cameras have a low resolution for video.

Michal Čihař: New phpMyAdmin features - user interface cleanup

Next article highlighting some new features which will come in phpMyAdmin 3.4 is about more little changes. You can try them it out on our demo server. Or just wait few hours for first alpha release of 3.4. The user interface cleanup is partly connected with previous changes, because some functionality simply required more Javascript (for example editor for ENUM/SET fields). Besides these there were major changes in export and import pages, which now should be easier to understand, hiding advanced options from unexperienced users, while still providing ability to change them. New phpMyAdmin export page This feature was implemented by Lori Lee during Google Summer of Code 2010.

Filed under: Phpmyadmin 0 comments Flattr this!

Eddy Petrișor: This was a ton of fun

This past saturday I passed by this man in the park. He was calling and feeding some squirrels.

<object height="385" width="480"><param name="movie" value="http://www.youtube.com/v/eA1ASjWRmbs?fs=1&amp;hl=ro_RO"><param name="allowFullScreen" value="true"><param name="allowscriptaccess" value="always"><embed allowfullscreen="true" allowscriptaccess="always" height="385" src="http://www.youtube.com/v/eA1ASjWRmbs?fs=1&amp;hl=ro_RO" type="application/x-shockwave-flash" width="480"></embed></object>

He was kind enough to give me some walnuts to call the squirrels myself. He did this without even me asking.

<object height="385" width="480"><param name="movie" value="http://www.youtube.com/v/oL700Fnd3ck?fs=1&amp;hl=ro_RO"><param name="allowFullScreen" value="true"><param name="allowscriptaccess" value="always"><embed allowfullscreen="true" allowscriptaccess="always" height="385" src="http://www.youtube.com/v/oL700Fnd3ck?fs=1&amp;hl=ro_RO" type="application/x-shockwave-flash" width="480"></embed></object>

It was such a wonderful experience for me, especially since the last squirrel gently touched my index finger while trying to get the walnut from my hand. Its paw was so cold and it felt very smooth and gentile.

I don't know when, but I will definitely go with a bag of walnuts of my own and spend some time there.

Eddy Petri&#537;or: This was a ton of fun

This past saturday I passed by this man in the park. He was calling and feeding some squirrels.

<object height="385" width="480"><param name="movie" value="http://www.youtube.com/v/eA1ASjWRmbs?fs=1&amp;hl=ro_RO"><param name="allowFullScreen" value="true"><param name="allowscriptaccess" value="always"><embed allowfullscreen="true" allowscriptaccess="always" height="385" src="http://www.youtube.com/v/eA1ASjWRmbs?fs=1&amp;hl=ro_RO" type="application/x-shockwave-flash" width="480"></embed></object>

He was kind enough to give me some walnuts to call the squirrels myself. He did this without even me asking.

<object height="385" width="480"><param name="movie" value="http://www.youtube.com/v/oL700Fnd3ck?fs=1&amp;hl=ro_RO"><param name="allowFullScreen" value="true"><param name="allowscriptaccess" value="always"><embed allowfullscreen="true" allowscriptaccess="always" height="385" src="http://www.youtube.com/v/oL700Fnd3ck?fs=1&amp;hl=ro_RO" type="application/x-shockwave-flash" width="480"></embed></object>

It was such a wonderful experience for me, especially since the last squirrel gently touched my index finger while trying to get the walnut from my hand. Its paw was so cold and it felt very smooth and gentile.

I don't know when, but I will definitely go with a bag of walnuts of my own and spend some time there.

22 November 2010

Steve Kemp: This week I've mostly been playing with light

This week I've mostly been experimenting with lighting, most obviously in this silhouette - but also in a clubbing environment. Lighting in clubs is an interesting problem, because in general they are dark and the lighting which is present is usually unhelpfully tinted. Anyway minor problems with white-balance aside this past weekend was a lot of fun - although the Roller Derby match ended in a disappointing lost bout for the Auld Reekie Roller Girls ObQuote: "He'd offer me his blood if he wasn't in a coma." - Thirst (2002)

Sergio Talens-Oliag: The Freaky Wall (Part 1: Why?)

This post and the next to come are about a project I'm doing at work that I've called The Freaky Wall. The project has its origins on the idea that using multiples technologies is better for security; almost all the servers I use are running Debian GNU/Linux and use iptables locally, so when I decided that we had to build new firewalls at work I thought it was a good idea to look at different technologies, that is, a different kernel and firewalling tools. As I wanted to avoid iptables and the Linux kernel my first idea was to go after the free BSD systems (FreeBSD, OpenBSD or NetBSD), and soon realized that pf (the OpenBSD Packet Filter) was the way to go; it has a clean syntax and includes advanced features like CARP and pfsync that allow me to build redundant firewalls. Before going after the standard systems I looked at pfSense a firewall appliance built on top of FreeBSD that uses a PHP interface to do everything. At first it seemed that it was going to be a good option, but soon I felt that I wasn't in control of what the system was doing and I had to change the PHP code to do trivial things (I wanted to configure IP aliases on a CARP interface and it was not possible with the web interface, while it is trivial to do using the standard system configuration files), so I left the idea of using it. The second option was to use OpenBSD directly, as it is the system were pf has been developed. Soon I saw that I was going to be able to do what I wanted with the system, but I missed the Debian's way of installing and upgrading the system and the list of packages available. For different reasons the firewall project was left in a limbo for a little while and when I went back to it I already had to upgrade my test systems to a new OpenBSD release; after reading a little bit about how to upgrade and not liking the idea of doing it I remembered that jordi suggested that if we only want the kernel and the firewall tools the Debian GNU/kFreeBSD port could be an option instead of OpenBSD or FreeBSD. Before trying to install the Debian GNU/kFreeBSD system I saw Robert's post about a Debian installer with ZFS support and I decided to start with it, as the use of ZFS will allow us to use software RAID-1 and snapshots, something we have on almost all our Linux servers (we use software RAID for redundancy and LVM snapshots to be able to do our backups at any time of the day with consistent data, but that is for another post). On my next post I'll explain how I did the initial installation with ZFS, and after that I'll explain the changes I did to the kernel and some of the packages to be able to build a firewall as described on the Firewalling with PF document (that is, I needed pfctl, pflogd, a tcpdump with pflog support, the pf's ftp-proxy, etc.) and on the last document I'll explain how I've configured the firewalls The Debian Way .

Axel Beckert: Useful but Unknown Unix Tools: convmv

Ogg and MP3 files often contain the interpreter and song title in the file name. Which is (besides quoting blanks, etc.) usually no issue if you have English interpreters or song titles. But it may become a character set issue, if you have songs with e.g. French, German, Greek, or Russian interpreters or song titles: They may not be in your preferred character set and therefore don t display properly on your system. Worst case: You just see questions marks. If you face such problems, convmv (man page, Debian package) is your tool, because convmv converts filenames from one encoding to another:
$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 ????
$ convmv -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
Starting a dry run without changes...
mv "./??"       "./ "
No changes to your files done. Use --notest to finally rename the files.
$ convmv --notest -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
mv "./??"       "./ "
Ready!
$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17  
$ 
Of course it can also recursively rename your whole Ogg collection. And it can do it interactively, too:
$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17 ????
$ convmv -i --notest -f latin1 -t utf-8 *
Your Perl version has fleas #37757 #49830 
mv "./??"       "./ " (y/n) y
Ready!
$ ls -lF
total 4
-rw-rw-r-- 1 abe abe 4 2010-11-21 16:17  
$ 
So convmv takes away at least one UTF-8 migration fear. Update 23:03: Nearly forgot, but Mowgli reminded me: There is also convmvfs (Debian package), a FUSE based filesystem which converts file names on access.

Matthew Garrett: Fusion Garage GPL update

Fusion Garage have put up a website containing the source code for the kernel, bootloader and some miscellaneous tools (including their recovery system). As far as I can tell (by inspection - I haven't tried building binaries) it corresponds to what they were distributing, which is excellent. On the other hand, that's all the source they've provided. The pkgs.tgz binary they're providing is the binary packages shipped with the Joojoo, missing their source code. Many of these appear to simply be Ubuntu packages with the gpg signatures stripped, which is a GPL violation though not a terribly interesting one (GPLv2 only allows you to point at someone else's repositories if (a) you're not distributing commercially, (b) they gave you a written offer for the source code and (c) you pass on that written offer - however the source is out there, so it's not something I'm personally terribly upset about). In some other cases, there do seem to be genuine differences. udev, for instance, has a bunch of customisation. And finally there are packages that don't appear in Ubuntu at all, but which do contain GPLed or LGPLed material (such as webkit).

So, the take-home message is that even when given 6 months to provide source code, Fusion Garage are unable to adhere to the licenses of software they distribute. Don't give them any of your money, since unless they magically get significantly better it'll probably end up being spent on defending lawsuits.

Michal &#268;iha&#345;: Strange Monday

Today was quite strange day for me - at the morning I felt quite bad so I took sick day to rest a bit. During afternoon I turned on the computer to see announcement that Novell agrees to be bought in $2.2b deal. As this is company where I work, it of course has attracted my attention. Novell was bought by Attachmate, a company with one quarter of our employees, and over a third of our revenue, what sounds strange. And this is the first time I hear of this company... The information is now very scarce (we as employees were not really given more information than is available publicly), so all what remains to people is to speculate. What does it mean for openSUSE? What is the "certain intellectual property" transferred to CPTN Holdings LLC (owned by Microsoft Corp.)? And some people just comment on current situation of Novell and some of it's products :-).

Filed under: Suse 1 comments Flattr this!

Petter Reinholdtsen: Lenny->Squeeze upgrades of the Gnome and KDE desktop, now with apt-get autoremove

Michael Biebl suggested to me on IRC, that I changed my automated upgrade testing of the Lenny Gnome and KDE Desktop to do apt-get autoremove when using apt-get. This seem like a very good idea, so I adjusted by test scripts and can now present the updated result from today: This is for Gnome: Installed using apt-get, missing with aptitude
apache2.2-bin aptdaemon baobab binfmt-support browser-plugin-gnash cheese-common cli-common cups-pk-helper dmz-cursor-theme empathy empathy-common freedesktop-sound-theme freeglut3 gconf-defaults-service gdm-themes gedit-plugins geoclue geoclue-hostip geoclue-localnet geoclue-manual geoclue-yahoo gnash gnash-common gnome gnome-backgrounds gnome-cards-data gnome-codec-install gnome-core gnome-desktop-environment gnome-disk-utility gnome-screenshot gnome-search-tool gnome-session-canberra gnome-system-log gnome-themes-extras gnome-themes-more gnome-user-share gstreamer0.10-fluendo-mp3 gstreamer0.10-tools gtk2-engines gtk2-engines-pixbuf gtk2-engines-smooth hamster-applet libapache2-mod-dnssd libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libart2.0-cil libboost-date-time1.42.0 libboost-python1.42.0 libboost-thread1.42.0 libchamplain-0.4-0 libchamplain-gtk-0.4-0 libcheese-gtk18 libclutter-gtk-0.10-0 libcryptui0 libdiscid0 libelf1 libepc-1.0-2 libepc-common libepc-ui-1.0-2 libfreerdp-plugins-standard libfreerdp0 libgconf2.0-cil libgdata-common libgdata7 libgdu-gtk0 libgee2 libgeoclue0 libgexiv2-0 libgif4 libglade2.0-cil libglib2.0-cil libgmime2.4-cil libgnome-vfs2.0-cil libgnome2.24-cil libgnomepanel2.24-cil libgpod-common libgpod4 libgtk2.0-cil libgtkglext1 libgtksourceview2.0-common libmono-addins-gui0.2-cil libmono-addins0.2-cil libmono-cairo2.0-cil libmono-corlib2.0-cil libmono-i18n-west2.0-cil libmono-posix2.0-cil libmono-security2.0-cil libmono-sharpzip2.84-cil libmono-system2.0-cil libmtp8 libmusicbrainz3-6 libndesk-dbus-glib1.0-cil libndesk-dbus1.0-cil libopal3.6.8 libpolkit-gtk-1-0 libpt2.6.7 libpython2.6 librpm1 librpmio1 libsdl1.2debian libsrtp0 libssh-4 libtelepathy-farsight0 libtelepathy-glib0 libtidy-0.99-0 media-player-info mesa-utils mono-2.0-gac mono-gac mono-runtime nautilus-sendto nautilus-sendto-empathy p7zip-full pkg-config python-aptdaemon python-aptdaemon-gtk python-axiom python-beautifulsoup python-bugbuddy python-clientform python-coherence python-configobj python-crypto python-cupshelpers python-elementtree python-epsilon python-evolution python-feedparser python-gdata python-gdbm python-gst0.10 python-gtkglext1 python-gtksourceview2 python-httplib2 python-louie python-mako python-markupsafe python-mechanize python-nevow python-notify python-opengl python-openssl python-pam python-pkg-resources python-pyasn1 python-pysqlite2 python-rdflib python-serial python-tagpy python-twisted-bin python-twisted-conch python-twisted-core python-twisted-web python-utidylib python-webkit python-xdg python-zope.interface remmina remmina-plugin-data remmina-plugin-rdp remmina-plugin-vnc rhythmbox-plugin-cdrecorder rhythmbox-plugins rpm-common rpm2cpio seahorse-plugins shotwell software-center system-config-printer-udev telepathy-gabble telepathy-mission-control-5 telepathy-salut tomboy totem totem-coherence totem-mozilla totem-plugins transmission-common xdg-user-dirs xdg-user-dirs-gtk xserver-xephyr
Installed using apt-get, removed with aptitude
cheese ekiga eog epiphany-extensions evolution-exchange fast-user-switch-applet file-roller gcalctool gconf-editor gdm gedit gedit-common gnome-games gnome-games-data gnome-nettool gnome-system-tools gnome-themes gnuchess gucharmap guile-1.8-libs libavahi-ui0 libdmx1 libgalago3 libgtk-vnc-1.0-0 libgtksourceview2.0-0 liblircclient0 libsdl1.2debian-alsa libspeexdsp1 libsvga1 rhythmbox seahorse sound-juicer system-config-printer totem-common transmission-gtk vinagre vino
Installed using aptitude, missing with apt-get
gstreamer0.10-gnomevfs
Installed using aptitude, removed with apt-get
[nothing]
This is for KDE: Installed using apt-get, missing with aptitude
ksmserver
Installed using apt-get, removed with aptitude
kwin network-manager-kde
Installed using aptitude, missing with apt-get
arts dolphin freespacenotifier google-gadgets-gst google-gadgets-xul kappfinder kcalc kcharselect kde-core kde-plasma-desktop kde-standard kde-window-manager kdeartwork kdeartwork-emoticons kdeartwork-style kdeartwork-theme-icon kdebase kdebase-apps kdebase-workspace kdebase-workspace-bin kdebase-workspace-data kdeeject kdelibs kdeplasma-addons kdeutils kdewallpapers kdf kfloppy kgpg khelpcenter4 kinfocenter konq-plugins-l10n konqueror-nsplugins kscreensaver kscreensaver-xsavers ktimer kwrite libgle3 libkde4-ruby1.8 libkonq5 libkonq5-templates libnetpbm10 libplasma-ruby libplasma-ruby1.8 libqt4-ruby1.8 marble-data marble-plugins netpbm nuvola-icon-theme plasma-dataengines-workspace plasma-desktop plasma-desktopthemes-artwork plasma-runners-addons plasma-scriptengine-googlegadgets plasma-scriptengine-python plasma-scriptengine-qedje plasma-scriptengine-ruby plasma-scriptengine-webkit plasma-scriptengines plasma-wallpapers-addons plasma-widget-folderview plasma-widget-networkmanagement ruby sweeper update-notifier-kde xscreensaver-data-extra xscreensaver-gl xscreensaver-gl-extra xscreensaver-screensaver-bsod
Installed using aptitude, removed with apt-get
ark google-gadgets-common google-gadgets-qt htdig kate kdebase-bin kdebase-data kdepasswd kfind klipper konq-plugins konqueror ksysguard ksysguardd libarchive1 libcln6 libeet1 libeina-svn-06 libggadget-1.0-0b libggadget-qt-1.0-0b libgps19 libkdecorations4 libkephal4 libkonq4 libkonqsidebarplugin4a libkscreensaver5 libksgrd4 libksignalplotter4 libkunitconversion4 libkwineffects1a libmarblewidget4 libntrack-qt4-1 libntrack0 libplasma-geolocation-interface4 libplasmaclock4a libplasmagenericshell4 libprocesscore4a libprocessui4a libqalculate5 libqedje0a libqtruby4shared2 libqzion0a libruby1.8 libscim8c2a libsmokekdecore4-3 libsmokekdeui4-3 libsmokekfile3 libsmokekhtml3 libsmokekio3 libsmokeknewstuff2-3 libsmokeknewstuff3-3 libsmokekparts3 libsmokektexteditor3 libsmokekutils3 libsmokenepomuk3 libsmokephonon3 libsmokeplasma3 libsmokeqtcore4-3 libsmokeqtdbus4-3 libsmokeqtgui4-3 libsmokeqtnetwork4-3 libsmokeqtopengl4-3 libsmokeqtscript4-3 libsmokeqtsql4-3 libsmokeqtsvg4-3 libsmokeqttest4-3 libsmokeqtuitools4-3 libsmokeqtwebkit4-3 libsmokeqtxml4-3 libsmokesolid3 libsmokesoprano3 libtaskmanager4a libtidy-0.99-0 libweather-ion4a libxklavier16 libxxf86misc1 okteta oxygencursors plasma-dataengines-addons plasma-scriptengine-superkaramba plasma-widget-lancelot plasma-widgets-addons plasma-widgets-workspace polkit-kde-1 ruby1.8 systemsettings update-notifier-common
Running apt-get autoremove made the results using apt-get and aptitude a bit more similar, but there are still quite a lott of differences. I have no idea what packages should be installed after the upgrade, but hope those that do can have a look.

Rapha&#235;l Hertzog: How Ubuntu builds up on Debian

I have been asked how Ubuntu relates to Debian, and how packages flow from one to the other. So here s my attempt at clarifying the whole picture.

Where do the packages come from? Most packages are created by Debian contributors and they are uploaded in Debian unstable (or Debian experimental). New packages are reviewed by the Debian ftpmasters before being accepted in the official archive. The packages are held in the NEW queue until the review is over, and the time spent there varies between a few hours and a few months (usually they are processed within one week or two). Ubuntu imports all the official Debian packages, but they also add some packages of their own. About 7% of the Ubuntu packages are third-party software that have been packaged for Ubuntu but not for Debian. What are the changes made by Ubuntu? From all the source packages coming from Debian, 17% have additional changes made by Ubuntu. Many of them are part of the main repository, which is actively maintained by Canonical and Ubuntu core developers. The universe repository is usually closer to the official Debian packages. Many of the changes made by Ubuntu are the results of the decisions taken during the Ubuntu Developer Summit in order to reach specific goals: provide a better user interface, offer faster boot times, become a better platform for third-party software developers, offer a good integration with their online services (Launchpad, Ubuntu One), etc. Other changes are simply the result of fixing bugs reported by Ubuntu users. Note that even non-modified source packages will result in different binary packages for Ubuntu. That s because Ubuntu has made changes to the build environment. They only support Intel-based computers with a 686-class (or newer) CPU, they enable some compiler options that Debian doesn t, etc. And all binary packages are modified by a program called pkgbinarymangler. Ubuntu s release cycle and the relation with Debian Ubuntu releases every 6 months (that s what time based releases is about). Debian has a very different schedule. How does Ubuntu manage to reuse Debian s work? Ubuntu imports packages from Debian unstable (even experimental sometimes) to get the newest packages. If the Ubuntu package already has Ubuntu-specific changes, they merge their changes in the updated Debian package. Otherwise the Debian package is simply grabbed and rebuilt in Ubuntu. This works well because Debian unstable is much more usable than the name suggests. And this process only goes on during the first 2 months of the cycle (until the Debian Import Freeze), so there s plenty of time afterward to fix the biggest problems. In the third and fourth month, it s still possible to pick updated packages from Debian but it must be requested by a developer, it won t be done automatically. At the end of the fourth month, the feature freeze is put in place. The 2 months left are dedicated to bug fixing and polishing the distribution. There are various sub-freezes that happen in this period, you can check the Natty release schedule as an example. Picking updated packages from Debian is now the exception, it will only be allowed if the update on the Debian side is a bug-fix only release. Credits: some figures taken from a talk of Lucas Nussbaum, they were collected based on the packages available in the Lucid Lynx release of Ubuntu. Click here to subscribe to my newsletter and get my monthly update on what s going on in Debian and Ubuntu.

15 comments Liked this article? Click here. My blog is Flattr-enabled.

Debian News: Brief Updates: Firebird 3, Iceweasel 4.0 beta, google code-in and DPL interview

Petter Reinholdtsen: Migrating Xen virtual machines using LVM to KVM using disk images

Most of the computers in use by the Debian Edu/Skolelinux project are virtual machines. And they have been Xen machines running on a fairly old IBM eserver xseries 345 machine, and we wanted to migrate them to KVM on a newer Dell PowerEdge 2950 host machine. This was a bit harder that it could have been, because we set up the Xen virtual machines to get the virtual partitions from LVM, which as far as I know is not supported by KVM. So to migrate, we had to convert several LVM logical volumes to partitions on a virtual disk file. I found a nice recipe to do this, and wrote the following script to do the migration. It uses qemu-img from the qemu package to make the disk image, parted to partition it, losetup and kpartx to present the disk image partions as devices, and dd to copy the data. I NFS mounted the new servers storage area on the old server to do the migration.
#!/bin/sh
# Based on
# http://searchnetworking.techtarget.com.au/articles/35011-Six-steps-for-migrating-Xen-virtual-machines-to-KVM
set -e
set -x
if [ -z "$1" ] ; then
    echo "Usage: $0 <hostname>"
    exit 1
else
    host="$1"
fi
if [ ! -e /dev/vg_data/$host-disk ] ; then
    echo "error: unable to find LVM volume for $host"
    exit 1
fi
# Partitions need to be a bit bigger than the LVM LVs.  not sure why.
disksize=$( lvs --units m   grep $host-disk   awk ' sum = sum + $4  END   print int(sum * 1.05)  ')
swapsize=$( lvs --units m   grep $host-swap   awk ' sum = sum + $4  END   print int(sum * 1.05)  ')
totalsize=$(( ( $disksize + $swapsize ) ))
img=$host.img
#dd if=/dev/zero of=$img bs=1M count=$(( $disksize + $swapsize ))
qemu-img create $img $ totalsize MMaking room on the Debian Edu/Sqeeze DVD
parted $img mklabel msdos
parted $img mkpart primary linux-swap 0 $disksize
parted $img mkpart primary ext2 $disksize $totalsize
parted $img set 1 boot on
modprobe dm-mod
losetup /dev/loop0 $img
kpartx -a /dev/loop0
dd if=/dev/vg_data/$host-disk of=/dev/mapper/loop0p1 bs=1M
fsck.ext3 -f /dev/mapper/loop0p1   true
mkswap /dev/mapper/loop0p2
kpartx -d /dev/loop0
losetup -d /dev/loop0
The script is perhaps so simple that it is not copyrightable, but if it is, it is licenced using GPL v2 or later at your discretion. After doing this, I booted a Debian CD in rescue mode in KVM with the new disk image attached, installed grub-pc and linux-image-686 and set up grub to boot from the disk image. After this, the KVM machines seem to work just fine.

Gintautas Miliauskas: Verdi's Requiem

The choir I sing in (Pro Musica) will be performing on January 22nd in Bremen, Germany. Together with two local choirs we will perform Verdi's Requiem, which is absolutely fabulous. See for yourself:
<object height="385" width="480"><param name="movie" value="http://www.youtube.com/v/TVjDP0vlem4?fs=1&amp;hl=lt_LT&amp;color1=0x3a3a3a&amp;color2=0x999999"> <param name="allowFullScreen" value="true"> <param name="allowscriptaccess" value="always"> <embed allowfullscreen="true" allowscriptaccess="always" height="385" src="http://www.youtube.com/v/TVjDP0vlem4?fs=1&amp;hl=lt_LT&amp;color1=0x3a3a3a&amp;color2=0x999999" type="application/x-shockwave-flash" width="480"></embed></object>
(Also try Solti's version for better sound quality, but no video.)
It's going to be a busy month preparing for the concert. Verdi's melodies are not complicated, but he's crazy with dynamics: I think it's the first time I've seen a quintuple pianissimo (ppppp) or a quadruple fortissimo (ffff) in sheet music.

21 November 2010

Axel Beckert: Useful but Unknown Unix Tools: netselect

Ever wondered which mirror of your favourite Linux distribution is the fastest at your location? Check it with netselect (code at GitHub). It checks for the number of hops and ping times to given hosts and tells you which one is the fastest of them:
# netselect -vv ftp.de.debian.org ftp2.de.debian.org \
                ftp.ch.debian.org ftp.nl.debian.org ftp.debian.org
Running netselect to choose 1 out of 5 addresses.
.......................................................
ftp.de.debian.org                       25 ms  16 hops   90% ok ( 9/10) [   72]
ftp2.de.debian.org                      17 ms  17 hops   90% ok ( 9/10) [   51]
ftp.ch.debian.org                        0 ms   3 hops   90% ok ( 9/10) [    0]
ftp.nl.debian.org                       22 ms  15 hops   90% ok ( 9/10) [   62]
ftp.debian.org                          22 ms  15 hops   90% ok ( 9/10) [   60]
    0 ftp.ch.debian.org
And if you re too lazy to optimize your sources.list with netselect manually, just use the netselect-apt package. It will do it for you.

Gintautas Miliauskas: Laptop trouble

My trusty somewhat-old laptop has been under the weather lately. It will not boot when it's cold, and it's fairly picky. ~20 C seems too cold already; so I have to heat it up at little.
Now that winter is coming, this needs to be done almost every morning, and that is extremely annoying. Does anybody have an idea of what could be wrong? It seems that the problem is not just a loose connection, as the laptop goes up in stages:

1) when the power button is pressed the power LED will blink for a split-second and nothing will happen,
2) the LED will light up for slightly longer, but the laptop still will not boot,
3) the laptop starts booting but shuts off spontaneously after while on the BIOS screen,
4) the laptop goes through the BIOS screen, but shuts down before showing the boot menu.
Past that, apparently the CPU generates enough heat for stable operation.
This is fairly complicated behaviour for a hardware problem. I wonder, maybe the power supply is at fault? Any ideas, anyone? This is a good excuse to buy a new laptop, but I would rather not splurge right before Christmas season.

Next.

Previous.