Search Results: "ron"

27 December 2007

Miriam Ruiz: SMV Video Format

Some small portable MP3 players also allow reproducing images and videos (does anyone know why they call them MP4 players?), but they need the latter to be in some funny format called SMV (for devices with SigmaTel chipset. Other devices use different formats, such as AMV for devices with Actions chipset, MVS for those with Sunplus chipsets, MTV, DMV or ASF). SMV seems to stand for “SigmaTel Motion Video”, and was developed by SigmaTel Inc to allow users to transform common PC video files and watch them on their portable multimedia players (PMP). I’ve been trying to find out how to convert videos to this format for a while, to be able to watch them in a small gadget I bought some time ago, but haven’t been able to do so from Linux yet. All the converters seem to be privative, and just for Windows, although they seem to work with wine according to some reports I’ve read on the net. I’m not putting any closed-source software in my lappy anyway, so that’s not an option for me. I recently found a web page of someone developing a conversion tool for SMV called smvconv, based on mplayer and ffmpeg and released under a MIT/X11 license. It’s just a small C program and a shell script, and seems to be in a very preliminary stage, but the most important part is that he has also published the structure of SMV files, which seem to be, in fact, quite a simple format composed of some headers, a WAV audio file and some JPEG frames: An SMV file has three sections, concatenated together:
  1. An IMA ADPCM .wav file, with header.
  2. An SMV header.
  3. A number of .jpeg files, with headers, each prepended by a 24-bit (3-byte) little-endian value representing the size, and buffered at the end with nulls, so that they are at predictable locations (the jpeg modulo). SMV is a constant-bit-rate format, but the underlying format (jpeg) is not. Proprietary encoders -according to smvconv’s author- probably make up for this by encoding at various qualities until a jpeg of the proper size is made.
The SMV header is a magic string in ASCII (”SMV002000″) followed by a number of 24-bit (3-byte) little-endian values representing the width, height, header length (in 3-byte words, starting with some magic “002″), some apparently constant value (1), the jpeg modulo, fps, number of frames. some apparently constant values (a 1 and a 0), the number of frames per jpeg, and 4 padding values (65793 == 0×010101). The default qualities seem to be 35kBps (”High”), 25kBps (”Medium”) and 16kBps (”Low”). Lets hope to have some working converter soon, so we can use this feature in those cheap players. I don’t remember where I put the USB connector for mine, but as soon as I find it -it seems to be non-standard, so no way to replace it with a new one, AFAIK- I’ll start experimenting with it.

26 December 2007

MJ Ray: Cyclocross: Hofstade

Hope you had a good Christmas Day if you celebrate it. The cycling starts again already: RAI Sport Sat (free-to-air 13e) have highlights of today’s seventh Cyclocross World Cup event from Hofstade Belgium at 22:30 CET today (Wednesday). No idea whether it will be repeated… my recent predictions seem to have been a bit wrong. Hofstade is the sixth elite men’s event to actually happen, after the cancellation of their race at the Milan round. Sven Nys of Belgium leads the elite men’s rankings by almost 500 points from compatriots Bart Wellens and Klaas Vantornout. Daphny van den Brand of the Netherlands leads the elite women by 340 points from USA’s Katherine Compton. 300 points for a win, then 260, 220, 190, 170, and so on. The contests aren’t over yet. Update: It’s on again 22:30 CET today (Thursday) and repeated at 11:10 tomorrow (Friday) according to teletext.

25 December 2007

Adeodato Sim : Moving to ikiwiki

As many others, I’ve moved this blog to ikiwiki. In fact, I’ve moved my whole 3-page website to ikiwiki, and it only felt natural to migrate the blog as well (I may miss hobix’s webpage, though). Anyway, this gives me a bit more flexibility when creating the RSS feeds, so I’ve now created a separate feed for Planet Debian (which won’t have only Debian-related content, mind you), and left the old RSS URL to include all posts, e.g. some more personal ones, or some that I’d find unsuitable for the planet. If you’re interested, here is the full feed. I run ikiwiki in my laptop to generate static html, and rsync the generated output to the webserver. I have a Bazaar branch in my laptop as well, but it doesn’t interact with ikiwiki in any way. Finally, I’m very grateful that Joey committed this minutes after me mentioning on IRC, and I’m very grateful as well to whoever wrote the typography plugin. Migrating the old blog wasn’t hard, just tedious and boring, particularly since I didn’t want to keep using Textile, but migrate to Markdown. I switched off usedirs to keep old URLs working (I’d rather not mess with redirections), and used the date option of the meta plugin, so that the “Posted” information for old entries when inlining would be accurate. This still left the “Last modified” timestamp in the individual pages wrong, so I just used a script that would change the mtime of the entries in the filesystem to their creation time. One wart I’ve found is that I’m very used to having both a filename and a title for each entry, and have the filename just present in the navigation bar of the browser, and the title shown in big inside the page. You can do this in ikiwiki by using the title option of the meta plugin, and giving a H1 element in the entry content. However, this makes the H1 show in the RSS feed, which I had always noticed in Zack’s blog, and always found inconvenient. Oh well. It also makes for the same title to appear twice in the default [[inline]] page; to solve it, I just hide with CSS the H1 there — zack, e.g., does not use [[meta title]], and aligns the permalink to the right. Update: So I was washing the dishes and it struck me: maybe with a conditional! And, effectively, it is possible, given that there is a convenient included test. So now I start my entries with: 
  [[meta title="This is the long title"]]
  [[if test="!included()"
       then="# This is the long title"]]
Yay! Update (2007-12-27): Lars does it differently, by only providing [[meta title]], and modifying the page template to include: 
  <h1><TMPL_VAR TITLE></h1>
Update (2007-12-29): I finally decided, at least for now, that is cleaner not having the title duplicated in two lines, and went with a template. I didn’t want the <h1>title</h1> bit in all pages, though, and Joey pointed out that I could use a different template for blog entries using the pagetemplate_plugin.

Frans Pop: Visually beautiful music

Most audio players offer visualizations. Unfortunately most of them either are just plain boring or don't really visualize the music very well. KDE's amarok has one that is really excellent: jakdaw. Jakdaw does a good job of both following and representing the intensity of the music. And it generates an amazing display of colors. But even with a good visualization a lot of music is not all that interesting to watch. Over the last month or so I've occasionally been "looking" through my music collection. The visualizations are surprisingly varied for different artists. And though a lot of them are nice to watch, some artists consistently produce much more varied and artistic displays than others. To some degree it is predictable, but not always. Quite often the visualization is a lot less interesting than you'd expect and especially some slower or quieter songs produce much more interesting displays than you'd expect. A blanket of sound does not produce a very interesting display, but strong accents and mood changes do. As is often the case in art, the real challenge in composing music seems to be leaving things out and to have individual instruments play their own tune instead of having everything move together. The two artists that really stand out so far are Björk and Faithless. Both consistently produce visualizations that are at times stunningly beautiful and also show a lot more detail in the music than others. For Björk It's Oh So Quiet (from Post) and Violently Happy (from Debut) are very nice, but two songs that really stand out are Hunter (from Homogenic) and Hyper Balad (from Post). For Faithless songs like We Come 1 and Tarantula (from Outrospective), and Salva Mea and Insomnia (from Reverence) are truly worth watching.

Ben Armstrong: Fr-eee-dom: roaming on the bus with wpa_supplicant

As I wrote earlier, I bought an Eee PC to use on the bus. Initially, I only used it offline to do some Debian packaging and blogging, but I soon discovered that I could do a few things online on the many open networks on my daily commute. While the connections are normally brief in good traffic conditions, even a few seconds here and there is enough to participate in irc discussions. Manually connecting to each got old very quickly. So, to automate those connections I settled on wpa_supplicant, (wpasupplicant is the Debian package name,) which is quite easy to set up in roaming mode, as outlined in /usr/share/doc/wpasupplicant/README.modes.gz. All you need is the following in /etc/network/interfaces: 
allow-hotplug ath0
iface ath0 inet manual
        wpa-driver wext
        wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
Then create your wpa_supplicant.conf as follows: 
ctrl_interface=/var/run/wpa_supplicant
network= 
     key_mgmt=NONE
That’s it. You can now roam from one open network to the next and the supplicant will connect you to each one in turn. I have used this to reconnect to my screen session each time the bus slows down for a stop or gets stuck in traffic to carry on conversations on irc for the whole length of the commute. If you also connect to networks requiring authentication, you’ll need to add a “network” clause for each network. See /usr/share/doc/wpasupplicant/examples/ for help with that, particularly the annotated conf file, README.wpa_supplicant.conf.gz. For finer control over the process, I use sudo wpa_cli so i can: The only things that I haven’t figured out how to do yet are to make the process of switching from one network to the next a bit faster (it normally takes about a minute; precious seconds of online time are lost when the bus is in motion,) and to blacklist certain essids, e.g. commercial hotspots that make you pay before they’ll route your packets to the Internet. For the first problem, I have already tweaked my dhcp settings, which helps a bit. For the second, I tried adding network blocks for specific networks and setting them to a lower priority, but that doesn’t work because then the ‘catch all’ open network block kicks in and picks them up anyway.

Clint Adams: Modern romance

As you know, he said, I am dating other women, but you are among the front-runners. You have a month, she retorted. A month he sputtered. I only need like a day!

24 December 2007

Steve Kemp: I saw momma kissing santa claus

Well I've been in Devon for the past week, and will remain here for another week. During that time I've averaged about ten minutes of online time a day! So far things are going well, but it will be wierd spending time with another family over Christmas. Still I've been vaguely productive. I've released a new version of chronicle - which has the CGI support for leaving comments upon my entries. (TODO: Reinstate some previous comments for Martin Krafft) One thing I'd not anticipated was the amount of spam I'd start recieving. The peak so far was 40+ comments a day, all with random URLs. I deleted them manually for the first day - but now I've written a shell script or two to classify comments via the spamfilter I use for handling email. I'm not 100% sure I should have done that. I suspect that over time I will find better results if I actually have a distinct "blog spam" and "blog ham" corpus - rather than risk confusion over "email" and "blog" texts to classify. Still I shall wait and see. The only thing that I think I want to do now is add a bit more fine control over the comment process, so the choice is not just between comments being globally on, or globally off. A nice middle-ground where I could say "Comments enabled for entries posted within the past two weeks or so". Anyway thats my quota for today. Now I have to walk the family dog ..

Martin Michlmayr: Joining HP

I joined HP's Open Source and Linux Organization (OSLO) at the beginning of the month as an Open Source Community Expert. I spent the last three weeks in Ft. Collins for some training and to get to know the team, but normally I'll be working from home in Europe. The main focus of my work will be governance issues related to the deployment of FOSS in enterprise environments. However, I've already started to get involved in a number of other areas I'm interested in, such as community relations. I'm pretty excited about my new position because it is a good fit for my skills and interests.

Adam Rosi-Kessel: Jason s Lesson s Learned About the Legal Academy and Getting In

Pelican has decided to abandon his efforts to scale the walls of the legal academy and proposes this simple eight-step program for others who would follow in his footsteps:
  1. Don t do interdisciplinary work. The legal academy doesn t know what to make of it unless it is economics.
  2. Don t go to a school without a law review or grades. I did and it was a huge problem.
  3. Get on law review, clerk, write.
  4. Check the faculty listings at most law schools. Go to the most often listed schools: Harvard, Yale, etc. It does matter as law faculty select their own, usually. I was told by a faculty member as a 1L expressing interest in the academy that I should transfer immediately to Harvard if possible. I didn t.
  5. If you think you want to be a legal academic, look at what is on the FAR form in your first or second year. Orient your academic career to produce a good looking FAR.
  6. Remarkably, the legal academy does not care about your ability to raise research money or bring in grants.
  7. Don t publish in interdisciplinary journals. Publish only in law reviews writing only dense and impenetrable texts.
  8. Demonstrated impact of your work in policy or law is not relevant.
This critique could be extended broadly to much of the academy (not just law schools). I had once thought I might like to teach law, but now I can only really see myself as a clinician. Two contradictory academic trends: Technorati Tags: , , , , , , , ,

Share This

23 December 2007

Christian Perrier: Moon-Mars occultation

Full moon is bright in the skies today and tomorrow. Moreover, it will share the skies with Mars, reaching its opposition point. So, for everybody around, just go outside a few hours after sunset and enjoy. Tomorrow (Dec. 24th), some will even enjoy an occultation with the Moon passing in front of Mars. See this page for occultation details and locations. In short: state of Washington, USA, British Columbia (CA) and other parts of Western Canada, Greenland, Scotland, Germany, Czech Republik, Hungary, Serbia, Greece, Turkey, Lebanon, Israel. Then some parts of Armenia, Iran, Uzbekistan, Russia (Siberia) and Aleoutians.

Russell Coker: Contract Pay Rates

In a comment on my post about Bad Project Management [1] Don Marti [2] says “the more money you charge, the less of your time people waste, and the more seriously they take you” and “you can affect the client more, whether you’re peddling idiocy or wisdom“. That’s a nice theory, however I don’t recall seeing an example of it anywhere that I have worked. There are two broad categories of employers, large and small. Large companies have HR departments etc who decide on rough pay scales. Once the budget is approved (by senior management and HR) for someone’s contract fees or salary the amount of money seems to be of little concern to the line manager. They have a group of N people reporting to them who are all paid by money that comes from some mysterious place (with no direct connection to the quality or quantity of work done) and therefore the amount of money paid is not very relevant. Often team-leaders don’t even know how much the people who report to them are being paid! Note that this applies to people who have a similar job title, if I am paid twice as much as another sys-admin in a corporate environment I don’t expect to get more attention, but someone who has a more senior job title (or in the case of the UK has been to a more prestigious school or has a better accent) may get more attention. Small companies are rarely in a position to be silly in this way. This doesn’t mean that the most highly paid person gets the most attention, it’s more a matter of making a logical case. Most of the small companies I have worked for have been run in a sensible manner, and my ideas have often received more attention than those of others because I have had a better technical background than my colleagues. However in some cases my suggestions have received less attention because I was receiving a higher pay rate. The reason is that I was being paid a rate that the company could not afford in the long-term and therefore the people who would work there in future years would have to maintain the systems in question which meant that their ability to do such maintenance was often considered to be more important than my ideas about the best way to design the system. I have worked for a small company that had no financial pressure to do sensible things (they had some sweet deals going with major corporations which essentially gave them money for nothing). They had chaotic management which they could get away with because there were no immediate consequences for doing silly things (such as not forcing their clients to have a method of backing up servers that they installed). I can’t claim that Don’s advice in this regard never applies, but my observations of situations where it didn’t apply lead me to believe that it’s applicable to some situations in limited areas and is not a general rule. Don also refers to “a high number that you would quote to a client you didn’t want in order to get them to go away“. This is an issue that causes a lot of disagreement. One consultant I respect advises people to never make such quotes, if you don’t want a job then say that you are not available - one reason is that if a position is so bad that you want the potential client to go away then you don’t want to be tempted to reconsider if they agree to pay you a silly amount of money, another is that it may be considered ethically wrong to make an offer is not genuine. Another consultant I respect claims that his greatest career success was when he turned up to an interview in casual clothes, explained to the potential client that he would never wear a suit and demanded an excessive amount of money. The client agreed to such terms and he then went on to charge other clients similar amounts of money and continued to refuse to wear a suit. I won’t claim that it’s wrong to demand excessive amounts of money to make an unwanted potential client go away and I will not rule out the possibility of doing so at some future time. But I think it should be noted that the practice is controversial and not referred to as something that anyone might do. It is something that I don’t recall doing at any previous time in my career (so I haven’t done it in a formal manner at least - I may have done so at the informal discussion phase and forgotten). Please note that I encourage you to read my previous post with Don’s comment in it’s original context to avoid any risk of misinterpretation. The purpose of this post is not to disagree with Don (I agree with him almost all of the time) but to disagree with some very commonly held beliefs which Don represented in his comment.

Share This

Christian Perrier: News from D-I i18n

Some more important things happened this week for Debian Installer i18n: All this work is meant to give some more push to D-I localization effort that paced for some time last months while I focused my attention on some other areas. It seems to show that things never work without someone to care about them, apparently..:-)...so, existing translators, watch out, I'm back on D-I i18n and will kick your asses if you don't keep your translations up to date..:-)

22 December 2007

Petr Rockai: on blogging

The last time I have blogged, few people told me that it sounds somewhat patronizing or such (especially that it may sound so to people that do not know me). Well, I have to admit that it is hard to write in a decent tone — the intention and motivation behind the text is oftentimes hard to express. Certainly, better writing skills would probably help — or choosing easier topics. But that is how I am — and afterall, you never learn much when you only do things you already know how to do. So please, dear reader, take no offense in my blog… If you know me, you can probably see through my writing and understand what I mean. If you do not and still enjoy it, the better. And if you do not, please accept my apology — but in that case, it probably was not meant for you. Yes, I know the blog is aggregated in a few places, and I will probably have to do something about this — aggregate only subsets of the blog, appropriate for each of the aggregators. But pretty please, bear with me, I am lazy and it always took me ages to fix even trivial problems.

Daniel Leidert: docbook-defguide - solving performance and timing issues with native code

Some days ago I wrote down my experiences with packaging docbook-defguide. The main (remaining) issues I mentioned were the resources and the time the package needs to build. Even on an AMD X2 4600+ with 6GB of RAM it needs 7-8 hours. Today I met with Torsten Werner. He mentioned, that there are some move JVMs I could try. So I tested alternatives to GIJ this night. I found this short summary about free JVMs, which was some kind of interesting. I began with cacao, which seemed to be fast, but it was killed very early in the build process with an java.lang.OutOfMemory error. Even playing around with the -Xms and -Xmx switches in buildtools/saxon.sh did not help. So I dropped cacao from the list. Seems both cacao and kaffe create similar problems here and are not suitable for building the package. Second alternative I tried was sablevm. It directly throw out some warnings or errors so I directly dropped it too. Next JVM was jamvm. But it was as slow as GIJ. So I dropped it from the list of alternatives too. Then I found an interesting statement in the article I linked somewhere above. The author said, that his perfomance test time with GCJ/GIJ reduced from 433 to 9 seconds, when he compiled his application into a native executable. So I took a fast look through the docbook-defguide build dependencies and found, that Debian already provides a natively compiled Xerces package libxerces2-java-gcj. But there were no packages for libsaxon-java, libxml-commons-resolver1.1-java and docbook-xsl-saxon. So short and dirty: I downloaded the source for these packages, added the necessary stuff to get natively compiled packages too, built and installed them. Fortunately packages with native code already exist (JAXP 1.3 and Xerces) for their dependencies. And what should I say: Now building the TDG needs less then 512MB RAM and it builds in around an hour … even on my system. I will ask the Debian Java maintainers to add -gcj packages for Saxon and XML-Commons and fix my own docbook-xsl-saxon package. This will hopefully help maintaining docbook-defguide.

21 December 2007

Ross Burton: It's That Time Of The Year Again

It starts off just as you'd expect.
Yes, it s that time of year again. No sooner does an important traditional religious holiday roll around than the PC-brigade feel the need to strip-mine it of its original significance, just so s no-one s feeling get upset.
However, it doesn't quite continue as you were thinking.
It is the Christians who have the most gall of all, daring to attach the name of some first-century Palestinian to a once-proud British festival. Yule I can live with, despite its being a continental bastardisation of our British pronunciation Geola , but Christmas is just wrong. You even have to mispronounce Christ to say it.
An excellent piece of satire from Nathaniel Tapley on Liberal Conspiracy, which is well worth a read for a good laugh. NP: Giant Steps, John Coltrane

Andrew Pollock: [tech] Waiting for a software RAID-1 to resync is like watching paint dry

I'm so annoyed with myself. So today was the day I was going to upgrade the disks in daedalus. I'd finally managed to convince Dell to sell me two new 140Gb disks to replace to two 70Gb disks that were fast approaching capacity. I'd organised for access to the datacentre from 10am. I arrived early. I didn't manage to get into the datacentre until about 11am, because they were having problems remotely unlocking the door, and I had to wait until an air conditioning repair guy turned up, who was fixing some air conditioning problem. I hadn't had a lot of time to plan how I was going to do the upgrade, but I figured it was going to be pretty straightforward. Each disk is partitioned into three partitions, all of which are mirrored using software RAID. The first partition is the root filesystem, which is just an ext3 filesystem, mirrored. The second partition is swap, also mirrored, and the third partition is an LVM physical volume. The first two partitions are only a couple of gigabytes. So the basic plan was to remove the second disk (disk 1, aka /dev/sdb), put in the new bigger disk, load the partition table from the original disk 1, resync the three mirrors, and then boot off the new disk 1, and rinse and repeat with disk 0. This was all going swimmingly, when right at the end (in fact after the third partition finished syncing according to the kernel), /dev/sda decided it had a few unreadable sectors. This made the software mirroring have a dummy-spit, and decide it needed to restart the syncing from scratch. I'd seen this before with a bad disk. So I wishfully hoped it was something to do with the new disk being bigger than the old disk, so tried to roll back to the original disk 1. As was pretty much expected, disk 0 exhibited the same problems at the same point. So the problem was disk 0, not the new disk 1. So it seemed like I just managed to pick the wrong half of the mirror to work from. I thought I'd try manually dd'ing the third partition from /dev/sda to /dev/sdb, ignoring the errors, but that didn't result in a usable half of a mirror either. Interestingly, these bad sectors hadn't made themselves known until this point. The RAID array had been otherwise healthy, and SMART hadn't uttered a peep. I think at this point, after spending about 3 hours fighting with it, I decided to move up a layer. I could successfully sync /dev/md0 (the root filesystem) and /dev/md1 (the swap device), and the only reason I could think of these bad sectors not giving me grief already, was because they were right at the end of the disk, and I hadn't quite used all the free physical extents in LVM yet. Lucky I hadn't decided to grow things any further. So I decided that rather than trying to convince this existing /dev/md2 to sync with the new /dev/sdb3, I'd just build a new degraded RAID-1 on /dev/sdb3 and move all the physical extents from the degraded RAID-1 on /dev/sda3 to the new degraded RAID-1 on /dev/sdb3. I tried the logical volume with /tmp on it first, since it was fairly sacrificial. It worked fine. So I tried the largest logical volume, /srv, which was 30Gb. That went fine as well, so I moved the rest. All of them went without complaint. So then I removed the now unused /dev/md2 from the volume group, and finally managed to boot with just the new disk 1. Then I was able to put in the new disk 0, and just do a standard RAID-1 rebuild onto it. Hooray for LVM saving the day yet again. And this is where I should have stopped. At this point it was about 3pm. I'd been aiming to leave by 4pm to pick up a rental car that I had to get by 5pm. I'd wanted to also upgrade to Etch while I was physically in front of the box, in case there were any nasty kernel/udev issues that made the box unbootable. The catch was, I needed to grow /var, which was quite full. The intention had been that once I'd got the two new disks in and all synced up, I'd create a fourth mirrored partition, and add this to LVM as a new physical volume, then I could just keep merrily growing things. It felt a little bit dirty having to make a new partition to get at the extra space, and I'd reading about mdadm's "grow" mode, whilst I was sitting around waiting for things to sync. So I thought to myself, I'd just delete the third partition, and recreate it using the remaining additional cylinders I now had at my disposal, then use this grow feature of mdadm to tell Linux the array was now bigger, then resize the physical volume, and go from there. This is where things went a bit pear-shaped. I deleted and recreated (larger) the third partition on each disk, and rebooted. I was then messing around with mdadm trying to update /etc/mdadm/mdadm.conf, since the UUID of /dev/md2 had changed, when I discovered that /dev/md2 had completely disappeared. It seems that recreating that partition made Linux completely fail to see /dev/sda3 and /dev/sdb3 as RAID-1 members, and so because of the fairly transparent nature of Linux's RAID-1, LVM had seen the underlying physical volumes, and just decided to run with /dev/sda3 as the physical volume, and everything had continued to work. I just had no redundancy any more. So it was about 3:20pm at this stage, and I was starting to panic, thinking I'd never get this sorted out before I had to leave, and I'd end up returning to the US with a non-redundant mess on my hands. Then I remembered that I had the backups of the partition tables of the old disks, from before I removed them. So I thought I'd go back to that state, and just have the additional cylinders unallocated. I hadn't yet attempted to grow the RAID-1 (partly because it had vanished), so in my haste, I didn't see this being a problem. The problem was, I'd booted into multi-user mode, and things had merrily been operating on just half the mirror. I restored the partitions, and rebooted, and lo and behold, /dev/md2 magically reappeared, already 100% consistent according to /proc/mdstat. I was somewhat surprised by that, but I booted back into multi-user mode again, only to discover that the /var filesystem was panicking and remounting itself into read-only mode. This was pretty weird, given that during boot, it was considered clean by e2fsck. So I was really getting bothered by this stage, so I rebooted into emergency mode (single-user mode in Debian starts way too much), and manually started up LVM, and forcibly fscked everything. There was a bit of filesystem damage to /var, /home and /srv. Thinking I'd now fixed the problem, I rebooted into multi-user mode again, but again /var, which was considered clean during the fscking/mounting phase of booting, panicked shortly after mounting, and remounted itself read-only. So I rebooted again, broke the mirror, and forcibly fscked the filesystems again (only operating on what was on /dev/sda3. Then I rebooted without incident into multi-user mode and resynced the mirror. At this point I ran away. So I'm really annoyed with myself. This should have been a blog post about how crap Linux's software RAID is at error recovery, and how LVM saved the day yet again, but it was overshadowed by me overreaching in my haste, and managing to cause silent data corruption. The morals of this story: I think that is all. I'm just so pissed off that what could have been a perfect (enough) upgrade with zero data-loss, even considering I had a bum disk, wasn't.

Phil Hands: Virgin Media (ex. NTL) support Linux just fine, despite their claims not to.

Today I helped a friend sign up with Virgin Media's broadband service. He was having trouble, as their front-page at http://activation.virginmedia.com/ does a brain-damaged test, and tells you that they only support Windows and MacOS X and that if you have anything else you'll need to phone them at a pound a minute so you can talk to some idiot until you lose the will to live (or some such). After a brief rummage, I noticed that if instead of doing what they tell you to do, you go to https://autoreg.autoregister.net/ you get prompted for whether you're signing up for Broadband or Dialup, and on clicking Broadband, you're lead through the sign-up procedure with no complaints about compatibility, and within 5 minutes he was online with his GNU/Linux (Ubuntu) system and Firefox browser. This makes me wonder why Virgin Media bother putting extra effort into turning users away when in fact the sign-up procedure would work perfectly if they hadn't bothered with the stupid browser test.

Ian Wienand: Playing with the x86 PMU

A discussion about commas lead to an excuse to have a play with the IA-32 processor performance managment unit (PMU). To start, take two versions of a program to count the number of commas in a text file — one in C and one in Python. The C one runs faster on the input data set of ~60MiB of random data, but why? The CPU performance monitors give are the key to getting some idea of where the programs spend their time. I like to use perfmon2 because it's what I know, but Oprofile can do it too. All the available events for IA-32 are described in the manual; I currently of no better way of finding out about them than just reading it. On Itanium I reccommend Caliper which, for the most common situations, does most of the work for you and presents it in a nice report. Intel's Vtune also does a similar thing. The first thing to investigate is if the CPU is getting enough instructions to keep busy. The IFU_MEM_STALL metric is a good place to start as it is triggered when the instruction fetch pipeline is stalled, presumably waiting on either the ITLB or the trace buffer (Icache). 
$ pfmon -e CPU_CLK_UNHALTED,IFU_MEM_STALL ./comma < ./randomcommas
340559375 CPU_CLK_UNHALTED
   192115 IFU_MEM_STALL
$ pfmon -e CPU_CLK_UNHALTED,IFU_MEM_STALL python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
1287100
4571257047 CPU_CLK_UNHALTED
 71981750 IFU_MEM_STALL
That works out to 0.05% of total cycles for the C version and 1.5% for the Python version, neither of which sets off immediate warning bells. If it did, we could start drilling down to things like the L2_IFETCH and ITLB_MISS events, or the BR_* branch events to try and see why the CPU is having to wait to get its next instruction. Next it is useful to find the CPI (cycles per instruction). This is calculated by the ratio of retired instructions against the number of CPU cycles; since a superscalar machine can issue more than one instruction per cycle this should ideally be much greater than 1 (for example, an Itanium can execute up to 6 instructions each cycle). 
$ pfmon -e INST_RETIRED,CPU_CLK_UNHALTED ./comma < ./randomcommas
542953593 INST_RETIRED
340612036 CPU_CLK_UNHALTED
$ pfmon -e INST_RETIRED,CPU_CLK_UNHALTED python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
1194455205 INST_RETIRED
4569931735 CPU_CLK_UNHALTED
This works out at a CPI of 1.59 for the C version and 0.26 for the Python version. The Python version is clearly spending a lot of time waiting, because it isn't even able to issue one instruction every cycle. At this point it seems the CPU has enough instructions to do, but it is sitting around waiting to get through those instructions. This suggests the waiting is related to getting data from the cache. The load and store requests from the L2 cache are accounted to the L2_LD and L2_ST events respectively. These have the ability to mask out cache lines in different states of the MESI protocol, but for this we don't care so just ask pfmon to show us everything. 
$ pfmon -e L2_LD:M:E:S:I,L2_ST:M:E:S:I ./comma < randomcommas 
102505 L2_LD:M:E:S:I
   167 L2_ST:M:E:S:I
$ pfmon -e L2_LD:M:E:S:I,L2_ST:M:E:S:I python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
3278774 L2_LD:M:E:S:I
  10457 L2_ST:M:E:S:I
This shows us that the Python version does quite a few more stores than the C counterpart. Considering this program should simply be reading the input stream and counting the number of commas, we do not expect much store traffic at all. This suggests the Python version is doing some extra copying, for whatever reason (maybe some Python expert can pinpoint it?). We can drill down a bit more into the memory related latencies. The DCU_LINES_IN event gives the total number of lines allocated in the cache. Another event, DCU_MISS_OUTSTANDING, gives a weighted measure of the cycles spent waiting for a cache line to be brought in. Each cycle spent waiting is weighted by the number of outstanding cache misses (I think the Pentium M I'm using can have up to 4 cache miss requests outstanding at once) and has some caveats, but can be considered a rough estimate of the time spent waiting for a cache line to be brought in. Therefore divding DCU_MISS_OUTSTANDING by DCU_LINES_IN gives us an approximate metric of how long a cache miss takes. 
$ pfmon -e DCU_MISS_OUTSTANDING,DCU_LINES_IN ./comma < randomcommas 
769736 DCU_MISS_OUTSTANDING
102387 DCU_LINES_IN
$ pfmon -e DCU_MISS_OUTSTANDING,DCU_LINES_IN python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
99810150 DCU_MISS_OUTSTANDING
 4240179 DCU_LINES_IN
So that works out to 7.5 cycles for the C version and 23 cycles for the Python version. This seems to strongly suggest that it is memory traffic that is weighing the Python version down. That is only the initial phase; the results give a high level idea of why one program is running slower than the other. The initial analysis phase generally consists of taking the ratios of certain events to try and get some idea in your head of what the program is doing. Then comes the really hard work; drilling down to figure out how to fix it! Some useful references:

20 December 2007

Anthony Towns: Risky advancements

The latest warning from Dresden Codak’s Aaron Diaz:
Do we really want to live in a society populated by geriatric 27-year-olds? In living so long and spending so much time thinking’ do we not also run the risk of becoming a cold, passionless race incapable of experiencing our two emotions (fear and not fear)?
Also interesting, is a talk by Vernor Vinge from back in February to the Long Now institute titled “What if the Singularity Does Not Happen?”, to which slides are available along with an audio recording.

MJ Ray: Current Issues / Severn Barrage Opposition Gathers

The Wildfowl and Wetlands Trust is hosting a meeting about wildlife concerns over the Severn barrage tonight at their place further up the river.  I've heard these concerns mentioned vaguely whenever the dam is discussed, but I don't really know what they are - does anyone?

The press release is a bit short on details and a quick look around the websites of the invited organisations didn't show anything obvious, except for Severn Barrage is not the answer, News: Tidal energy projects must not override existing environmental protection and Engineering our future nature: the dilemma of the Severn barrage on the WWF Cymru site.  Does that cover all the concerns, or are there more?

I'm still sort of surprised that the nation is seriously considering trying to put a barrier against the second-largest tidal flow in the world (Canada's Bay of Fundy seems to be the largest) and Britain's longest river, but I guess that's how our energy use is these days.

Next.

Previous.