Search Results: "ron"

17 October 2006

Russell Coker: planet debian, spam, and SE Linux

In regard to my post yesterday about Planet Debian I received the following response:
James Purser said I'm betting that your feed is an atom feed. We had the same problem on PLOA with Jeff and Pias feeds when they switched to atom. Planet needs to be upgraded.
Well I am using an atom feed, so this probably explains it. Sorry for the inconvenience to the Planet Debian readers, I guess that things will stay the way they are until it is upgraded.

Also when viewing my blog entry in Planet Debian I realised that much of a spam message had got pasted in to the URL field for the Planet Debian link. Oh the irony that I only found this embarassing error because of a bug in the Planet software.

This brings me to another issue, Security Enhanced X. With SE-X (before you ask, I didn't invent the acronym) you can use SE Linux to control communication between windows on an X desktop. With a modification to the clipboard manager (klipper in the case of KDE) every piece of data that's copied from an application will have a security context assigned to it and this context will be checked against the context of an application that is to be the target of a paste operation. Klipper will also have to support relabeling clipboard data. Therefore if I want to cut text from my email client (Kmail) and paste it into Firefox then I would have to relabel it with the appropriate MCS categories. This would permit me to paste text from an email into a web form with a few extra mouse clicks, but would prevent me from accidentally pasting the wrong text. Keeping in mind the fact that there are many more embarassing things that could be accidentally pasted into a blog entry than the contents of a spam this doesn't seem overly difficult.

PS Before anyone jumps to conclusions. When I receive GPG encrypted email or other material that should be kept confidential I try and avoid cutting it, and if I have to do so I clear the clipboard buffer afterwards. Keeping spam a secret is not really a priority to me so I didn't take adequate precautions in this case.

Russell Coker: more on clean energy

One new technology for saving fuel in cars is the 6 stroke engine. This is an engine that has two power strokes for every intake of fuel. The first power stroke is from the fuel burning, the second is from water being injected into the cylinder and boiling rapidly using steam for power. A significant amount of the weight of a car or truck engine is the cooling system. With water being injected into the cylinder after every burn the engine will require no other cooling, this can mean a weight reduction of up to 500KG for a truck engine! It is claimed that this technique can "improve a typical engine s fuel consumption by 40 percent". Note that this technique could be combined with the technology in a Prius for even greater efficiency.

Simon Richter wrote an interesting response to my blog about clean energy. I'm guessing that the part of Germany he's from is similar in some ways to Amsterdam (where I used to live).

In Amsterdam tiny cars are used a lot more than they have been in most places. There is the Smart Car but there is also a tiny car that can only seat one person (I'll update this entry with a link if someone provides me one). The tiny car appears to weigh ~200Kg, has a maximum speed of 30Km/h, and is narrow enough to fit in bike lanes. Such cars will take little petrol and can be used for shopping. The short-distance car idea is being used by many people already in northern Europe.

Simon suggests having a standard baggage container that fits in all cars. I don't think that would work as there are many different design factors (parking space, cargo capacity, and aerodynamics) which force designers to choose different shapes.

I think that a better idea would be a standard baggage trailer that could be towed by any small car. It should not be difficult to design a trailer that can be safely towed at 30KM/h behind a tiny car. The supermarkets could rent such trailers to shoppers for a nominal fee. Then most shopping trips could use the cargo capacity of the tiny car, but when buying supplies for a party you could rent a trailer from the supermarket.

Another option is having shops deliver goods to you. I have observed an increasing number of people doing this at my local supermarket. Of course you would still need to take ice-cream home yourself and maybe milk and meat too.

Electric scooters are also a good option for travel. Unfortunately in Australia there are few good options for securing them at the moment. As an electric scooter is light enough to be carried it needs to be chained to something secure. In the Netherlands this wouldn't be a problem as the bicycle infrastructure includes plenty of bike racks to which you can chain your bike, scooter, etc. Of course the Netherlands is entirely flat so there's no need for an electric scooter. Melbourne has a lot more hills and most people aren't fit enough to ride a bike so scooters are needed to replace cars.

Regarding depersonalised cars. That would require a significant social change as currently cars are extremely personalised. One thing that I had been thinking about is the idea of sharing cars with neighbors. For example if you have an apartment building and there are a few people you trust then you could share a tiny car for going shopping. Sharing a car used for driving to work or for entertainment would not work well as the car would spend most of it's time in use (or at least parked somewhere away from home). Sharing a car that's used for small journeys would be much easier as such a vehicle would spend most of it's time at home.

In Australia most families have two cars. One is used a lot (spends maybe 70 hours a week away from home) and the other is used much less (maybe 10 hours a week). Instead of owning two cars it would be possible for families to own one car and share another.

Regarding the Pebble Bed Reactor, could the people who advocate it please read the Wikipedia article. The limiting factor is not thermal expansion (solids do not expand nearly enough) but the Doppler effect (fast neutrons are not as effective at triggering fission). But in spite of that issue, let's not consider an untested new reactor design to be the savior of nuclear energy. I think that most people who read my blog have a science or engineering background and know from experience that new technologies often don't work too well in the first version. When a new CPU has a bug it's usually not a big deal. When a new OS or application has many bugs it's often expected (expecially when the OS or application comes from a monopolist). But if a new design for a nuclear reactor turns out to have a bug then it will be a more serious issue.

Russell Coker: what's a good blog server for serious blogging?

I'm getting sick of blogger. The main thing is that I'm simply not a user. Taking what someone else gives me and just putting up with any failings doesn't suit me at all. I can deal with bugs in things I control (such as Linux distributions) because I can fix the bugs I consider important at any time.

So now I'm looking for a serious blogging program. Wordpress was strongly recommended to me after my previous post on the topic of blogs, but that was in regard to a simple blog program for Intranet use. I am now after a blog program that is designed for Internet use, it must have good security, support multiple users (some of my friends will probably want to use my blog server machine), and not be overly difficult to customise (I am resigned to the fact that I will have to learn another programming language - probably re-learning PHP or Java as that is where web programming is at nowadays).

One thing that I want to do is to have the main web page that displays all recent posts display each post in a frame with a separate Adsense section. The topics of my posts vary a lot so I want to have adverts that match.

Another feature I want is to have multiple RSS feeds with different settings. One use for this is to have tags for each post to specify which channel(s) the post will end up on, another is for Adsense for feeds functionality which I want on for some feeds but off for others. I also want to generate multiple feeds for different syndication services. Ideally a syndication service such as Planet Debian or Planet Linux Australia would use a unique feed for sucking it's own data and also have a unique feed address advertised on it's site for the users (if this isn't supported or desired at the syndication level then I can do tricks in the web server to serve different content for different IP addresses). That way I can track use by the different services, work around bugs in syndication services that matter to me, and change settings for post summaries, etc to suit the syndication service.

In terms of HTML editing I only need the most basic functionality. I would be entirely happy to write blog entries in raw HTML, my friends would probably desire line breaks to be converted to paragraph or break tags and basic linking functionality, but they could probably deal with entering bold and italic tags themselves (the few of my friends who couldn't manage this would probably only want to write plain-text in paragraphs).

I also want to run my own syndication software. I guess I have to consider blog server and syndication server at the same time as there may be some dependencies (EG having them both written in the same language might be handy - I don't want to re-learn BOTH Java and PHP). The syndication software would ideally automatically collect the feeds from other syndication services that I specify (although I'm sure I could write a simple Perl script to scrape them from the Planet web sites). Then I want to provide an RSS feed of that content for anyone who wants it.

Please let me know via email or comments if you have any suggestions about which software to use.

Russell Coker: blogging software

Previously I asked for advice about running an Intranet blog, and running an Internet blog with hosting for friends.

In response to the question about running a small Intranet blog the recommendations were strongly for Wordpress, with a mention of Ikiwiki as well. One of the features that I consider desirable is for software to be reasonably popular which means that support is often easier to obtain. So Wordpress is my main candidate at this time for Intranet use. I'll install Wordpress and probably won't try anything else unless Wordpress fails in some significant way (which seems unlikely).

In response to my question about a blog server for serious blogging again Wordpress was well recommended. There is also a version of Wordpress in beta called Wordpress MU that supports blog server operations such as wordpress.com. Although I didn't mention it before I have had some ideas of starting my own server along such lines so Wordpress again does well.

Over the next few weeks I will start playing with Wordpress and Wordpress MU. If things go well I'll move my blog away from blogspot and to a domain I own in the near future.

Russell Coker: IT companies and toxic waste

Greenpeace has an interesting article about how IT companies rank in toxic waste problems.

Dell rates quite well, I feel happier about my recent purchase of a large Dell TFT monitor now. HP does reasonably well, that's fortunate as the Green party in Victoria has recently purchased a HP server. But next time we discuss such things I will suggest that more consideration be given to Dell servers because of this issue.

Lenovo does really badly, I'm surprised because I would have expected IBM to do reasonably well and I didn't think that Lenovo would make significant changes. From now on I will refrain from purchasing Lenovo products. I will still purchase second-hand IBM products, but nothing under the Lenovo brand until they clean up their act.

Also it's worth noting that computers manufactured with toxic chemicals will outgas some of the chemicals into the local environment (IE your server room, bedroom, or wherever else you have computers). Avoiding the computers manufactured with toxic chemicals is not only good for the environment, but also good for your health!

Russell Coker: vision

At the start of the year I suffered a minor injury to my left eye. This brought my vision down to slightly below average for my age. A doctor and an ophthalmologist both thought that my vision was good enough at this level and that I don't need glasses. But the ability to use a small font means that I can have more text on screen and thus do my work more effectively. Since about March I have worn glasses all the time. Regardless of what the experts say I want to have good vision all the time and there is a significant difference between my vision while wearing glasses and that without.

It's often recommended that people get their eyes tested about every 2 years from when they are in their late 20's and also get a test done a few months after any eye injury. I strongly support such recommendations for anyone who works in the computer industry. Even if your vision is slightly less than it could be it will make a difference to your work. I have not found glasses to cause me any inconvenience, I'm sure that it would be bad for children to wear glasses but for adults the only potential issue is when playing sport (which is less demanding on your vision than reading an 8 point font on a typical monitor). I am constantly amazed by TV and radio adverts which say "do you wear glasses, if so then you must have considered LASER eye surgery". Glasses are fine (IMHO) and I will not consider having a LASER or other dangerous object pointed at my eyes unless I am in danger of death or blindness!

The testing apparatus that an optometrist uses is interesting to inspect if you like machines. It is a device with a huge selection of levers for inserting different lenses in front of your eyes until they discover the combination that gives you the best vision. If you get your eyes tested make sure you get the optometrist to explain everything, you can learn many interesting things.

Recently, I have had a further eye problem. Recently my central vision went blurry with a shimmering affect for about 40 minutes, but the peripheral vision was OK. An ophthalmologist diagnosed this as a migraine, apparently I got a very mild one with only a minor visual affect but no headache etc. One thing that is apparently possible when experiencing migraine related vision impairments is the ability to act on visual information without consciously knowing it (EG catch a ball without seeing it).

A migraine is a better diagnosis than I had feared. The movie Super Size Me (the topic of tomorrow's blog entry) contains an interview with a main who describes a diabetic vision problem that sounded similar to mine. Migraines don't cause permanent problems while diabetes can, and diabetes requires lots of unpleasant blood tests and injections.

Apparently migraines are often triggered by physical events or food items. If you suffer one then it's recommended that you write down as much information as possible about the previous 24 hours to help track down the cause. If you experience something similar to what I describe then do not take this blog entry as medical advice, instead rely on a doctor and whichever specialist they refer you to. But do write down everything that happened prior to the medical problem (keeping an accurate diary can't do any harm).

The Wikipedia entry on Migraines has a lot more interesting information, but don't take it as medical advice either. One particularly interesting thing in the Wikipedia article is the note that migraines may be caused by not having suitable glasses, or prevented by wearing special glasses.

Russell Coker: review of Australian car web sites

It seems that Toyota isn't alone in having non-functional web sites. In fact it's better than some, the basic information on the cars is available and it is possible to get contact information for car dealers, also they have a feed-back form on their web site (to which I submitted my previous blog post). Incidentally the Lexus site had much the same problem as the Toyota site (hardly surprising as Lexus is the luxury marque from Toyota). But I expect that if I phoned Lexus to ask about their vehicles I would get a better call-center experience which would make me less inclined to blog about them.

Daihatsu vehicles are sold by Toyota. Their web site doesn't use Flash, but it has so little content that it doesn't count.

I decided to quickly review the web sites of car manufacturers that sell in Australia for a fair comparison. I found three sites worse than Toyota, two sites that were equal (counting Lexus), and six that were better than it.

Holden has the worst site, they don't display any information if you don't have flash, they don't even display a phone number! I wonder how much Adobe pays web programmers to pull this sort of stunt. I can't imagine Holden management saying "if a customer comes to our web site and doesn't have Flash then don't display our phone number or any other contact information, they can use Flash or buy a Ford instead". Obviously some web monkey has run amok and done their own thing without following directions. Probably some people need to be sacked in the Holden web development group.

Volvo Cars has a very bad site. Most of the content is involved with Flash in some way and refuses to load. There is a mailto reference that is broken, and the overview page for the S60 seems to have a JavaScript loop (I aborted the load after it loaded 245 pictures and was still going). The Volvo page for their other business is quite functional although minimal.

Hyundai has a bad site. The front page works OK, but some of the sub-sites to display information on vehicles redirect to sites such as evolveddriving.com.au which are "optimised for 1024x768" and require Flash and Quicktime while others do strange things like changing the size of the browser window. Overall it's a very bad site, but at least I could find the contact details for my nearest dealer, and it has a feedback form.

Subaru has an OK site. The only thing I couldn't access without Flash is information on their AWD (All Wheel Drive) technology. Unfortunately they provide no email address and no form for sending feedback.

The main Ford web page claims that Flash is required, but their site just works without it. In a quick test I was unable to find any functionality on the Ford site that is missing because of not having Flash. Ford have a well designed site.

The Volkswagen site makes no mention of the fact that I don't use Flash, it does however have some strange unused spaces in the middle of the screen. I guess that it recognised that I don't have Flash and made a semi-successful attempt to work around it. I could get all information I wanted including dealer contact details.

The main Mazda web page displays a message about Flash not being installed and offers a link to a non-Flash version of the site. The Flash section is at the center and the buttons at the sides work if you don't have Flash. This seems to be a well implemented site.

Citroen has an OK site, no flash that I noticed (although there were large blank areas on the screen at times indicating that something was missing), the information was all available and browsing was reasonably easy. One thing that annoyed me was that there were movies available but only through some sort of JavaScript that tried to play them in my browser. I have never bothered setting up my web browsing machine for playing movies (among other things it has no speakers) so this is a problem for me.

Peugeot has a good site. No apparent flash and it's reasonably easy to use. It has more pictures than Kia but the JavaScript navigation stuff is fancy. One nice feature is a single page with pricing summaries for all models. If you have $X to spend on a Peugeot you will easily discover which ones you can afford.

Kia has the best site I saw! Not only is there no flash, but it's well designed, easy to navigate and it loads quite quickly. Please review the Kia site as an example of how to do it properly!

Let me know if I've missed any makes and I'll post an update.

Hanna Wallach: mirror neurons and autism

Awesome. Two articles on mirror neurons and autism: (Thanks to Steve for pointing out the Sci. Am. article.)

Axel Beckert: Next Shell Quoting Talks

There are a several events coming up where I plan to hold my Shell Quoting Talk: First, there will be the BarCamp Zurich on October, the 28th at ETH Z rich HG and then there will be the 8th Linuxday.at on November, the 18th at the HTL at Dornbirn (Vorarlberg, Austria) organised by the LUG Vorarlberg. It’s also possible that, in addition to the Shell Quoting talk, I’ll also give a talk for beginners about Commandline Helpers. (Probably all the talks will be held in German.) BarCamp Z rich I’m quite curious on both events, for very different reasons. On the one hand, a BarCamp is something completely new for me and it sounds like a very interesting mixture of a real life Wikipedia meeting and a flash mob to me. On the other hand, this year’s Linuxday.at will have several new facettes for me: First there were several changes in the organising team, so I wonder if and in that case how much this will change the face of the event. Then it’s the first Linxuday.at since I live in Zurich, which means it’s the first Linuxday without 1000km travelling during that weekend, so I also have some time to meet friends in the area in advance to or after the event. Yeah!

Benjamin Seidenberg: New Debian maintainer Benjamin Seidenberg

Dear Benjamin Seidenberg! Your account ‘benjamin’ has just been created in the central LDAP database of the Debian project. Please note that it needs a bit of time until this information is synced with all developer-accessible machines. You should be able to login or upload packages after about 30-60 minutes. The password for this account can be found appended to this message, encrypted with your GPG key. Email sent to benjamin@debian.org will be forwarded to astronut@dlgeek.net, to change this visit http://db.debian.org/forward.html.
This has definately made my day. I would like to thank my sponsor and advocate, Anibal Monsalve Salazar, the Debian Cyrus Team (especially Sven and Henrique) and especially my AM, Cl ment Stenac (Zorglub). I couldn’t have asked for a better AM; he answered every mail within a day as far as I remember, was extremely supportive and helpful. I remember when I sent him the last T&S response he appologized on IRC that he couldn’t write the recommendation until that evening, whereas some AMs have taken months to do it.
For those interested in the NM queue’s proccessing time, here is an approximate summery of my progress through NM:
Waiting for advocate: 2.5 weeks (This should have been shorter, I had my advocate lined up, but then there was a question about me signing the key of the guy who signed mine, but I hadn t checked his ID).
Waiting for AM: 6 months
Going through the checks: 2.5 months. (This would have been shorter had it not been right around the end of my senior year of high school, with the mad rush to cram everything in before exams)
Waiting for FD: 1.5 Months
Waiting for DAM: 1.5 Months
Waiting for account creation after DAM approval: 2 months.

16 October 2006

Simon Richter: The not-so-recent events in Debian

During the past years, I have usually kept outside of heated arguments, but since I slipped on debian-private today anyway, I might as well go all the way and write a proper opinion, disguised as an outside view to simulate objectivity. Intellectual Icebergs Episode 10 talks about "Scales of Superiority". Simplified, we have a set of fields that we consider important to us, and where we will go to great lengths (and be irrational about it) to achieve a state where we can feel superior to others. To a large extent, that is what drives us on. If you think ballroom dancing is fun, you pick it up as a hobby, but if you place your "scale of superiority" in this field, you will go beyond that, and train every day, until you finally win the competition. Obviously, this is highly irrational to someone who is just not that "into it", but as they are generally not offended by you secretly sneering at their inferior dancing skillz (because they could hardly care less), this is not a problem. Now, inside Debian, we also have our scales by which we measure our success. For some of us, it is suitability for a particular field or work, for others it is rating by others. Debian has been suitable for a lot of tasks (since it is the universal operating system, after all), and has been held in high esteem for quality, so both camps were happy. So, what has changed? I believe that the biggest change was the advent of Ubuntu, and specifically, the community's reaction to it. A lot of people we considered our peers jumped ship, claiming that Ubuntu was "better". If you only cared about whether Debian was suitable for your needs, then your reaction was probably "good riddance". If you cared about peer rating, you were in hell now, as someone else was getting all the attention (and, since that is what your brain cares about really, all the MOTAS). In short, Debian's status as the superior distribution, your "herd" as the supreme rulers of the land of stability, your entire self-worth as defined by your "scale of superiority" was challenged. This is the situation we're in. Half of us feel threatened by Ubuntu, half of us do not, and to each group the behaviour of the other is completely irrational. Most of the behaviour we have seen in the past years is a direct result from this. The discussion about Ubuntu "not giving back patches" in the proper way was seen as an attempt at building a competitive advantage over Debian. That Ubuntu ships non-free drivers for hardware that is unsupported otherwise is a direct threat, because it allows them superiority in the field of hardware support. Now, with "LTS", they have even challenged our status as the distribution with the longest (heh) security support, while Debian shows severe stamina problems. That my girlfriend runs Ubuntu on her laptop is a direct threat to my masculinity, because I am unable to "perform" a network install on her laptop via wireless LAN. "I am sorry, dear, I have no idea why this device is unsupported, I never had any problems with that before." So we are looking for ways to regain our confidence. Some of us seek superiority by shifting their scales, claiming that while we're not as sexy as Ubuntu, we at least do not tie our users to our desktop, where they have to do kinky stuff to get to a harddisk, with us it is straight fsck, mount, umount. Some of us are trying to recreate Ubuntu's success by copying their methods (mandatory team maintenance, paying core developers) in the hope of eventually surpassing them. I can see that Debian also shares the goal of being a strong "contender" on the desktop, because "universal" simply includes everything, so I can understand that some developers are feeling threatened here. But I think I should ask you to get a broader view here. We are not measured by the number of one-user-only desktop installs we have. While it gives some short term satisfaction, it is not the same thing as a long-lasting installation on an university cluster and watching the users grow from nano to vi users. Debian is behaving as if it had a massive midlife crisis. I think we should try to remain calm here. Because Debian with a wig will not look young, it will look ridiculous, and in the long run, Ubuntu will either grow up, try to become another Debian (which makes them redundant), or they will stay in their niche and score the desktop users, which need not bother us at all, because they are bad in the sack anyway. I'm now going to turn myself in to Dr. Freud, and you go and get laid.

James Morrison: Pour Some Sugar On Me!

I went out to see Def Leppard and Journey last night. It was certainly an all ages show and Def Leppard was better than Journey. I discovered today that Corb Lund was playing in the city last night. I thought they were tonight, this is the second time I've gotten the date of a concert wrong that I wanted to see :(

James Morrison: Climate crisis

So, about a week ago I said Harper's budget looked sane. I was wrong. The cons have managed to drop two really useful programs. One is the main Energuide program and the other is the low income energuide program. Now if anyone we don't need more evironmental programs that hit every level of society, then explain the weather.

So now with obvious things to do: Bike on bike to work day and don't stop biking to work after that day.

James Morrison: Awesome ride

Yesterday, I had an awesome ride to Santa Cruz, the clipless pedals were very nice to use. The ride was pretty simple, get to the top of Skyline (take redwood gulch) to down to big basin, get our water refilled, go to the top of empire grade (Jamison Creek, or some fireroad) and then down to Santa Cruz. On the way to Redwood gulch I we were stopped at a stoplight, then all of a sudden I leaned slightly towards the pedal I was still clipped into. So, yup, my first day with the clipless pedals I fell. During this fall, I managed to loosen the clips on my shoes, so I needed to stop shortly afterwards to adjust my shoes. The next adventure came when 3 of us decided to take the fireroad up to Empire Grade. The fireroad starts somewhere in HP's Little Big Basin area, and is a nice trail to hike. It was rather steep, it was sandy, it was rocky and everything in between, so we had to walk our bikes up this trail. I took my shoes off and simply wore my wool socks to hike up the trail. Coming down Empire Grade was a lot of fun, not too many cars, and one of those cars even decided that passing me was a bad idea and got out of my way.

To top off such a fun day, when I finally got back to my apt, I discovered that I had the wrong set of keys. I had my one set of keys that didn't have an apt key. Oops.

Marc 'Zugschlus' Haber: Me? A nice guy? Never!

Andreas, I am not a nice guy. I am only lazy. If the change to exim4 (it now displays a debconf note to everybody who tries do dpkg-reconfigure exim4, -base or a daemon package telling them to dpkg-reconfigure exim4-config) saves #debian from answering the question “how do I reconfigure exim4, dpkg-reconfigure exim4 does nothing!” twice a day, it is a good change. I basically agree with you that people who not read the minimum basics of documentation are a nuisance, but they’re unfortunately real. You need to hurl the docs into their faces. And even then they’re going to ignore them and google for answers instead. And on google, they’re going to find wrong or outdated docs.

Uwe Hermann: Project Censored: Top 25 stories ignored by media in past year

Ok, this list should be pretty old news by now, but — by definition — the content of it is not. Because... it's simply not in the news: The Top 25 stories ignored by media in past year (via Project Censored): Full list with more details here... (via Boing Boing)

15 October 2006

Florian Ragwitz: Random seeking on gzip streams

After asking the lazyweb on how to seek on gzip streams I got a very useful and comprehensive reply by Paul Sladen. As my previous searches on this topic didn't turn up to much useful things I'd like to publish Pauls reply (with his approval) here to help other people having the same problem as well:
Paul Sladen:
Seeking on compressed streams is a little like reading sectors from a harddisk; to read one byte requires reading a much large sectors. Two things are needed, the sector size and the sector length. On a hard-disk the sector length is nominally 512 with the mapping between an offset and sector start often as simple as: $offset >> 9. In compressed streams, the relationship between stream position and sector start is non-linear. Using a non-linear mapping is more involved as searching a lookup table is required. Mapping between an uncompressed position instead cannot be calculated and a 'zmap' table listing all sector start position and corresponding on-disk points needs to be created. The size of a sector must be determined aswell. Bzip2: It is fairly easy to seek on compressed bzip2 streams. In a Bzip2 stream, each block is totally separate and self contained---have a look a the 'bzip2recover' program. 'bzip2recover' scans through a bzip2 stream locating each block/'segment' (normally 900kB of input, so ~200kB of compressed output), then copying each segment to separate bzip2 file. Gzip: Seeking on gzip streams is somewhat more involved; Gzip uses a dictionary design where back-references are made to uncompressed data within the last 32kB ("the window"). The only safe place to start reading a 'sector' in a gzip stream is somewhere where that the dictionary size is zero. An empty dictionary occurs at the start of a stream, or where the stream has been 'reset'. Gzip resets: A gzip 'sector' (the length required to be read, to safely decode a byte) could be the full length of the file-size, or could occur more frequently. A gzip sector start occurs when the stream is reset---resetting the stream more frequently leads to more sector starts, but a reduction in compression thanks to less use of the sliding dictionary. Trade-off. The "gzip --rsyncable" is an example of stream resetting; in this case the stream is reset (and the dictionary closed) each time that the sum of the last 4096 octets of input data, modulo 65536(?) equals a magic number (zero normally). In exchange for the extra reset/start points, you get a size increase of 4-5%, but a better "hit rate" for random access. Gzip reset point lookup tables: More likely you'd want to reset the stream every eg. 4096kB of input, then save the corresponding position of the output stream for a give input position. This lookup table of start positions needs to be stored somewhere. 'squashfs' does in a separate file; 'zsync' in the zmap. (If you wanted to define a standard, these reset points could be stored in the 'extra' or 'comment' fields at the start of a gzip file; storing them with a suitable header would enable some degree of seeking for sufficiently intelligent readers---files could be post-processed to add this data). Storing a LUT takes space; adding extra reset points and storing the LUT data for these extra points takes more space---but I found storing the LUT as a set of double-deltas cut it down; could be gzipped aswell. Gzip partial resets: A gzip stream doesn't just contain 'reset' points, but also 'partial reset' points, a partial resets the huffman tables, but doesn't reset the dictionary. It's safe to stop reading at a partial reset (which effectively shortens the sector length). Whilst not normally possible, partial reset points /are/ able to be used as a starting point /if/ you can populate the previous uncompressed 32kB that forms the dictionary window from another source---'zsync' does this when reconstructing a file, however 'zsync' only does construction in a linear fashion, from the start of a file. Further Reading: Magic words to search for: zsync, squashfs, apt-sync, succinct Succinct is my unfinished project. Succinct involves many parts, one of which covers what you're doing. It might be good to break separate off seeking of compressed streams into a separate project---many small pieces make big problems easier! What I might do is store the LUT in the EXTRA field of the gzip header (limited to 64kB though). Though, within 64kB it would always be possible to construct a useful LUT; based on the final length of the input data, it would be possible to reduce the granularity of the LUT such that it only mentioned 50%, 30%... even 5% of reset points. The LUT itself could be gzipped for extra gain. Hope that's useful, -Paul PS. In all the cases (except one) that I mentioned gzip, I probably meant 'zlib' or 'deflate'! :)
In a later mail Paul also added the following:
Paul Sladen:
BTW, in my search around a bit later for something I turned up two more things that might be interesting: dictzip, examples/zran.c in the zlib source 'dictzip' is a similar idea to what I suggested at the end of the previous email. A lookup table is built and stored in the EXTRA section at the start of the gzip file and the utilities ('dictzcat' et al) then use this look-up table for random-access on the data. 'dictzip' are still slightly wide of the mark, forcing a flush/chunk every 58969 bytes. That number is chosen so that even in the worst case, the compressed version will never be more than 216 in length. However I think their math is slightly wrong as even in the worse case, gzip only generates 5 bytes per 32kB for an uncompresed store. -> ~65514 by my reckoning. Squashfs uncompressable chunks by setting the top bit of the file offset to indicate a true store. Their lookup table format appears to be flexible enough that I /might/ be able to create a table by scanning an existing stream rather than re-encoding. Depends about the 64kB limitation. Mmm.
Thanks Paul!

Erich Schubert: Journalisten...

Auf den ersten fl chtigen Blick k nnte die Front genauso gut die R ckseite sein. Denn anders als die meisten Notebooks besitzt es vorne wie hinten eine sehr steil und pl tzlich abfallende Kante. Dank den Statusleuchten und Anschl ssen gestaltet sich das Finden der Vorderseite gl cklicherweise denkbar einfach.
Hier gefunden. Wenn man nicht weiss, was man schreiben soll, sollte man es vielleicht einfach lassen...

Christian Perrier: Work continues

So, finally, myself deciding to focus on technical work last months turned out to be a good idea which seems to be shared by a majority of fellow Debian developers. Of course, about some points, "le mal est fait" and for many, we are just a community of kids who are disconnected of reality. I'm afraid that Debian reputation is currently seriously damaged in the overall free software community and this, not because we are wrong on issues we raise (in short, I support the IceWeasel fork), but most of the time because we are unable to discuss issues without being uneccessarily childish and provocant. So, now for the important news, D-I has entered the final stages of the preparation for RC1. String freeze in progress, which means a window for translators to reach the heaven of being 100% everywhere.

14 October 2006

Russell Coker: more about Fedora

In a comment on a previous blog entry I was described as an active Fedora advocate, I don't think that is an accurate description. I advocate it to appropriate people, which is mostly non-programmers - but as I mentioned that means a larger proportion of the population than to whom I can advocate Debian. It's not that I'm trying to advocate Fedora, just that it fills a need for many people. I believe that the term Fedora advocate means someone has an objective of increasing the use and to use Fedora, I don't have such an objective. I am a Linux advocate, a Free Software advocate, and sometimes a Unix advocate (Unix meaning the entire family of Unix-like operating systems). Merely promoting something does not make you an advocate for it. I don't think of myself as a Debian advocate at this time, but as I am a Debian developer this may change.

It seems that the people who run the Fedora Planet think that my blog has suitable Fedora content, it's been added to that planet. Also the Fedora Planet appears to be running an older version of the Planet software as it has the same problem with my blog that Debian Planet had before the upgrade.

Now on the issue of gratis vs libre: As I am not a Red Hat employee I can't maintain a kernel-xen-nopae package and give it the same status as the kernel-xen package. Even when I was a Red Hat employee I couldn't have done that - it would require some amount of management approval. I believe that this fundamentally makes Fedora less of a libre distribution. There is no room in Fedora for someone who is an upstream developer and who just wants to maintain their own package. There is Fedora-Extras, but that has a second-class status. Only Red Hat employees can maintain packages in Fedora Core. This makes Fedora fundamentally less libre than Debian. I am not trying to suggest that Red Hat change things in this regard, I believe that Fedora is meeting all it's goals and that making Fedora as libre as Debian is not possible given the goals of making a profit on selling support of RHEL.

Chris made a good point. I also believe that MP3 codecs should not be in Debian/main. But I believe that people making mistakes about some issues is not a factor in judging the entire project. I believe that Debian is more libre although some bad decisions were made - largely due to lack of overall management. Fedora has hierarchical management, so when the legal team declares that some software can not be distributed then it gets removed without debate. I guess I could propose a GR to exclude MP3 codecs from main.

Also it should be noted that RHEL Extras has some of this software that is not in Fedora (RealPlayer for example). The Red Hat legal advice was that MP3 codecs need a license, so they ship a licensed version in their commercial distribution. This is the right thing to do for their customers (it's handy to have and I'm sure that they get a good deal by paying license fees for all their customers) and removing such things from Fedora is the right way to offer a gratis product without unreasonable legal liability.

Naturally Fedora is much more libre than any secret-source OS. Every user has the option of downloading the Fedora source and recompiling it as they wish. I could compile Fedora with a Xen kernel that runs on my hardware and with SE Linux policy that is more restrictive than that which Fedora currently has. I could build custom Fedora install CDs to install things the way I want (which I considered doing when I worked for Red Hat). But the liberty to fork a project does not compare to the liberty to join it, and the liberty to create your own packages in extras does not compare to the liberty to add your own packages that do things differently to the default package.

There are of course positive and negative aspects to this. I started work on SE Linux in Debian in 2001. In 2003 I joined Red Hat to work on SE Linux, in Red Hat I was not the only person dedicated to SE Linux work and other people spent part of their time working on it. The SE Linux work in Red Hat soon eclipsed that of Debian because there was management support. There was no possibility for a package maintainer to refuse to fix a bug that affected SE Linux simply because they didn't care for it. The positive side of this is that the SE Linux work proceeded quickly and efficiently. The negative side of this is that things which don't have management support don't appear in Fedora Core. Exim is a fine MTA but is not in Fedora Core. Some people think that AppArmor is a better option than SE Linux, they are wrong - but in Debian any developer has the option to add AppArmor support and neither I nor any other DD can prevent them. The libre nature of Debian means that as long as basic technical criteria are met DDs can add any package that they wish to the distribution.

These issues however are all related to people who are actively involved in Free Software development. For a typical Free Software user it often doesn't make much difference, until of course your favourite program doesn't get management approval to appear in Fedora Core. But the counter argument is that the quality of some of the >10,000 packages in Debian is not so high. You can install a Fedora Core package and have a reasonable expectation about how well it works, but Debian packages are sometimes rather experimental.

I also don't believe that Debian is a very functional Democracy. Some of the problems of Direct Democracy are demonstrated in Debian. In many ways it is more anarchistic, anarchy gives you liberty for good and bad. Maybe we should consider a Representative Democracy model for Debian.

Next.

Previous.