Search Results: "ag"

22 August 2007

Christoph Berg: Disclaimer

Still my favourite l.d.o posting: Re: plain language disclaimer.

Russell Coker: LED Headlights in Audi Sports Car

It’s interesting to see that Audi is releasing a car with LEDs for all lights including the headlights. This is being promoted as an environmental benefit, however a quick google search revealed that my Volkswagen Passat apparently takes 55W headlights (giving a total of 110W of electricity used). Even allowing for some inefficiency in the alternator this would make a very small impact on the fuel use of a engine rated at 140KW. The Audi in question is the R8 (wikipedia link because the Audi web site is badly broken) and has a 300KW engine… A simple implementation of LED headlights will do some good for plug-in hybrid cars and all-electric vehicles where saving power is more important - when the technology filters down to cheaper vehicles. Also one possible use for the technology is to dim the headlights by turning off some of the LEDs in the bank (according to the LED Wikipedia page it is currently impossible to create a single LED that takes more than 1W of power, so a bank of LEDs would be used). Currently you have a choice of using “parking lights” or “head-lights” when driving, and when driving just before sun-set or at night in the city (where the street lights are bright) you need head-lights to allow other drivers to clearly see you but don’t need them as bright as they have to be when driving at night in the country. So a range of levels of luminosity could be effectively used in headlights to increase efficiency in some situations and increase light levels in others. According to the Luminous efficiency Wikipedia page current LEDs are up to three times as efficient as quartz halogen incandescent globes and future developments are likely to increase that to six times the efficiency. Combine that with more effective use of headlights to provide the light at the location and level that’s needed and the result could be using at little as 10% of the electricity for headlights on average! Another thing that I would like to see is the Adaptive Headlights feature of the better BMWs (which I referenced in a previous post about the BM 5 and 7 series) implemented in a cheaper and more reliable manner. The feature in question is that the headlights will turn when driving around a corner to show the road ahead instead of just shining off the edge of the corner. Implementing such a feature with incandescent lights is difficult because they have to be physically turned and moving parts tend to break (which increases maintenance costs and decreases the overall reliability of the vehicle). An obvious alternate design is to have a set of LEDs pointing in different directions and which LEDs get power would determine where the light goes (this would also react faster than physically moving a light). Once LED headlights become common the Adaptive Headlights feature could be implemented in the cheapest cars on the road with minimal extra cost - currently it’s a feature that would be expensive to implement and would increase the sale price of a small car and probably the service price too.

Norbert Tretkowski: Updated policy

Thanks to Alexander Wirt we now have an updated backports.org policy, please take a look at his mail to our mailinglist if you are responsible for one or more backports. The two most important changes are a new version number scheme and the now required inclusion of more changelog entries in the changes file.

Wouter Verhelst: countmail

Jordi blogs about mail (in reply to a number of other people), and mentions a rather interesting application.
wouter@country:~$ countmail 
ZERO!
ZERO MAIL MESSAGES!
HAHAHAHAHA!
wouter@country:~$ 
Unfortunately, all is not what it seems. Apparently countmail doesn't support Maildir.

Romain Francoise: emacs-snapshot 20070822-1

emacs-snapshot is now taken from the CVS trunk rather than from the Emacs 22 release branch. You may want to put your current version on hold, or use the emacs22 packages from Debian. However, if you feel adventurous and choose to upgrade, you will have:

Mike Hommey: Problems and expectations

What would you expect a software such as VMware ESX server, in its latest version, to do when it technically can’t do what you would like it to do ? Well, I, for one, would expect it, at least, to tell me… but it doesn’t. If you don’t have VT enabled on an Intel 64bits processor-based server, and want to run a 64bits OS in a VM configured to host a 64bits guest, it doesn’t tell you. All you have for your eyes to stare at is an error message from the guest OS saying that the processor doesn’t support 64bits instructions. You have to gather from this message that it only needs VT extensions to be enabled. Now, if you’re not very familiar with these technical details, what would be your first test on such a server ? I’d say, most probably, try to run the 64bits OS on the bare hardware… which would succeed, indeed, leaving the user in a big blur. Note that in the “processor” part of the configuration panel in the Virtual Infrastructure Client, while there is information about Hyperthreading being enabled or not, there is nothing about of VT.

Lucas Nussbaum: Opening mutt s HTML attachments with epiphany?

I just ran into a rather funny problem. I’d like to use epiphany to open text/html attachments or mail parts. Mutt uses mailcap, so I added an entry in ~/.mailcap:
text/html; epiphany-browser '%s'; description=HTML Text; nametemplate=%s.html This should work fine, but the problem is that mutt creates a temporary file (/tmp/mutt.html) and removes it when the command terminates, which is nearly immediately with epiphany. And epiphany has a really nice feature: it monitors local files for changes and reload them when they change. Which results in seeing the page briefly, then a File "/tmp/mutt.html" not found. error page. Anyone using mutt and epiphany together successfully?

Evan Prodromou: 4 Fructidor CCXV

It's been a while since I've posted here; I've had a busy two weeks, and posting on my blog hasn't been a priority. Something I should do more often. Of course, as time backs up it seems harder and harder to actually get around to posting. I wanted to bring the attention of my friends and colleagues, though, to a notice that Maj put on the Wikitravel shared logbook. As of 15 August 2007, Maj is no longer working on Wikitravel. In addition, my time has been curtailed quite a bit, and I'm concentrating only on community and content issues -- no more programming or sysadmin work. This is all coming about for several reasons. The main one is that Internet Brands, the company that now owns wikitravel.org, wanted to have their own staff do the programming and sysadmin work. The end of our service contract seemed like an ideal time to effect this change. On our side, it's been challenging for us to work with a large organization not specialized in Open Content and wiki sites. We've gotten along really well with the IB people, but we haven't always seen eye-to-eye on the best way to grow and develop the site. As the decision was made to remove us from our technical roles, Maj decided it was a good time to opt out entirely. I'm going to miss her tremendously. As most people reading this already know, we started the Wikitravel project together, and over the last 4 years it's been a constant important part of our family life. Maj's information-architecture skills and dedication to travel have been crucial to the development of the site. She's been a strategizer, a spokesperson, a content organizer and a content builder. Probably most important for the future of the project, she developed almost single-handedly the Wikitravel Extra personal experience component of the site. We've got a lot to keep us busy, though. We recently launched Keiki, a project to create an Open Content parenting guide. For myself, I've also started Vinismo, an Open Content wine guide (sensing a pattern here...?). And we'll be working with our partner Jani P. on Wikitravel Press, an independent project to publish printed travel guidebooks based on Wikitravel articles. Despite the shakeups, I anticipate that the future of Wikitravel will be good. We have an incredibly strong and smart community, dedicated to an important and noble purpose. I'll remain involved on a part-time basis. We've got a lot of traction in public opinion and great Web traffic. I don't think that IB is going to make any bad decisions about the site, but if they do there's a local and global community of people that take Wikitravel and Open Content travel guides extremely seriously. I think things are probably going to be OK. tags:

On the subject of wiki The South by Southwest 2008 Panel Picker app is up for public review right now. SXSW Interactive, in Austin, is one of the best tech conferences of the year. I've got a panel proposal up, on Wiki 2.0 -- I really want to talk about some of the interesting new wiki apps, new wikis, and new wiki business models coming up. There are a couple of other wiki-related panels proposed -- including one on commercial wikis, which I presented about last year (Talks/SXSW07). Folks who are interested in wikis, or in SXSW, should take some time to give their feedback on the proposed panels and improved the level of discussion for 2008. tags:

Jeff Bailey: Cincinnati, and a big hello to Planet Ubuntu!

Firstly, a big hello to Planet Ubuntu! As an Ubuntu community member and Ubuntu core-dev, I've decided to add my livejournal to there. Since I usually don't make any effort to sensor my thoughts on here, I chose not to add it while I was still with Canonical in order to make sure that there was no confusion as to the OfficialStatus(tm) of anything I write.

The last two days have taken us from Toronto through to Cincinnati. Andrew took us down to the Buffalo airport, we met Jason and Brian from Koolu on the way day and finally wound up down there at 22h30 despite having left NewMarket at 11am. When we stopped at Babur for lunch, we wound up seeing Nancy and Alexis walk past.

We stayed overnight at the Mum's Red Carpet Inn, just over the Pennsylvania border. It didn't have Red Carpet. It *did* have Orange Shag on the side of the bed, however. High class. The continental breakfast was Tang, Bananas, vegan doughnuts and twinkies. The vegan doughnuts were full of palm oil, I don't think my heart rate has recovered yet.

On the drive towards Kentucky, we got hungry (and tired of Subway) when we approached Columbus. We first tried to go to a place called "Leaf Restaurant", which it turns out can feed us exactly zero things. Down to the pre-made chef salad, and the spaghetti sauce having meat in it. We went a bit further, and decided to try the university district. We pulled over after seeing an Ethiopian place, and saw Ben Sostrom, whom we'd just met at Opus the week before! We spent the evening hanging out with him, michael dobson, amber, and the rest of their YA group. They meet on Tuesday nights a couple blocks from where we parked! It was nice to have a post-Opus day like that, but their stories of carjackings and cops randomly pulling people over are hopefully overstated by several orders of magnitude!

We continued the drive down to just outside of Cincinnati. We're meeting up with Stephanie Truax today for our tour of the Creationism Museum. After this, we'll turn west and try to find our way to California.

Russell Coker: SE Linux vs chroot

A question that is often asked is whether to use SE Linux or a chroot to restrict a program. In Unix chroot is a way of running a program with a restricted set of directories available (it used to be merely a sub-tree but with bind mounts it can be any arbitrary set of directory trees). A chroot can be implemented in a daemon (it can call the chroot(2) system call before it drops it’s privileges) or by a shell script (through the chroot(8) utility). The disadvantages of a chroot are that root can escape from it, a chroot process can see the existence of non-chroot processes (ps and similar programs work in the same way in all chroot environments), and inter-process communication is not prevented. One solution to this is to have an enhanced chroot environment (which typically requires a kernel patch) where the chrooted processes can not run ps without restriction and have other limits applied to what they are permitted to do (there are several kernel patches that implement such restrictions). In the early days of SE Linux development I implemented similar functionality in SE Linux policy (here is the paper I presented at Linux Kongress 2002). Configuring a chroot environment is inconvenient. If it is configured in the traditional manner (copying files to the chroot instead of bind mounting the directories) then old versions may exist in the chroot after new versions with security fixes have been installed in the main environment. SE Linux provides better security than a typical chroot environment by controlling all interaction between processes. It provides more flexibility than an enhanced chroot environment by being configured entirely by policy and not requiring a kernel recompile to change the way it works. I believe that the correct thing to do is to cease using chroot entirely and use SE Linux instead.

Adrian von Bidder: /etc/apt/preferences

I use apt's preferences to give decreasing priorities to our Debian releases/suites. On my server, sarge has 800, etch has 750, lenny has 700 and unstable has 600. There are addtional values for backports and some other repositories. Generally, the newer, the lower the priority. So I can mix and match between these repositories and have aptitude upgrade generally do the "right thing" (and aptitude's dependency resolution is great to solve even complex issues). This is, for me, one of Debian's strongest points. But sometimes, there are these strange effects... Like when I decided that I'd follow etch and etch-volatile for clamav (the server is still in progress from sarge to etch), but then the new available version in sarge-volatile is higher than the currently installed version, which came from etch-volatile. I'm not sure if a "correct" solution is possible or if I would even want one - after all, I did specify sarge on a higher priority than etch, and by using volatile I stepped basically a bit "outside". It's probably just to keep me on my toes... But still, the clamav people win the award for the most illustrating use of multiple repositories and apt priorities I've seen so far.
clamav:
  Installed: 0.91.1-1~volatile1
  Candidate: 0.91.2-0volatile1
  Version table:
     0.91.2-1 0
        600 http://sunsite.cnlab-switch.ch unstable/main Packages
     0.91.2-1~volatile1 0
        750 http://mirror.switch.ch etch/volatile/main Packages
     0.91.2-0volatile1 0
        800 http://mirror.switch.ch sarge/volatile/main Packages
     0.91.1-1 0
        700 http://sunsite.cnlab-switch.ch lenny/main Packages
 *** 0.91.1-1~volatile1 0
        100 /var/lib/dpkg/status
     0.91.1-1~bpo.2 0
        740 http://www.backports.org etch-backports/main Packages
     0.91.1-1~bpo.1 0
        200 http://www.backports.org sarge-backports/main Packages
     0.90.1-3.1lenny3 0
        700 http://security.debian.org lenny/updates/main Packages
     0.90.1-3etch4 0
        750 http://security.debian.org etch/updates/main Packages
     0.90.1-3etch2 0
        750 http://sunsite.cnlab-switch.ch etch/main Packages
     0.84-2.sarge.17 0
        800 http://security.debian.org sarge/updates/main Packages
     0.84-2.sarge.15 0
        800 http://sunsite.cnlab-switch.ch sarge/main Packages

Neil Williams: Moving away from gedit to geany

gedit can be a real pain sometimes - mainly through what it still lacks: code folding, bookmarks and the ability to run two instances of gedit side by side (each with their own tabs) without having to start the second in a different workspace and then move it. Of these, code folding and bookmarks and the biggest problems with how I wanted to use gedit - for Perl source code.
Anjuta is overkill for perl (and doesn't support perl particularly well) and bluefish perl syntax highlighting is plain broken - it gets the syntax wrong frequently (especially when using copy and paste and then requires you to edit the line at which the syntax failed before it will move on and fail a few lines later. Bah!
So I've turned to geany - never used it before but it's great for Perl. It identifies all the objects in a Perl script or module and allows me to click on any object to go to that part of the file. It has code folding (and it works). It has bookmarks (although it would be nice to be able to click on a button to go from one mark to the next). It handles "projects" quite well too. It also makes no complaints about running in a second instance. This is really helpful because I can have apt-cross in one instance of geany - with all the relevant files in tabs - and dpkg-cross in another, with its files.
There's a 'scribble' area in geany too which is neat - no need to have an "untitled document" tab that takes up more space on the tab bar than it deserves AND nags you when you eventually close gedit.
I'll still use gedit for some things but I guess it's about time that I moved away from a simple but powerful text editor and used a proper IDE that can cope with Perl. For me, that is geany.

Ian Wienand: Fun with <tt>-funroll-loops</tt>

I was playing with loop unrolling, and decided to investigate it a little. The basis is this very simple program which looks like it could do with some loop unrolling.
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#define SIZE (1500 * 1024 * 1024)
int fn(int *buf)  
        int i;
	for(i=0; i < (SIZE/sizeof(int)); i+=1)
		buf[i] = i;
	return 0;
 
int main(void)
 
        int *buf = malloc(SIZE);
        assert(buf);
	return fn(buf);
 
Firstly, lets have a look at what happens to the function when -funroll-loops is applied. Below is the de-compilation with some comments on what each line is doing.
Non-optimised
                           nop.m 0x0
r14=0                      mov r14=r0
save the "loop counter"    mov.i r2=ar.lc
                           nop.m 0x0
one less than iterations   movl r15=0x176fffff;;
                           nop.m 0x0
initalise loop counter     mov.i ar.lc=r15
                           nop.i 0x0;;
                           loop:
*buf = i, buf++            st4 [r32]=r14,4
i++                        adds r14=1,r14
dec lc, branch if > 0      br.cloop.sptk.few .loop
                           nop.m 0x0
restore loop counter       mov.i ar.lc=r2
Loop unrolled
                           nop.m 0x0
r26 = 0                    mov r26=r0
save lc                    mov.i r2=ar.lc
                           nop.m 0x0
iter / 32                  movl r14=0x2edffff;;
                           nop.m 0x0
setup loop counter         mov.i ar.lc=r14
                           nop.i 0x0
                           loop:
r3 = buf                   mov r3=r32
r8 = 1                     adds r8=1,r26
r25 = 3                    adds r25=3,r26
r23 = buf[3]               adds r23=12,r32
r24 = 4                    adds r24=4,r26
r22 = buf[4]               adds r22=16,r32;;
buf[0] = 0; r3=buf+1       st4 [r3]=r26,4
r21 = 5                    adds r21=5,r26
r19 = buf[5]               adds r19=20,r32
r20 = 6                    adds r20=6,r26
r18 = buf[6]               adds r18=24,r32
r16 = 7                    adds r16=7,r26;;
                           nop.m 0x0
buf[1] = 1, r3=buf+2       st4 [r3]=r8,4
r15 = buf[7]               adds r15=28,r32
r17 = 2                    adds r17=1,r8
r26 += 8                   adds r26=8,r26
buf += 8 (for next iter)   adds r32=32,r32;;
buf[2] = 2                 st4 [r3]=r17
buf[3] = 3                 st4 [r23]=r25
                           nop.i 0x0
buf[4] = 4                 st4 [r22]=r24
buf[5] = 5                 st4 [r19]=r21
                           nop.i 0x0
buf[6] = 6                 st4 [r18]=r20
buf[7] = 7                 st4 [r15]=r16
loop, dec branch           br.cloop.sptk.few .loop
                           nop.m 0x0
restore lc                 mov.i ar.lc=r2
                           br.ret.sptk.many b0;;
We can see that in the first case the loop counter is setup to run 393216000 times (e.g. 1500*1024*1024/sizeof(int)) and a very simple loop of storing the value and incrementing is undertaken. In the second version, the code becomes more complex. We can see the loop now executes 49152000 times (e.g. 1500*1024*1024/sizeof(int)/8) so we can infer that for each loop iteration, 8 values have been unrolled to be written into our buf array. Indeed, when pulled apart we can see the 8 stores (numbers just reflect the first time through). So, is it faster?
$ time ./loop
real    0m1.209s
user    0m0.388s
sys     0m0.820s
$ timme ./loop-unroll
real    0m1.056s
user    0m0.244s
sys     0m0.812s
Yes! Great. We can suggest a number of reasons why, but it would be interesting to get some hard statistics on why this is so. Thanks to the Itanium performance counters, we can. Firstly, lets start by seeing how cycles are being spent.
$ pfmon -e cpu_cycles,ia64_inst_retired_this,nops_retired,back_end_bubble_all ./loop
 547923609 CPU_CYCLES
1179995250 IA64_INST_RETIRED_THIS
    104431 NOPS_RETIRED
 154285534 BACK_END_BUBBLE_ALL
$ pfmon -e cpu_cycles,ia64_inst_retired_this,nops_retired,back_end_bubble_all ./loop-unroll
 311570590 CPU_CYCLES
1327451286 IA64_INST_RETIRED_THIS
 147560435 NOPS_RETIRED
  16088889 BACK_END_BUBBLE_ALL
Now, the Itanium can issue up to six instructions (IA64_INST_RETIRED_THIS) in two bundles of three instructions per cycle (CPU_CYCLES). However, you can't have dependencies between registers within a 3-instruction bundle (and branches and interruptions play havoc) so sometimes you need to pad with no-ops (this is why you need a good compiler). We can work out some figures from this, namely the useful instructions per cycle (e.g. instructions retired - nop instructions / cycles).
TestUseful instructions/cycle
loop1179995250 - 104431 / 547923609 = 2.15
unroll1327451286 - 147560435 / 311570590 = 3.78
This tells us that for each given cycle, the unrolled-loop version is doing more work (ideally we'd like that to be a the theoretical maximum of 6). So what is the unrolled version doing for all this time? Bubbles are where the CPU is waiting for something; a prime candidate is moving data from caches to registers. Bubbles make up (154285534/547823609) 28% of the cycles on the loop version, but only (16088889/311570590) 5% of the time for the unrolled version. This means the processor is spending a lot less time waiting for things to happen with the unrolled loop version. We can drill down further to see what bubbles are occurring.
$ pfmon -e be_l1d_fpu_bubble_all,be_exe_bubble_all,be_RSE_bubble_all,Back_end_bubble_fe ./loop
 14229581 BE_L1D_FPU_BUBBLE_ALL
    97669 BE_EXE_BUBBLE_ALL
     1260 BE_RSE_BUBBLE_ALL
131354866 BACK_END_BUBBLE_FE
$ pfmon -e be_l1d_fpu_bubble_all,be_exe_bubble_all,be_RSE_bubble_all,Back_end_bubble_fe ./loop-unroll
22676169 BE_L1D_FPU_BUBBLE_ALL
   83064 BE_EXE_BUBBLE_ALL
    1263 BE_RSE_BUBBLE_ALL
  718850 BACK_END_BUBBLE_FE
We are now breaking the stalls down to see where they are occuring. L1D_FPU_BUBBLES are essentially related to micro-architectural issues with the caches; things like the L1D cache being overwhelmend and interactions with the hardware page-table walker. BE_EXE bubbles are more simply described a waiting for data to come from caches (or even main memory) to registers. BE_RSE bubbles are related to the register stack engine, and "front end" bubbles (BACK_END_BUBBLE_FE) are where there are not enough instructions coming from the "front-end" for the "back-end" to issue out to functional units in the processor. In this case, the major cause of problems for the simple loop seems to be not being able to get enough instructions through to keep the processor busy. We can try to pin down what is happening in the front-end:
$ pfmon -e fe_bubble_imiss,fe_bubble_tlbmiss,fe_bubble_branch,fe_bubble_bubble ./loop
   27727 FE_BUBBLE_IMISS
    4910 FE_BUBBLE_TLBMISS
98094189 FE_BUBBLE_BRANCH
52615907 FE_BUBBLE_BUBBLE
$ pfmon -e fe_bubble_imiss,fe_bubble_tlbmiss,fe_bubble_branch,fe_bubble_bubble ./loop-unroll
780729 FE_BUBBLE_IMISS
  4858 FE_BUBBLE_TLBMISS
291620 FE_BUBBLE_BRANCH
111089 FE_BUBBLE_BUBBLE
What we can see here is that the increased code size of the unrolled version causes us a lot more i-cache misses, as expected. However, the problem for the unrolled version seems to be the time spent by the front-end dealing with the branch. This is not surprising since there is a rumor that there is a front-end buffer which fills up after 3 repeated cycles with branch instructions; apparently this is a problem addressed in the new Montecito chips (if anyone has one, I'll update this if you send me the numbers). Since we are doing 393216000 branches all within a single bundle its not surprising we've hit it! Just to confirm, we can break-down the BE_EXE bubbles. Remember, BE_EXE bubbles are caused by stalls in the execution phase primarily due to cache latencies or inter-register dependencies. Looking at our code, we don't re-use the same register over-and-over so we would not expect to spend a long time waiting for registers, and indeed this is what we see.
$ pfmon -e be_exe_bubble_grall,be_exe_bubble_grgr ./loop
79601 BE_EXE_BUBBLE_GRALL
  919 BE_EXE_BUBBLE_GRGR
$ pfmon -e be_exe_bubble_grall,be_exe_bubble_grgr ./loop-unroll
76627 BE_EXE_BUBBLE_GRALL
  918 BE_EXE_BUBBLE_GRGR
Therefore in this case, the speed increase isn't coming from better cache behaviour as we might have expected, but greater ability to keep the processor busy with instructions between branches. We can also examine the major events happening with the unrolled case. Here we see BE_L1D_FPU bubbles being the major overhead. On Itanium floating-point avoids the L1 cache (it is too big, and not re-used enough to make it useful) hence the concatenation of the event acronym. These bubbles can be broken down into the following events (measured in two runs, since you can only do 4 at a time).
$ pfmon -e be_l1d_fpu_bubble_l1d,be_l1d_fpu_bubble_l1d_dcurecir,\
  be_l1d_fpu_bubble_l1d_tlb,be_l1d_fpu_bubble_l1d_stbufrecir ./loop-unroll
 9623193 BE_L1D_FPU_BUBBLE_L1D_DCURECIR
     414 BE_L1D_FPU_BUBBLE_L1D_TLB
     282 BE_L1D_FPU_BUBBLE_L1D_STBUFRECIR
$ pfmon -e be_l1d_fpu_bubble_l1d_fullstbuf,\
    be_l1d_fpu_bubble_l1d_l2bpress,be_l1d_fpu_bubble_l1d_tlb ./loop-unroll
      23 BE_L1D_FPU_BUBBLE_L1D_FULLSTBUF
17220177 BE_L1D_FPU_BUBBLE_L1D_L2BPRESS
     425 BE_L1D_FPU_BUBBLE_L1D_TLB
These events expose micro-architectural details, the full extent of which is only gleaned by reading the processor manuals (a PhD in computer architecture probably helps too!). The store buffers are not an issue here -- the caches are more than able to keep up with our few writes. The TLB is also not involved; we have pretty good TLB coverage thanks to the virtual-linear page table and hardware page-table walker. The two clues are DCURECIR and L2BPRESS. Essentially DCURECIR happens when references have to "re-circulate"; this is a side-effect of the data not being available. The L2BPRESS figure suggests why this is so; this event happens when the L2 cache can not handle any more requests. The L2 cache keeps a queue of outstanding requests; when this queue is full (for example, full of long latency requests to get data from main memory) the L2 cache causes "back-pressure" which stops the L1 cache from issuing new requests. This suggests the program is chewing through a lot of data and not using the cache very effectively, which is exactly what we are doing. Pre-fetching hints can help (but as described in a previous entry on hacking Enigma may not always help), but in this case there probably isn't any spare bandwidth to pre-fetch in because essentially no processing is happening. What does all this mean? I don't know, but playing with performance monitors is fun!

Adam Rosi-Kessel: Crazy Amtrak Rule of the Day

My itinerant brother, presently in Tucson, Arizona is trying to visit my less-itinerant brother in Tahoe, California. (The latter is an editor and photographer for the Tahoe Daily Tribune, now available online in full print-like layout). Later this week, the itinerant brother flies from Tucson to Sacramento, the least expensive airport closest to Tahoe. Amtrak has a route from Sacramento to Tahoe (actually nearby Truckee), but it is actually a bus. Having made his plane reservations, my brother waited until today to order his Amtrak tickets online, only to discover the following: You can't take a bus without taking a train. Yes, it’s true, you can’t actually board that bus (the only way to get to Tahoe after my brother’s plane arrives), unless you’ve at least set foot on a train first. Some research suggests that California State Law prohibits Amtrak from offering bus-only services within the state, apparently so it won’t compete with bus companies. I did figure out what I hope is a clever workaround: the next closest train stop on one of the lines to Sacramento is Davis. Amtrak allows you to buy a ticket from Davis to Truckee via Sacramento for basically the same price as the bus from Truckee to Sacramento. Hopefully no one will check to make sure he actually traveled from Davis to Sacramento before getting on the bus to Truckee. There must be some profound lesson here. I would love to have been a fly on the wall during the floor debate in the state legislature when this law was enacted.

Neil Williams: dpkg-cross 1.99+2.0.0pre1 in NEW

(I had to omit some of the smaller changes - it should all be in the ChangeLog but ask if you spot something major that I haven't mentioned here.)

IMPORTANT! dpkg-cross_1.99+2.0.0pre1 will BREAK things!
(hence it is going into experimental)

  1. BREAKAGES!
    The changes in dpkg-cross are so extensive that apt-cross and emdebian-tools are broken. Current SVN for emdebian-tools builds 0.3.9 which works with pre1. There is a branch of apt-cross (apt-nc-cross) that builds 0.2.9 which works with dpkg-cross pre1. HOWEVER apt-cross 0.2.9 has VERY limited functionality (even --get is incomplete) so dpkg-cross pre1 should only be installed in a chroot or on a system that does not need apt-cross or emdebian-tools. There's little point adding a NEWS.Debian comment on this because it would be too late but Conflicts: are in place:
    Conflicts: apt-cross ( 0.2.9), emdebian-tools ( 0.3.9)
    (The watchful will note that the previous Conflict on pkg-config has been dropped because the conflicting version is no longer in stable or oldstable.)
    The dpkg-cross pre-releases will all go into experimental and no upload will be made to unstable until apt-cross 0.2.9 is available as a package in Emdebian and is also ready to go into Debian as v0.3.0. emdebian-tools needs less changes, some of which can be done AFTER dpkg-cross 2.0.0 reaches unstable.
    I anticipate at least three pre-releases, maybe more.

  2. REMOVED PERL FILE:
    /usr/share/perl5/dpkg-cross.pl
    Replaced by:
    use Debian::DpkgCross;
    There are some functions not universally exported but which can be brought in manually:
    gccross : use Debian::DpkgCross qw(simplify_path);
    dpkg-shlibdeps : use Debian::DpkgCross qw(dpkgcross_application);
    dpkg-buildpackage : use Debian::DpkgCross qw(setup_cross_env);
    The universally exported functions are:
    read_config setup get_architecture create_tmpdir convert_path get_keepdeps get_removedeps get_config get_version detect_arch convert_ld_library_path get_tool rewrite_pkg_name dump_debug_data check_arch

  3. Perl changes:
    use strict;
    use warnings;

    All scripts and the new module are now fully strict and -w compliant. Some perl modules have also been added to the dependency list.

  4. MAKEFLAGS are no longer exported!
    This is a big change and could affect a lot of packages but the honest
    reason is this:
    We should not be exporting MAKEFLAGS ourselves. (in the context of debian-embedded) Packages that were buggy with MAKEFILES are now non-buggy and packages that relied on MAKEFLAGS are now buggy.
    CDBS packages have a single line fix in debian/rules:
    DEB_CONFIGURE_SCRIPT_ENV =
    I'll be filing a bug in due course.
    This allows packages like libx11-6 to build internal tools using CC_FOR_BUILD (set to gcc) and run them to make certain generated source files during the cross-build. I expect it to also work for fontconfig. This method is far preferable to either creating patches for the generated content or trying to replicate the generation process in an interpreted language.

  5. New function:
    &read_config();
    &setup();


    $config = &get_config;
    get_config is new because the configuration is now an internal variable within the module and is not in scope for the scripts. It is a hash table:
    $config 'crossbase' = $crossbase;
    $config 'crossprefix' = $crossprefix;
    $config 'crossdir' = $crossdir;
    $config 'crossbin' = $crossbin;
    $config 'crosslib' = $crosslib;
    $config 'crosslib64' = $crosslib64;
    $config 'crossinc' = $crossinc;
    $config 'crossroot' = $crossroot;

    The values from the two config files:
    $conffile = "/etc/dpkg-cross/cross-compile";
    my $home = File::HomeDir->my_home;
    $private_conffile = "$ home /.dpkg-cross/cross-compile";

    are read in using Config::Auto which removes all the syntax problems of the previous bespoke parser. Comments are ignored as before.
    Everything you specify in either of the config files is now available within the module but only "recognised" options are exported. Config file values are merged - data from /etc/ is read first and data from /home/ is overlaid so that any data not in /home/ comes from /etc/ and any data from /etc/ is overwritten if it also exists in /home/
    @removedeps and @keepdeps need to be sorted out still.

  6. REMOVED CONFFILES:
    /etc/dpkg-cross/cross-config.common obsolete
    /etc/dpkg-cross/cross-config.gnu obsolete
    /etc/dpkg-cross/cross-config.linux obsolete
    /etc/dpkg-cross/cross-config.w32 obsolete
    /etc/dpkg-cross/crosstools obsolete

  7. TESTING
    Ah, yes, testing. So far, only two packages have been tested. I plan to test more tomorrow. Please check for regressions and other brokenness and report via the BTS. (I may forget if it's only done via the debian-embedded list.) Please tag all pre1 issues 'experimental'. Ta.

  8. Speaking of bugs - 6 closed so far, a few more possible. I would like
    to close all but one of the current dpkg-cross bugs. (I'm not sure how
    to fix #285569 )

  9. Debian::DpkgCross.3 manpage - new but horribly incomplete.

  10. %pkgvars - currently disabled. I will merge scope: into the existing %config which then leaves no place for %pkgvars or crosstools but this is pending for pre2.

  11. debug output.
    You now get two debug files in /tmp/:
    gccross2.log : Details the environment, %config and check process of the last call (the last failed call if the build breaks).
    dpkg-cross2.log : Actually misnamed, this is a dpkg-buildpackage log file that contains %config, PATH changes, $ENV changes and the contents of the gccross temporary directory for the last build, including variables set and modified via the module. I'll change the name in pre2.
    The debug output is meant for debian-embedded, not other users. To this end, I'm experimenting with debian/buscript which could go into /usr/share/bugs/ to prompt users reporting bugs for the content of these logs and other configuration details, exactly as with apt. If this works, I'll ditch the idea of turning --debug off and leave it enabled for all builds.
    debian/bugscript is not enabled in pre1. There again, because the module is a new package, pre1 will spend some time in the NEW queue. All changes for pre1 are now in CVS so it should be easy for everyone to build their own packages until pre1 clears NEW and enters experimental. Just be careful where you install dpkg-cross_1.99+2.0.0pre1_all.deb !!!
    (And remember: install libdebian-dpkgcross-perl_1.99+2.0.0pre1_all.deb)

21 August 2007

Russell Coker: Music Videos

I’ve been thinking about music videos recently while compiling a list of my favourite videos of all time. It seems that YouTube has changed things through the re-mixes of videos and the ability of anyone to publish for a mass-market (although without the possibility of directly making money from it). Also today all new PCs (and most PCs that are in use) are capable of being used for video editing and the compute power needed for 80’s and 90’s quality special effects is also commonly available (in most cases good art doesn’t need more technical quality than that). So anyone can produce videos (and a quick search of YouTube reveals that many people are producing videos for their favourite songs). I think that we need a music video for the Free Software Song. One possibility is to base it on the 1984 Apple advert (because it’s the free software community that is opposing Big Brother not Apple). I think it would be good to have multiple versions of the Free Software Song (with matching videos), there could be the version for young children, the Hip-Hop version, the Punk version, etc. Also I think that there is potential for the creation of other songs for the free software community. One possible way of doing this would be to have a contest for producing music and videos. Maybe a conference such as LCA or OLS could have the judging for such a contest. I would be prepared to donate some money towards the prize pool and I’m sure that other individuals and organisations would also be prepared to do so. If I get some positive feedback on this idea I’ll investigate how to run such a contest. Here are my favourite videos of the moment. Please let me know of any videos that you think I would like based on this list.
  • Smashing Pumpkins:
    Ava Adore, interesting way of changing scenes, and a very artistic and strange video (matches the song)
  • Duran Duran (who incidentally named their group after a character in Barbarella: Queen of the Galaxy - strangely the spelling is different though):
  • Too Much Information - they should re-do this and include a reference to the Internet in the lyrics. ;)
  • Wild Boys - Mad Max 3 as a film clip

  • UNKLE:
    • Eye for an Eye - strange and disturbing, as any serious art that is related to war must be
  • Rabbit in Your Headlights - surprising end, I wonder if anyone was injured trying to emulate this clip

  • Nine Inch Nails:
    Head Like a Hole, strange and a bit bizarre at times. Not the greatest of my favourite clips but the music makes up for it.
  • Queen:
    • I Want to Break Free, strangely amusing and very artistic


  • Chemical Brothers:
    • Let Forever Be - my favourite clip of all time. Fractally weird, you can watch it dozens of times and still be missing things.
  • Setting Sun - the world would be a better place if more cops could dance like that! Also is it just me or does the drummer guy look like a Narn from Babylon 5?
  • Out of Control - surprise ending. I would appreciate it if someone who knows the non-English language (probably Spanish) in the clip could point me to a translation.
  • Star Guitar - a real work of art but no plot and I didn’t enjoy the music, I recommend watching it once
  • The Golden Path - I used to wonder whether office work was really so grim in the 60s and 70s, but then I worked for a financial company recently…

  • Fat Boy Slim:
    Praise You - why can’t reality TV be this good?
  • Falco:
    Rock Me Amadeus - let’s represent two totally diffent cultures (bikers and Austraian high society) in a film clip, silly but amusing
  • Madonna:
    Like A Prayer - I wonder how many racist organizations banned that
  • A-Ha:
    Take On Me - mixing multiple art forms (in this case film and animation) can work really well. Beat Kill Bill to the idea by a couple of decades.
  • Robert Palmer:
    Simply Irresistable - pity that they didn’t hire more women who can dance or at least put the dancers in front of the models. It’s interesting to note that one of the models appears to be actually playing a guitar.
  • Garbage:
  • Michael Jackson:
    Billie Jean - class is timeless.
  • Romain Francoise: A study of file duplication in the Debian archive

    [Long post, sorry. If you're short on time skip to the end for the juicy parts.]

    It all started about a week ago when I decided to find out how many files were unique in the whole Debian archive, and how many were duplicated in the same package or in other packages. I had done some work on duplication detection before, and I knew that the process involves getting some kind of hash value of every document, then finding duplicate values in the list of hashes.

    I already had a local Debian mirror so the raw data was all there. I basically had two options: either compute the hash value of every file myself from the packages in my mirror, or find some other (ideally faster) way to determine if a file is unique. I quickly decided that the best option was to use the md5sums embedded in most Debian packages (in the DEBIAN/md5sums control file). That would give me MD5 hashes of all regular files, excluding conffiles.

    So the first step was to check how many packages have embedded md5sums, and a simple script showed that less than 3% don't have them. This first check exposed a bug in python-debian, which was duly reported. Along the way, my post prompted a discussion on debian-devel about the state of md5sums, and I set up a daily check to keep track of things.

    The next step was to make sure that all md5sum-enabled packages had usable information. It turned out that python-debian choked on 103 packages because of embedded spaces in filenames, and that it also had a blocker bug when used in Python 2.4. I filed two more bugs.

    All that remained was the easiest part: write the program to find duplicate files. I did that, and I now have all kinds of funny statistics:If you're still reading so far, thanks! You can see the report file, and the program itself. If you want to run it, note that it's optimized for speed rather than memory efficiency; it runs in under 3 minutes but uses up to 1.5GB of memory (my home desktop has 4GB).

    The grand conclusion is that all things considered, there is very little file duplication in the archive: the #1 duplicate represents less than 0.4% of the two million analyzed files, and it doesn't actually use any space since it's an empty file... :)

    Peter Makholm: On handling email

    Watching the popular Inbox Zero video would probally improve my email handling skills better. But this took less time and I wanted to play around with Net::XMPP;
    
    #!/usr/bin/perl -l
    use warnings;
    use strict;
    use Config::Simple;
    use Linux::Inotify2;
    use Email::Abstract;
    use Net::XMPP; # aka Jabber and Gtalk
    use POSIX;
    my $DEBUG = 0;
    our %config;
    Config::Simple->import_from("$ENV HOME /.mailnotifyrc", %config);
    deamonize() unless $DEBUG;
    my $inotify = new Linux::Inotify2
         or die "Unable to create new inotify object: $!";
    # Different Maildir implementations triggers different events"
    #   IN_MOVED_TO: rename("tmp/file","new/file")
    #   IN_CREATE:   link("tmp/file", "new/file") && unlink("tmp/file")
    #
    # At this point the file should be written and closed.
    $inotify->watch("$config Maildir /new", IN_MOVED_TO IN_CREATE, sub  
            my $e = shift;
            my ($fh, $xmpp, $email, $message);
            debug('Got event for ' . $e->fullname);
            open $fh, "<", $e->fullname;
            $email = new Email::Abstract join("", <$fh>);
            close $fh;
            $message = 'Mail from ' . $email->get_header('From') .
                       ' concerning "' . $email->get_header('Subject') .'"';
            debug("Getting ready to send [$message]");
            my $sender   = new Net::XMPP::JID ($config Sender );
            my $receiver = new Net::XMPP::JID ($config Receiver );
            $xmpp = new Net::XMPP::Client();
            $xmpp->Connect( hostname => $sender->GetServer,
                            port     => $config port    5222,
                            tls      => $config usetls    0,
                          );
            $xmpp->AuthSend( username => $sender->GetUserID,
                             password => $config password ,
                             resource => $sender->GetResource   'mailnotify',
                           );
            $xmpp->MessageSend( to   => $receiver,
                                type => $config MessageType    'chat',
                                body => $message,
                              );
            $xmpp->Process(1);
            $xmpp->Disconnect;
            debug("Event done");
     );
    1 while $inotify->poll;
    sub deamonize  
            my $pid = fork();
            if ($pid)  
                    exit 0;
             
            ### close all input/output and separate
            ### from the parent process group
            open STDIN,  '</dev/null' or die "Can't open STDIN from /dev/null: [$!]n";
            open STDOUT, '>/dev/null' or die "Can't open STDOUT to /dev/null: [$!]n";
            open STDERR, '>&STDOUT'   or die "Can't open STDERR to STDOUT: [$!]n";
            ### Change to root dir to avoid locking a mounted file system
            ### does this mean to be chroot ?
            chdir '/'                 or die "Can't chdir to \"/\": [$!]";
            ### Turn process into session leader, and ensure no controlling terminal
            POSIX::setsid();
     
    sub debug   return unless $DEBUG; print STDERR for @_;  
    __END__
    

    Miriam Ruiz: PiX Frogger: help the frog cross the street

    Does the name “Frogger” ring a bell? Introduced in 1981, developed by Konami and distributed by Sega/Gremlin, it’s one of the most classic from the earliest ages of computer videogames. PiX Frogger is a clone of the classic game Frogger, in which you must help a frog cross the street to avoid becoming roadkill by cars and trucks. The frog starts at the bottom of the screen and the only control the player has is navigating the direction for the frog to hop. The game allows 4 players playing simultaneously with the keyboard. PIX Frogger is on the way. Implemented in F nix, an interpreted script programming language, specially designed to developing and running 2D games and added to Debian repositories some weeks ago, it has really cute graphics and the visual layout has been nicely taken care of. The game in itself is quite simple, so it shouldn’t be hard for smaller kids to play, especially when they can compete against each other 4 at a time. For more grown-ups, I guess it’s so cool to have a version in Debian from this historical game. I’m sure many people missed it. I hope it enters the repositories soon, but meanwhile you can get it from here. Thake care while crossing the street, though.

    Rapha&#235;l Hertzog: Deprecating cvs.debian.org in favor of Alioth

    It’s very difficult to discuss with DSA and make things evolve if none of the DSA member express an interest in something related to your goal: here comes an example of a story like another in my desperate quest to try to help the DSA team. :-) Last time gluck ran out of space, a few non-DSA people (me, taggart, Ganneff, and others I might have forgotten) contacted people to ask them to clean their home directories. Following that discussion we discussed a bit about the opportunity to move some services from gluck on another host. Among the services on gluck, there’s cvs.debian.org. As an Alioth administrator, it struck me that cvs.debian.org is the only VCS service that’s handled by the DSA team. It seems logical to not duplicate the administrative work and have all the VCS repositories handled by the same team. The logical conclusion is that cvs.debian.org should be deprecated in favor of Alioth. So I made the suggestion in RT ticket #146 (login with guest/readonly). I got exactly zero response from DSA. No support and no opposition. So I went ahead and contacted the last users of cvs.debian.org: All in all, the debian-doc and debian-www folks are rather supportive of the move, but it requires adjustment to the build infrastructure, in particular to keep track of the status of translations. I have no answer from DSA and the buildd guys however. The web team started a wiki page to evaluate the VCS that they would switch to. Volunteers would be welcome to organize the conversion of the repositories and to fix the build infrastructure accordingly. This a nice little project for new contributors that want to learn. :-)

    Next.

    Previous.