Search Results: "spa"

6 March 2009

Robert McQueen: This is a local mail for local people, we ll have no trouble here!

all programs that interact with e-mail are broken in one way or another. Please be careful. - Lars Wirzenius
I seem to have a cunning knack of finding problems with configuring server software, particularly involving e-mail, where a) I can t find answers in Google, b) most people I go and ask for help say they d usually ask me such things, and c) if I go onto IRC or mailing lists I end up helping other people and not getting any help with my problem. It s quite likely this is just because I m something of a perfectionist, so the ridiculous crappy hacks people come up with and seem content to entrust their mail to are unacceptable to me for one reason or another. Anyway, in my ongoing quest for the perfect mail system, I ve painted myself into a corner again. (I m currently running with postfix, postgrey, clamav-milter, dspam, dovecot using LDA, managesieve and the cmusieve & antispam plugins. If I can get the current incarnation working, I ve had enough requests to write up a full HOWTO, and seen enough around with pretty questionable content, that I ll probably do it before too long.) I ve got postfix s local transport configured to hand mail to dspam over LMTP, using mailbox_transport = lmtp:unix:/dspam/dspam.sock. dspam is configured to listen here, add X-DSPAM-Result and signature headers, and then deliver the mail with dovecot s deliver LDA (which I ve set to 4750 root:dspam). From dspam.conf: ServerParameters "--deliver=innocent,spam -d %u"
ServerDomainSocketPath "/var/spool/postfix/dspam/dspam.sock"
...
Preference "spamAction=tag"
Preference "signatureLocation=headers"
...
TrustedDeliveryAgent "/usr/lib/dovecot/deliver" My dovecot configuration is pretty standard, using PAM for both passdb and userdb, and provides the auth-master socket that deliver needs. The problem I have is that postfix s local transport is qualifying the local username with the FQDN of the machine before delivering it to dspam with LMTP (the local mail transfer protocol), even for locally-originated mail which was only addressed with a bare username! dspam doesn t mangle it or care if the user is local or not, and then cheerfully invokes deliver -d robot101@omega.example.co.uk, which returns EX_NOUSER (addressee unknown) because my username is just robot101. From mail robot101: Mar 6 02:25:40 omega postfix/pickup[13607]: DAA4942F41F: uid=1000 from=<robot101>
Mar 6 02:25:40 omega postfix/cleanup[13637]: DAA4942F41F: message-id=<20090306022540.DAA4942F41F@omega.example.co.uk>
Mar 6 02:25:40 omega postfix/qmgr[12552]: DAA4942F41F: from=<robot101@omega.example.co.uk>, size=339, nrcpt=1 (queue active)
Mar 6 02:25:40 omega dovecot: auth(default): passwd(robot101@omega.example.co.uk): unknown user
Mar 6 02:25:40 omega dspam[13527]: Delivery agent returned exit code 67: /usr/lib/dovecot/deliver -d robot101@omega.example.co.uk
Mar 6 02:25:40 omega postfix/lmtp[13640]: DAA4942F41F: to=<robot101@omega.example.co.uk>, orig_to=<robot101>, relay=omega.example.co.uk[/dspam/dspam.sock], delay=0.08, delays=0.05/0.01/0.01/0.03, dsn=4.3.0, status=deferred (host omega.example.co.uk[/dspam/dspam.sock] said: 421 4.3.0 <robot101@omega.example.co.uk> Delivered (in reply to end of DATA command)) So, no e-mail for me. Dearest lazyweb, which of the three components is behaving wrongly, and how can I fix it? (And no, I m not just going to switch to GMail. I store my data on hard drives, which are sometimes in servers, not in the cloud . Until about a month ago, most people I knew spoke about clouds which were made of particles of water in the sky, rather than as a data storage media. What if it rains? ;)

Robert McQueen: This is a local mail for local people, we ll have no trouble here!

all programs that interact with e-mail are broken in one way or another. Please be careful. - Lars Wirzenius
I seem to have a cunning knack of finding problems with configuring server software, particularly involving e-mail, where a) I can t find answers in Google, b) most people I go and ask for help say they d usually ask me such things, and c) if I go onto IRC or mailing lists I end up helping other people and not getting any help with my problem. It s quite likely this is just because I m something of a perfectionist, so the ridiculous crappy hacks people come up with and seem content to entrust their mail to are unacceptable to me for one reason or another. Anyway, in my ongoing quest for the perfect mail system, I ve painted myself into a corner again. (I m currently running with postfix, postgrey, clamav-milter, dspam, dovecot using LDA, managesieve and the cmusieve & antispam plugins. If I can get the current incarnation working, I ve had enough requests to write up a full HOWTO, and seen enough around with pretty questionable content, that I ll probably do it before too long.) I ve got postfix s local transport configured to hand mail to dspam over LMTP, using mailbox_transport = lmtp:unix:/dspam/dspam.sock. dspam is configured to listen here, add X-DSPAM-Result and signature headers, and then deliver the mail with dovecot s deliver LDA (which I ve set to 4750 root:dspam). From dspam.conf: ServerParameters "--deliver=innocent,spam -d %u"
ServerDomainSocketPath "/var/spool/postfix/dspam/dspam.sock"
...
Preference "spamAction=tag"
Preference "signatureLocation=headers"
...
TrustedDeliveryAgent "/usr/lib/dovecot/deliver" My dovecot configuration is pretty standard, using PAM for both passdb and userdb, and provides the auth-master socket that deliver needs. The problem I have is that postfix s local transport is qualifying the local username with the FQDN of the machine before delivering it to dspam with LMTP (the local mail transfer protocol), even for locally-originated mail which was only addressed with a bare username! dspam doesn t mangle it or care if the user is local or not, and then cheerfully invokes deliver -d robot101@omega.example.co.uk, which returns EX_NOUSER (addressee unknown) because my username is just robot101. From mail robot101: Mar 6 02:25:40 omega postfix/pickup[13607]: DAA4942F41F: uid=1000 from=<robot101>
Mar 6 02:25:40 omega postfix/cleanup[13637]: DAA4942F41F: message-id=<20090306022540.DAA4942F41F@omega.example.co.uk>
Mar 6 02:25:40 omega postfix/qmgr[12552]: DAA4942F41F: from=<robot101@omega.example.co.uk>, size=339, nrcpt=1 (queue active)
Mar 6 02:25:40 omega dovecot: auth(default): passwd(robot101@omega.example.co.uk): unknown user
Mar 6 02:25:40 omega dspam[13527]: Delivery agent returned exit code 67: /usr/lib/dovecot/deliver -d robot101@omega.example.co.uk
Mar 6 02:25:40 omega postfix/lmtp[13640]: DAA4942F41F: to=<robot101@omega.example.co.uk>, orig_to=<robot101>, relay=omega.example.co.uk[/dspam/dspam.sock], delay=0.08, delays=0.05/0.01/0.01/0.03, dsn=4.3.0, status=deferred (host omega.example.co.uk[/dspam/dspam.sock] said: 421 4.3.0 <robot101@omega.example.co.uk> Delivered (in reply to end of DATA command)) So, no e-mail for me. Dearest lazyweb, which of the three components is behaving wrongly, and how can I fix it? (And no, I m not just going to switch to GMail. I store my data on hard drives, which are sometimes in servers, not in the cloud . Until about a month ago, most people I knew spoke about clouds which were made of particles of water in the sky, rather than as a data storage media. What if it rains? ;) Update: The problem is fixed! Even though arguably the problem is dspam s for not knowing which users are local or not, it s fixable in dovecot 1.1.x using the auth_username_format = %n option. Thanks so much to Angel Marin for helping me out. Update 2: There s also a patch for dspam floating around which adds a StripRcptDomain option, which makes the LMTP server truncate the e-mail address at the @, so essentially assumes everyone to be a local user. The problem with both of these fixes is that they re both blunt instruments which will break virtual users on the same host. I think the real fix would be something more like a LocalDomains option in dspam, to choose which domains should be considered local and truncated from the e-mail addresses for delivery purposes.

5 March 2009

Ingo Juergensmann: Gallery and missing exif information

Dear Lazyweb,
some time ago I upgraded my Gallery2 installation to 2.3-1 and since then I'm missing the exif information in the block below the foto. Although the exif plugin is activated and the exif plugin is properly configured (default). The Theme is configured to show "EXIF/IPTC photo info" to show the exif block on photo pages, but as you can see on this example there's no exif data showing up.

The exif console tool is showing the correct information for those images, so I can't figure why gallery2 fails here. Any tips how to get exif information back into my gallery2, dear lazyweb?

Martin F. Krafft: Don't get fooled by the cost-average effect

I have a bit of money invested in stocks and funds, passively managed by my bank. I ve made a bit with risky speculations, and I ve lost about the same amount with the markets dive of the last 18 months. Net benefit: zero cash, a bit of experience. Time to move on I am not an interesting customer, so I was all the more surprised to get a call from an employee of the bank the other day. He identified himself as Crisis Manager or something similarly ridiculous, mentioned at least 20 times that he was a regular employee and wouldn t receive provisions, and he also kept assuring me that he s doing himself what he s suggesting to me: leveraging the cost-average effect. I turned him down, probably a bit too briskily. Let s investigate why. Banks are good at devising ingenious tricks to fool people; it s their business after all. One of those tricks is the so-called cost-average effect. It s awesome, they say, because it is the first product that adapts to the ever-changing market: when prices are up, it buys smaller numbers of shares, and in low times, it stocks up on them by buying larger quantities, automatically. Fancy, no? It isn t, because the entire idea is to get your signature on a contract that obliges you to pay a fixed sum each month, with which shares are then bought. Obviously, when prices are up, numbers you get for your money are low, and vice versa. Observe how well the banks manage to turn that basic rule into a selling point. They are good at that. In fact, that s probably all they re good at. Obviously , inflation is also factored in, so the monthly fixed sum actually increases by a projected 5% each year, which they call inflation protection . While this strategy investing fixed-sum installments rather than lump sums certainly has its benefits, and can level out the risk of throwing a large sum of money at the market, it is commonly considered a suboptimal strategy, especially when employed over longer spans of time. For more information, I refer you to the frequently cited article Nobody gains from dollar cost averaging analytical, numerical and empirical results , published in the 1992/1993 issue of the Financial Services Review journal. So why would the bank approach me with this offer ? Let s go hypothetical and check out Ashley s investment practices: Ashley bought a fund five years ago at 100 per share, and it s now down to 80 . The loss is thus 20%, a -4% factor (20% / 5 years) that affects the overall performance of the portfolio. Ashley doesn t really care much about the percentages. After all, the only thing that counts is the worth of the shares when sold right now. Everything else is history and only influences the size of the tears one weeps in times like this, which is a psychological malfunction, not a function of the financial markets crisis. The bank, however, does not like negative performance values. Positive percentages yield happy (blinded?) customers, who get much more enthusiastic (greedy?) to put more money into the giant machine that makes the banks tick which is still in the process of blowing up into their faces. But negative performance values cause investors to become conservative and leery. They d rather hold on to their hard-earned cash for fear of losing more. That s the flip-side of aforementioned psychological malfunction, and this is the core of the reason why some of the big investors get bigger even in times like this they do not succomb to the feeling . The banks, however, need cash, and loads of it, and while the state is busy printing notes, they take every chance to get more. Private clients are wonderful prey, as they are less informed about the system than investors. So, to get cash from the clients, the banks have to circumvent the psychological malfunction. They cannot boost the market value share, but there are two values that factor into a percentage, and the cost-average effect, sold as the best thing since sliced bread, is nothing but a way to affect the other number: the sale price, the divisor. If the fund stays below the original sale price (100 ), each time Ashley pumps more cash into the machinery each month, this divisor decreases, effectively decreasing the percentage value. In the long run, this yields happier customers, who are willing to put more money in, especially if the performance turns positive. The baseline is now lower, so a return to previous heights would correspond to larger performance values than ever before. Obviously, a positive market causes the baseline to grow, but the cost-average deals continues to pay off for the banks. Since Ashley put that signature on the contract, the bank has a guaranteed cash flow for the contract s duration, while Ashley wonders why the percentage remains dampened. Cost-average means decreasing the percentage, in good as well as in bad times. Just now, the bank doesn t have to work so hard to get even more money from you. It gets worse: usually, those contracts are embedded in some sort of life insurance deals with grand promises in 40 years to come, and over all that time, it ll be quite clear who wins and who loses. Scary, huh? I was going to end with the advice to stay away from the cost-average effect, but I think I can just as well make it more general: if you don t understand a deal, don t sign it. If you abide by that rule, you would stay away from cost-average effect deals all by yourself. Maybe my little writeup has helped, nonetheless. NP: DJ Ti sto: Live at Innercity

Eddy Petri&#537;or: Glest: how upstream can be a real dick

At some point in the past, when I was still thinking that Glest can be packaged in a sane way, I had a conversation with upstream developer martinho on the Glest forum (they call it board) about them integrating a patch in order to support big endian machines.

After some messages on their board, it was clear that they don't want to integrate any patches and started to throw out random fake arguments against the integration of my patch and other patches.

When he figured out that I replied and debunked all of his fake arguments, he went on and actually removed my posts from the forum. It was so weird since I have never seen that kind of behaviour before and some of his posts and other people's posts were replies to my posts and things didn't made any sense after the deletion.

At that moment I have managed to save one small bit of the conversation via Google's cache, but now I have finally decided to make it public.


Note: Back then, I kept this post private in the hope that I was maybe too involved in the conversation at that moment to make a rational decision about it, and hoped things would get better in time. They didn't. Meanwhile they migrated to another forum software and they said they lost most/all(?) of the previous content from the old forum.



Here is the conversation bit I managed to save.
Original text is in italic font. Regular font is for my current comments for clearing up the context of the discussion. The text retains the BB syntax and [quote]-d text is martinho's own words.



[quote="martinho"]thebohemian, eddyd,

I think you guys are missing the point, this is not the place for any [/quote]

Congrats, you are avoiding to answer the points I have raised.

[quote]
- The Glest project is made by a bunch of friends for fun, we ask for nothing, we answer to no one. So I beg you not co complain, because it is out of order.
[/quote]

In that case I don't see the point of the "development" forum. care to explain?
And, from my PoV, these are your users you're arrogantly dismissing.


The following was a reply to my request to add big endian support through a patch I provided at that time. The patch wasn't complete to actually make glest work on BE machines, but it didn't broke little endian support at all. He was trying to argue that there were no valid big endian machines on which the game would run. Integrating the patch would have made my life a lot easier in continuing the work on the patch.

[quote]
- Our target plarforms are PCs running Windows (I'm responsible for this) or Linux (matze is the boss here). We don't care if Glest doesn't run in a big endian dishwasher with Solaris.
[/quote]

Linux on PowerPC is a viable platform. So is Mac OS X on PowerPC.



I made the mistake of saying that some parts of the code where not that clean and could be improved. More on that below.

[quote]
- One of the keys of Glest success is how extremely clean our code is. Bringing random people in would break this, and we are not going to do it. [/quote]

So you are the only keepers of true knowledge of clean code? Hahahaha!

There were at least 3 patches that provided this through word/double-word swapping. I don't see how embedding the patch can affect the unity of the code.

BTW, you are aware you can "carve" people into coding in your style before accepting in, aren't you?



I suggested using int8, uint8, int16, uint16 in order to make sure that the parameters are of the expected fixed size since plain char, int, long, long long have different sizes depending on the architecture. The usage of int and int16 was inconsistent and I also suggested using always the numbered variants.

I said the approach that was then taken in code was not portable and that was the reason for the proposed change.

[quote]
- Glest has been coded with portability in mind, some of your comments about portability are really naive (compilers don't care if you declare function parameters with fixed byte length, they just use whatever is the fastest int in the platform). Fixed length data types are used when needed,
[/quote]

What? When did I say anything about that? I said the code uses inconsistently "int" and "int16" and should use the later always because the code assumes the sizes on i386, which are not always applicable - see long on amd64.


If you mean that I can't enforce the width of the parameter, then you should know better, those types are defined in a platform dependent way to be platform independent. Yes, there are cases where [b]on the stack[/b] the size of the passed data is bigger (they might be padded), but that is transparent to the functions and, in spite of that, the functions will see the proper values for the parameters.

Also, the packing behaviour is compiler and architecture dependent, and some compilers produce the weirdest packing (if you are allowed access to such docs, look for Green Hills compilers' for V850 and you'll be stunned).


Please, don't make a fool of yourself by trying to discredit me, just not to loose face.

Later he just removed my posts not to loose face.




As a sane programmer, I made my patch to do the swapping only on big endian platforms, and the code was enabled in through the precompiler, so there was no performance hit on the little endian arches.

Of course, any more-than-newbie programmer would agree that having two data sets for a single game is stupid and is better to do the swapping in the application itself to avoid space wastage and maintaing an alternate big endian data set.

[quote]
no more, no less. About endianness, we don't care. If we wanted to fix the problem anyway, the right way is to do the byte swap in the data, this is how it is done in commercial games, since it is more efficient.
[/quote]

You just proved how much bullshit you can deliver in one phrase just for the sake of winning an argument. All portable games and all applications that have data files prefer to do swapping on the spot just because swapping the data on the disk (or providing BE and LE data) is plain stupid and inefficient from a disk and bandwidth PoV.

For Glest, swapping the data after reading from disk is not even affecting performance in a manner that could be significant or noticed, not even for a processor running at 500MHz, since that would be done only on load, once, and just for the bits of data which are relevant; these are mostly headers, since the actual data, in most cases, is just a long byte vector. This translates in not more than 3 or 4 instructions in assembler (an init, an exchange, an increment and/or a loop instruction), and that would happen before starting a game, which is dead time anyway and wouldn't be noticed unless the data would be huge (order of hundreds of MB or GB) and the whole data set would need swapping.

Even if the data itself (not headers) represents data which needs to be swapped, that is still irrelevant in terms of speed.


BTW, the reason that in commercial games it might be done like that is because there the games are distributed in a package per platform anyway and having two data sets is not an issue since they are using the same amount of CDs/DVDs, but doing the same for a game like Glest would mean doubling the storage used and it would increase the complexity for a release, not to count the confusion of the users.

I still doubt commercial games do this, but I might be wrong.

Eddy Petri&#537;or: Glest fuckyou: From the book of 'we want people to think we're open'

The Glest FAQFuckYou says (emphasis is mine):

[..] as a rule of thumb we don t integrate any patches in our mainline, since that would require us to check them first and then maintain them, however feel free to post them at the board.

Glest development sucks. I really hope Glest Advanced Engine effort takes speed and replaces Glest.

This is the main reason I don't feel any incentive to maintain this package in Debian.

4 March 2009

Uwe Hermann: Building an ARM cross-toolchain with binutils, gcc, newlib, and gdb from source

I've been planning to write about building custom ARM toolchains for a while (I used stuff from gnuarm.com in the past, but I switched to the lastest and greatest upstream versions at some point). Among other things, recent upstream versions now have ARM Cortex support. First you will need a few base utilities and libs: 
  $ apt-get install flex bison libgmp3-dev libmpfr-dev autoconf texinfo build-essential
Then you can use my tiny build-arm-toolchain script, which will download, build, and install the whole toolchain: 
  $ cat build-arm-toolchain
  #!/bin/sh
  # Written by Uwe Hermann <uwe@hermann-uwe.de>, released as public domain.
  TARGET=arm-elf                         # Or: TARGET=arm-none-eabi
  PREFIX=/tmp/arm-cortex-toolchain       # Install location of your final toolchain
  PARALLEL="-j 2"                        # Or: PARALLEL=""
  BINUTILS=binutils-2.19.1
  GCC=gcc-4.3.3
  NEWLIB=newlib-1.17.0
  GDB=gdb-6.8
  export PATH="$PATH:$PREFIX/bin"
  mkdir build
  wget -c http://ftp.gnu.org/gnu/binutils/$BINUTILS.tar.bz2
  tar xfvj $BINUTILS.tar.bz2
  cd build
  ../$BINUTILS/configure --target=$TARGET --prefix=$PREFIX --enable-interwork --enable-multilib \
    --with-gnu-as --with-gnu-ld --disable-nls
  make $PARALLEL
  make install
  cd ..
  rm -rf build/* $BINUTILS $BINUTILS.tar.bz2
  wget -c ftp://ftp.gnu.org/gnu/gcc/$GCC/$GCC.tar.bz2
  tar xfvj $GCC.tar.bz2
  cd build
  ../$GCC/configure --target=$TARGET --prefix=$PREFIX --enable-interwork --enable-multilib \
    --enable-languages="c" --with-newlib --without-headers --disable-shared --with-gnu-as --with-gnu-ld
  make $PARALLEL all-gcc
  make install-gcc
  cd ..
  rm -rf build/* $GCC.tar.bz2
  wget -c ftp://sources.redhat.com/pub/newlib/$NEWLIB.tar.gz
  tar xfvz $NEWLIB.tar.gz
  cd build
  ../$NEWLIB/configure --target=$TARGET --prefix=$PREFIX --enable-interwork --enable-multilib \
    --with-gnu-as --with-gnu-ld --disable-nls
  make $PARALLEL
  make install
  cd ..
  rm -rf build/* $NEWLIB $NEWLIB.tar.gz
  # Yes, you need to build gcc again!
  cd build
  ../$GCC/configure --target=$TARGET --prefix=$PREFIX --enable-interwork --enable-multilib \
    --enable-languages="c,c++" --with-newlib --disable-shared --with-gnu-as --with-gnu-ld
  make $PARALLEL
  make install
  cd ..
  rm -rf build/* $GCC
  wget -c ftp://ftp.gnu.org/gnu/gdb/$GDB.tar.bz2
  tar xfvj $GDB.tar.bz2
  cd build
  ../$GDB/configure --target=$TARGET --prefix=$PREFIX --enable-interwork --enable-multilib
  make $PARALLEL
  make install
  cd ..
  rm -rf build $GDB $GDB.tar.bz2
The final toolchain is located in /tmp/arm-cortex-toolchain per default, and is ca. 170 MB in size. I explicitly created the build script in such a way that it minimizes the amount of disk space used during the build (ca. 1.2 GB or so, compared to more than 3 GB in the "naive" approach). Using the "-j 2" option for make (see script) you can speed up the build quite a bit on multi-core machines (ca. 30 minutes vs. 60 minutes on an AMD X2 dual-core box). Also, you can change the script to build for other target variants if you want to (arm-elf or arm-none-eabi, for example). Checkout the blog entry How to build arm gnu gcc toolchain for Mac OS X by Piotr Esden-Tempski for similar instructions for Mac OS X users. Oh, and while I'm at it does anybody have any idea why there are no pre-built toolchains for embedded (microcontroller) ARM targets in Debian? There are some toolchains for other microcontroller architectures (avr, m68hc1x, h8300, z80) but not too much other stuff. Is there some specific reason for the missing ARM toolchains (other than "nobody cared enough yet")? I have heard about Emdebian, but from a quick look that seems to be more intended for toolchains with Linux/libc, not for microcontroller firmware (i.e. no MMU, no Linux, no libc etc.), but maybe I'm wrong?

Martin F. Krafft: Searching Tangerine Dream's Barbakane

Dear recording industry: I hate almost all of you. One of my all-time favourite albums, Poland: The Warsaw Concert by Tangerine Dream, has been castrated, probably to save cost by squeezing a double LP onto a single CD. Today, Wikipedia turned my world upside down. Screw you, money-greedy recording industry. Please fuck off and leave music to those who care. This includes smaller labels who know what they are doing. I now must have the full-length track. Unfortunately, it s seemingly impossible to come by. I searched the Web, even dreaded sites like Ebay, but after an hour I gave up without success. There s a seller on Amazon, but s/he consistently gets negative feedback, so that s a bit risky for the price asked. Thus, dear world, please save me: does anyone have the full-length third track of that album, titled Barbakane , in a high-quality digital recording and would let me have it? I will pose with the album and today s newspaper to prove legal ownership. Heck, go ahead and make a ridiculous request, such as standing on my head. I m desperate. NP: Tangerine Dream: Phaedra

3 March 2009

Uwe Hermann: How to use the full size of your (x)term when connecting to some other box serially with minicom

minicom terminal size 1 Just as a reminder for myself, as I'll probably need this more often in the future (and maybe it's helpful for others): Per default, if you use a big xterm (232x74 in my case) where you start minicom, the $COLUMNS and $LINES environment variables will be 80x24 nevertheless, and those applications which honor them (vim, for example) will be too small and not use the full xterm size of 232x74. 
  $ minicom
  $ echo $COLUMNS; echo $LINES
  80
  24
minicom terminal size 2 You can fix that like this: 
  $ eval  resize 
  $ echo $COLUMNS; echo $LINES
  232
  74
After you do eval resize the variables are updated and vim will use the full xterm size. That's all.

Asheesh Laroia: Segmentation grace

Ladies and gentlemen, it seems that Google has finally done it: they shipped Multics for 386 (and compatible). NoiseBridge, this evening: I descended the stairs after teaching my introduction to programming class and found Geoff Schmidt in our San Francisco hacker space. I sat next to Geoff and overheard a lovely conversation with Mike Kan. Geoff told the sad tale of a beautiful, efficient, and dead operating system: Multics. Multics was written in the 1960s as a fast operating system for many users to share an expensive mainframe. It posed design problems never seriously tackled before, and after a decade, it had a practically perfect implementation for each of them. To achieve speed, it expected help from the hardware: registers on the processor split memory into different "segments," creating safe zones for each program to run in. The segment system was powerful and secure enough that a running process could execute code from another without the operating system kernel getting involved. The result was a fantasy come true: protection rings creating legendary security, the flexibility of a multi-user, multi-tasking operating system, and all this at hardly any performance overhead. (Naturally, such perfection is the result of years of work at MIT. So is Geoff.) This perfection came at a price: while Multics was uniquely well-designed and well-integrated, it expected specific support from the hardware it ran on. Geoff's expression deflated, and he pointed out that another operating system arrived on the scene: UNIX. Anyone could study the UNIX source code, and it ran on whatever you gave it. UNIX (a joke on the name Multics) eventually won by being worse: it was slow, unreliable, and worst of all: incorrect. But anyone could read it and make it work on the computer he happened to have. By the mid-1970s, UNIX's dominance over Multics was clear. Geoff skipped forward a decade to the 1980s. Intel had wanted to build a CPU that could be used as a modern computer, and users had shown that the puny memory protection system offered by the 286 wasn't adequate. The chip designers went back to the drawing board, and they brought back features that Multics invented: segmentation registers and protection rings. When shown these powerful, complex features, today's operating systems mostly ignore these Multics tricks and do the least work possible to build a UNIX-like flat memory model. Mike interrupted and howled about how he can't buy a "real Macintosh" anymore; Apple's computers were once based on a simple architecture, but now they are built with the same complex Intel CPUs everyone else uses. But I am writing this because of the release of Google Native Client, a browser extension that allows your computer to securely run machine code written by untrusted people on the Internet. How can it achieve this fantasy? The Wikipedia article summarizes the native client research paper:
Native Client is notable for its novel sandboxing technique which makes use of the x86 architecture's rarely-used segmentation facility.

Lucas Nussbaum: Creating a large file without zeroing it: update

Given the large number of comments I got (26!), I feel obliged to post a summary of what was said. First, the problem:
I want to create a large file (let s say 10 GB) to use as swap space. This file can t be a sparse file (a file with holes, see wikipedia if you don t know about sparse files).
Since I m going to mkswap it, I don t care about the data that is actually in that file after creating it. The stupid way (but only solution on ext3) to create it is to fill it with zeroes, with is very inefficient. Theodore Tso provided more information in a comment, which I m copying here:
Yes, it will work on ext4. A convenient which makes this easy to use can be found here at http://sandeen.fedorapeople.org/utilities/fallocate.c. It was written by Eric Sandeen, a former XFS developer who now works for Red Hat, who has been a big help making sure ext4 will be ready for Fedora and Red Hat Enterprise Linux. (Well, I guess I shouldn t call him a former XFS developer since he still contributes patches to XFS now and then, but he s spending rather more time on ext4 these days.) One warning about the program; it calls the fallocate system call directly, and it doesn t quite have the right magic architecture-specific magic for certain architectures which have various restrictions on how arguments need to be passed to system calls. In particular, IIRC, I believe there will be issues on the s390 and powerpc architectures. The real right answer is to get fallocate into glibc; folks with pull into making glibc do the right thing, please talk to me. Glibc does have posix_fallocate(), which implements the POSIX interface. posix_fallocate() is wired to use the fallocate system call, for sufficiently modern versions of glibc. However, posix_fallocate() is probablematic for some applications; the problem is that for filesystems that don t support fallocate(), posix_fallocate() will simulate it by writing all zeros to the file. However, this is not necessarily the right thing to do; there are some applications that want fallocate() for speed reasons, but if the filesystem doesn t support it, they want to receive the ENOSPC error message, so they can try some other fallback which might or might not involve writing all zero s to the file. The other shortcoming with posix_fallocate() is that it doesn t support the FALLOC_FL_KEEP_SIZE flag. What this flag allows you to do is to allocate disk blocks to the file, but not to modify the i_size parameter. This allows you to allocate space for files such as log files and mail spool files so they will be contiguous on disk, but since i_size is not modified, programs that append to file won t get confused, and tail -f will continue to work. For example, if you know that your log files are normally approximately 10 megs a day, you can fallocate 10 megabytes, and then the log file will be contiguous on disk, and the space is guaranteed to be there (since it is already allocated). When you compress the log file at the end of the day, if the log file ended up being slightly smaller than 10 megs, the extra blocks will be discarded when you compress the file, or if you like, you can explicitly trim away the excess using ftruncate().
fallocate works fine: creating a 20 GB file is almost immediate. Also, syncing or umounting the filesystem is also immediate, and reading the file returns only zeros. I m not sure how it is implemented, but it looks nice :-). However, it still doesn t solve my initial problem: mkswap works, but not swapon: 
:/tmp# touch tmp
:/tmp# /root/fallocate -l 10g tmp
:/tmp# ls -lh tmp
-rw-r--r-- 1 root root 10G Mar  3 11:01 tmp
:/tmp# du tmp
10485764	tmp
:/tmp# mkswap tmp
Setting up swapspace version 1, size = 10737414 kB
no label, UUID=a316ce8e-cf33-412b-8dc0-e10d9f2ebdbb
:/tmp# strace swapon tmp
[...]
swapon( /tmp/tmp )                      = -1 EINVAL (Invalid argument)
write(2,  swapon: tmp: Invalid argument\n , 30swapon: tmp: Invalid argument
) = 30
exit_group(-1)
(swapon works fine if the file is created normally without using fallocate()). Any other ideas?

Obey Arthur Liu: Google Summer of Code 2009 at Debian needs you

2009-summer-of-code-logo-final-r3-no-url-011 In case you ve been living under a rock these past years, I shouldn t have to tell you what this is about :). Well, just in case.
The Google Summer of Code (GSoC) is an international program that offers student developers stipends to write code for various open source software projects. Debian has participated since 2006, mentoring dozens of students on Debian projects. The important part of the 2009 edition of the Google Summer of Code is going to start next week with the Organizations application period (March 9th). By that time, we should have listed a reasonable number of ideas on the dedicated wiki page. We will try this year to improve on the performance of the previous years, starting first with more and more dedicated manpower. If you have some time to spare to help manage our Summer of Code bid this year, we can make it a much better experience for the students and for Debian. Help is immediately needed in many areas. Some examples:

By mid-March, we will need: Further in the spring: In the summer: As you can see, there is much work ahead. It s time to pitch out your favorite pet project :)
Remember! The Summer of Code has become a prominent outreach forum for open source organizations. It puts a spotlight on each of them to provide a well-rounded and deep experience of open source development. Right now, you can help with the Wiki, join us on IRC on #debian-soc on OFTC or join the mailing-list at soc-coordination on alioth. The short flyers-friendly URL for the Summer of Code at Debian is : <http://wiki.debian.org/gsoc>. Publicize it! See you this summer!

2 March 2009

Ingo Juergensmann: HowTo: Migrate RAID1 to RAID5

As I wrote last week (German), I had the plan to make a RAID5 out of my existing RAID1 after I received my third 1 TB harddisk. Migrating to RAID5 doesn't work online yet, because mdadm doesn't support the use of --grow and --level together, as you can read in the man page when looking for "-l, --level":


-l, --level=
Set raid level. When used with --create, options are: linear, raid0, 0, stripe, raid1, 1, mirror, raid4,
4, raid5, 5, raid6, 6, raid10, 10, multipath, mp, faulty. Obviously some of these are synonymous.

When used with --build, only linear, stripe, raid0, 0, raid1, multipath, mp, and faulty are valid.

Not yet supported with --grow.


But there's always a way to obtain your goal. If you're brave enough you can find the solution by a quick internet search. Ok, I'm not that brave to try this without a backup, but you shouldn't do that kind of work without having a good backup anyway.

But before I describe the way how to do it, as usual a strong disclaimer: use this howto on your own risk! There's no warranty that this will work for you and you should take care to have a backup of your data. Whoever is doing this migration without having a backup risks the loss of all the data!

I'll describe the migration using an example... there are three logical volumes (LVs) with 1 GB in size each. This is enough for testing purposes and demonstrates how the migration works. You can use real partitions like swap partitions if you want. The LVs are r1, r2, and r3 and are within the Volume Group (VG) /dev/vg:


r1 vg -wi-a- 1.00G
r2 vg -wi-a- 1.00G
r3 vg -wi-a- 1.00G


Two of these LVs are being setup for a RAID1 /dev/md5:

CODE:
muaddib:/home/ij# mdadm --create /dev/md5 -l 1 -n 2 /dev/vg/r[12]
mdadm: /dev/vg/r1 appears to be part of a raid array:
level=raid1 devices=2 ctime=Thu Feb 26 16:16:59 2009
mdadm: /dev/vg/r2 appears to be part of a raid array:
level=raid1 devices=2 ctime=Thu Feb 26 16:16:59 2009
Continue creating array? yes
mdadm: array /dev/md5 started.


The warning to be a part of another raid array can be ignored. It's just a testing array. The result should look similar to:


md5 : active raid1 dm-12[1] dm-11[0]
1048512 blocks [2/2] [UU]
[========>............] resync = 40.2% (422900/1048512) finish=0.7min speed=14096K/sec


Now a filesystem can be created on the raid array and some data can be stored there:

CODE:
muaddib:/home/ij# mkfs.xfs -f /dev/md5
meta-data=/dev/md5 isize=256 agcount=4, agsize=65532 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=262128, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=1200, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
muaddib:/home/ij# mount /dev/md5 /mnt
muaddib:/home/ij# cp pics/backdrops/*.jpg /mnt/
muaddib:/home/ij# dir /mnt/
amigager_back1.jpg Dansk-Dunes14.jpg Dansk-Nature-2.jpg
CapeCod_085.jpg Dansk-Dunes15.jpg Dansk-Priel.jpg
ct-1-p1600.jpg Dansk-Dunes16.jpg Dansk-Woods-1.jpg
ct-2-p1600.jpg Dansk-Dunes17.jpg earth_lights_lrg.jpg
Dansk-Beach1.jpg Dansk-Dunes18.jpg HanseSail-AlterStrom2.jpg
Dansk-Beach2.jpg Dansk-Dunes2.jpg HanseSail-AlterStrom.jpg
Dansk-Beach3.jpg Dansk-Dunes3.jpg HanseSail-Mast.jpg
Dansk-Beach4.jpg Dansk-Dunes4.jpg HanseSail-Warnemuende.jpg
Dansk-Beach5.jpg Dansk-Dunes5.jpg LinuxInside_p1600.jpg
Dansk-Beachstones.jpg Dansk-Dunes6.jpg prerow.jpg
Dansk-Cliffs.jpg Dansk-Dunes7.jpg sgi-1440.jpg
Dansk-Dunes10.jpg Dansk-Dunes8.jpg sgi.jpg
Dansk-Dunes11.jpg Dansk-Dunes9.jpg Sonnenuntergang-2.jpg
Dansk-Dunes12.jpg Dansk-Dunes.jpg Sonnenuntergang.jpg
Dansk-Dunes13.jpg Dansk-Nature-1.jpg
muaddib:/home/ij# umount /mnt


After umounting the real challenge arise: creation of a RAID5 label! Normally this is no issue, but I guess you like your data and think twice before proceeding... but hey! You have a backup anyway, don't you?!

CODE:
muaddib:/home/ij# mdadm --create /dev/md5 --level=5 --raid-devices=2 /dev/vg/r[12]
mdadm: /dev/vg/r1 appears to be part of a raid array:
level=raid1 devices=2 ctime=Mon Mar 2 18:18:09 2009
mdadm: /dev/vg/r2 appears to be part of a raid array:
level=raid1 devices=2 ctime=Mon Mar 2 18:18:09 2009
Continue creating array? yes
mdadm: array /dev/md5 started.
muaddib:/home/ij# cat /proc/mdstat
md5 : active (auto-read-only) raid5 dm-12[2](S) dm-11[0]
1048512 blocks level 5, 64k chunk, algorithm 2 [2/1] [U_]
muaddib:/home/ij# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md5 : active raid5 dm-12[2] dm-11[0]
1048512 blocks level 5, 64k chunk, algorithm 2 [2/1] [U_]
[>....................] recovery = 3.1% (33152/1048512) finish=0.5min speed=33152K/sec


After stopping /dev/md5 the new RAID5 label can be written. The data ought to be safe because of the RAID5 algorithm when running with 2 disks. To verify the theory you can mount /dev/md5 again. The next steps can be done with the array online.


muaddib:/home/ij# mount /dev/md5 /mnt
muaddib:/home/ij# dir /mnt/
amigager_back1.jpg Dansk-Dunes14.jpg Dansk-Nature-2.jpg
CapeCod_085.jpg Dansk-Dunes15.jpg Dansk-Priel.jpg
ct-1-p1600.jpg Dansk-Dunes16.jpg Dansk-Woods-1.jpg
ct-2-p1600.jpg Dansk-Dunes17.jpg earth_lights_lrg.jpg
Dansk-Beach1.jpg Dansk-Dunes18.jpg HanseSail-AlterStrom2.jpg
Dansk-Beach2.jpg Dansk-Dunes2.jpg HanseSail-AlterStrom.jpg
Dansk-Beach3.jpg Dansk-Dunes3.jpg HanseSail-Mast.jpg
Dansk-Beach4.jpg Dansk-Dunes4.jpg HanseSail-Warnemuende.jpg
Dansk-Beach5.jpg Dansk-Dunes5.jpg LinuxInside_p1600.jpg
Dansk-Beachstones.jpg Dansk-Dunes6.jpg prerow.jpg
Dansk-Cliffs.jpg Dansk-Dunes7.jpg sgi-1440.jpg
Dansk-Dunes10.jpg Dansk-Dunes8.jpg sgi.jpg
Dansk-Dunes11.jpg Dansk-Dunes9.jpg Sonnenuntergang-2.jpg
Dansk-Dunes12.jpg Dansk-Dunes.jpg Sonnenuntergang.jpg
Dansk-Dunes13.jpg Dansk-Nature-1.jpg


The next step is to add the third "disk":

CODE:
muaddib:/home/ij# mdadm --add /dev/md5 /dev/vg/r3
mdadm: added /dev/vg/r3
muaddib:/home/ij# cat /proc/mdstat
md5 : active raid5 dm-13[2](S) dm-12[1] dm-11[0]
1048512 blocks level 5, 64k chunk, algorithm 2 [2/2] [UU]


That was the first step. As you can see the third "disk" has been added as a spare device. Because we don't want a spare but a RAID5 we need to expand the number of disks to 3 and tell the RAID to grow it's capacity:

CODE:
muaddib:/home/ij# mdadm --grow /dev/md5 --raid-devices=3
mdadm: Need to backup 128K of critical section..
mdadm: ... critical section passed.
muaddib:/home/ij# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md5 : active raid5 dm-13[2] dm-12[1] dm-11[0]
1048512 blocks super 0.91 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
[>....................] reshape = 3.1% (33472/1048512) finish=3.0min speed=5578K/sec


The last two commands were made with the mounted filesystem that just needs to get enlarged. With XFS you make this happen by invoking xfs_growfs:

CODE:
muaddib:/home/ij# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/md5 1020M 21M 1000M 2% /mnt
muaddib:/home/ij# xfs_growfs /mnt
meta-data=/dev/md5 isize=256 agcount=4, agsize=65532 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=262128, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=1200, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 262128 to 524256
muaddib:/home/ij# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/md5 2.0G 21M 2.0G 1% /mnt


Done!

If you have an LVM on your RAID you'll need to use the appropriate LVM tools to enlarge the PVs and VGs as well. But the hardest work is already done. The reshaping can take a lot of time, as you can see here:


md4 : active raid5 sda6[0] sdc6[2] sdb6[1]
969161152 blocks super 0.91 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
[>....................] reshape = 2.8% (27990592/969161152) finish=1485.0min speed=10562K/sec


There might be other fallpits if you change your setup from 2-disk arrays to 3-disk arrays. For example you'll need to adopt the changes to your configuration files like /etc/mdadm/mdadm.conf. But this has not much to do with the migration from RAID1 to RAID5 itself.

Many thanks to Goswin Brederlow for his advice and writing a wishlist bug: #517731

Alastair McKinstry: Titan, by Stephen Baxter

Titan, by Stephen Baxter describes a near-future manned mission to Titan, Saturn's moon, by Space Shuttle, no less. This is the first book i've read by Stephen Baxter, and will probably not be the last. It's well written, the most science-based outer planets mission story i've read in since Arthur C. Clarke, though with a pessimistic edge. The story turns on a waning NASA using its remaining hardware on one big mission, to Titan, where life may have been discovered. Most of the book concentrates naturally enough on getting to Titan, rather than what they found when they get there, but Baxters ammonia-based chemistry is intriguing. The physics of what it would be like to walk on the moon are a little underdone, though. Its an understatement to say that Titan is cold: serious work on insulation would be needed to stop your base melting the "permafrost" and sinking. While we don't know what life might be present, it would be good to imagine the real tasks involved in being there. But it's his idea that NASA and others really don't want space that is most interesting. That the military want space as their baliwick, and would rather frustrate the development of technologies so that no other nations or groups interfere is well worth thinking about. To a degree, I would agree: frustrate, but not he puts it, stop. Sooner or later the technology has other uses on Earth to the point where even amateurs would launch their own satellites. The vast bulk of technology development for future space use will be done not only on Earth but for Earth use: the minaturization that NASA and the military did for the space race in the 1960's now happens for consumer electronics. 3 D fabricators are being built for prototyping, and the prospect of getting a self-contained space mission together (not reliant on Earth) draws closer. I fully expect private groups, rather than national efforts, will be the first to colonise outer space, assembling pre-existing technologies. Tags , ,

Ian Wienand: YUI ButtonGroup Notes

Some tips and things to check if your YUI ButtonGroup isn't behaving as you wish it would. Hopefully, this will save someone else a few hours!

Theodore Ts'o: SSD s, Journaling, and noatime/relatime

On occasion, you will see the advice that the ext3 file system is not suitable for Solid State Disks (SSD s) due to the extra writes caused by journaling and so Linux users using SSD s should use ext2 instead. However, is this folk wisdom actually true? This weekend, I decided to measure exactly what the write overhead of journaling actually is in actual practice. For this experiment I used ext4, since I recently added a feature to track the amount of writes to the file system over its lifetime (to better gauge the wear and tear on an SSD). Ext4 also has the advantage that (starting in 2.6.29), it can support operations with and without a journal, allowing me to do a controlled experiment where I could manipulate only that one variable. The test workload I chose was a simple one: For the first test, I ran the test using no special mount options, and the only difference being the presence or absence of the has_journal feature. (That is, the first file system was created using mke2fs -t ext4 /dev/closure/testext4, while the second file system was created using mke2fs -t ext4 -O ^has_journal /dev/closure/testext4.)
Amount of data written (in megabytes) on an ext4 filesystem
Operation with journal w/o journal percent change
git clone 367.7 353.0 4.00%
make 231.1 203.4 12.0%
make clean 14.6 7.7 47.3%
What the results show is that metadata-heavy workloads, such as make clean, do result in almost twice the amount data written to disk. This is to be expected, since all changes to metadata blocks are first written to the journal and the journal transaction committed before the metadata is written to their final location on disk. However, for more common workloads where we are writing data as well as modifying filesystem metadata blocks, the difference is much smaller: 4% for the git clone, and 12% for the actual kernel compile. The noatime mount option Can we do better? Yes, if we mount the file system using the noatime mount option:
Amount of data written (in megabytes) on an ext4 filesystem mounted with noatime
Operation with journal w/o journal percent change
git clone 367.0 353.0 3.81%
make 207.6 199.4 3.95%
make clean 6.45 3.73 42.17%
This reduces the extra cost of the journal in the git clone and make steps to be just under 4%. What this shows is that most of the extra meta-data cost without the noatime mount option was caused by update to the last update time for kernel source files and directories. The relatime mount option There is a newer alternative to the noatime mount option, relatime. The relatime mount option updates the last access time of a file only if the last modified or last inode changed time is newer than the last accessed time. This allows programs to be able to determine whether a file has been read size it was last modified. The usual (actually, only) example that is given of such an application is the mutt mail-reader, which uses the last accessed time to determine if new mail has been delivered to Unix mail spool files. Unfortunately, relatime is not free. As you can see below, it has roughly double the overhead of noatime (but roughly half the overhead of using the standard Posix atime semantics):
Amount of data written (in megabytes) on an ext4 filesystem mounted with relatime
Operation with journal w/o journal percent change
git clone 366.6 353.0 3.71%
make 216.8 203.7 6.04%
make clean 13.34 6.97 45.75%
Personally, I don t think relatime is worth it. There are other ways of working around the issue with mutt for example, you can use Maildir-style mailboxes, or you can use mutt s check_mbox_size option. If the goal is to reduce unnecessary disk writes, I would mount my file systems using noatime, and use other workarounds as necessary. Alternatively, you can use chattr +A to set the noatime flag on all files and directories where you don t want noatime semantics, and then clear the flag for the Unix mbox files where you care about the atime updates. Since the noatime flag is inherited by default, you can get this behaviour by setting running chattr +A /mntpt right after the filesystem is first created and mounted; all files and directories created in that file system will have the noatime file inherited. Comparing ext3 and ext2 filesystems
Amount of data written (in megabytes) on an ext3 and ext2 filesystem
Operation ext3 ext2 percent change
git clone 374.6 357.2 4.64%
make 230.9 204.4 11.48%
make clean 14.56 6.54 55.08%
Finally, just to round things out, I tried the same experiment using the ext3 and ext2 file systems. The difference between these results and the ones involving ext4 are the result of the fact that ext2 does not have the directory index feature (aka htree support), and both ext2 and ext3 do not have extents support, but rather use the less efficient indirect block scheme. The ext2 and ext3 allocators are also someone different from each other, and from ext4. Still, the results are substantially similar with the first set of Posix-compliant atime update numbers (I didn t bother to do noatime and relatime benchmark runs with ext2 and ext3, but I expect the results would be similar.) Conclusion So given all of this, where did the common folk wisdom that ext3 was not suitable for SSD s come from? Some of it may have been from people worrying too much about extreme workloads such as make clean ; but while doubling the write load sounds bad, going from 4MB to 7MB worth of writes isn t that much compared to the write load of actually doing the kernel compile or populating the kernel source tree. No, the problem was that first generation SSD s had a very bad problem with what has been called the write amplification effect , where a 4k write might cause a 128k region of the SSD to be erased and rewritten. In addition in order to provide safety against system crashes, ext3 has more synchronous write operations that is where ext3 waits for the write operation to be complete before moving on, and this caused a very pronounced and noticeable stuttering effect which was fairly annoying to users. However, the next generation of SSD s, such as Intel s X25-M SSD, have worked around the write amplification affect. What else have we learned? First of all, for normal workloads that include data writes, the overhead from journaling is actually relatively small (between 4 and 12%, depending on the workload). Further, than much of this overhead can be reduced by enabling the noatime option, with relatime providing some benefit, but ultimately if the goal is to reduce your file system s write load, especially where an SSD is involved, I would strongly recommend the use of noatime over relatime. Related posts (automatically generated):
  1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
  2. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...
  3. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

1 March 2009

Dirk Eddelbuettel: Rcpp 0.6.4

A new maintenance version of Rcpp (now at 0.6.4) was just pushed to CRAN and has been uploaded to Debian. Rcpp is a set of utility classes that provide interfaces for transferring the major R data types to C++ and back which makes it easier to extend R with dynamically loadable code written in C or C++. This version changes how use the std namespace: all usage is now properly prefixed. Likewise, we now define R_NO_REMAP to not let R define utility functions as length() or error() which occassionally creates trouble with other include files -- so now use the more explicit forms Rf_length(), Rf_error() etc. Also, starting from this release, C++ class documentation created by Doxygen is included; it can also be seen from my box as both browsable html and a pdf file. Lastly, this version also adds a minor correction to the Windows build (spotted by Uwe and Simon).

Julian Andres Klode: debimg update - creating images


Much happened since the last time I wrote about debimg. The project is now registered on Alioth and has a mailing list. On the code side, there have also been several changes. First of all, the repository module has been merged into the master branch. This was the first step towards the creation of the image building, which happened today by introducing the image module. The code should be treated as Beta quality, but the project as a whole is Alpha, because the application utilizing debimg.core is still missing. As always, I hereby encourage to try out debimg, have a look at the examples, and help to develop it. Patches should be sent to the mailing list, created with git format-patch, and its usual settings (eg. prefixed with eg. [PATCH 1/2] ). The patches should be inline and validate in pyflakes and using pep8.py. See the README for some more recommendations. Because NEW seems to be really busy at the moment (and doko uploaded python2.6 and python3.1), I m not uploading the current state as 0.1.0~a1 to experimental, but wait until NEW is smaller (maybe it will be 0.1.0~b1 then). What is wanted/planned?
  • Compatibility configuration formats
    • (debimg.config.simple_cdd) A reimplementation of simple-cdd using debimg. This probably needs to wait until there is a module for dealing with the installer, but I want to implement an application to support simple-cdd configuration files.
    • (debimg.config.debian_cd) There could be an implementation of debian-cd s configuration format. This would allow people to easily try out debimg. We can only support a subset, though.
  • New configuration formats
    • (debimg.config.yaml) Debimg s native configuration format. This is the only one supported by the graphical frontends. Where possible, configuration between compatibility formats and this format will be provided.
  • Graphical Frontends:
    • (debimg.frontend.gtk2) The graphical GTK+ frontend for inexperienced users who just want to create their own disk.
    • (debimg.frontend.qt4) The graphical frontend written in QT4.
  • Text frontends
    • (debimg.frontend.text) The basic command-line frontend.
    • (debimg.ubuntu.frontend.text) A script to build Ubuntu images, which could be used to build the official Ubuntu Images. This requires interaction with the germinate tool for dependency resolution, and more. It s also not flexible enough for building custom images.
Requirements for 0.1 Beta
  • Implement at least one configuration format and the build frontend.
  • Have at least one external contributor.
More Changes Julian Andres Klode (6):
  • debimg/core/files: When creating a file object allow filename as source
  • debimg/core/resolver.py: Introduce Package.fullname, Package.component
  • debimg/core/repository.py: Merge the repository module.
  • debimg/core/resolver.py: Improve handling of certain dependency types
  • debimg/core/repository.py: Allow Repository.add_group to take a distro parameter
  • debimg/core/image.py: Introduce the image module.
Posted in Debian, debimg, Ubuntu

Frans Pop: The case of the self-perpetuating DNS errors

Ingredients: The last couple of days I've been plagued by some DNS errors that kept showing up in the logcheck mails for my home server which I was busy migrating from one box to another, doing an upgrade from etch/i386 to lenny/amd64 at the same time. So, plenty of stuff going on to confuse the issue. I kept getting the following messages every hour (anonymized): 
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
The times were fairly regular: once just before the hour, most 2 minutes after. I fetch mail at around that time, but also at other times, so possible but unlikely. The 2 minutes after was the first real clue: some cron job maybe? After disabling logcheck the messages no longer appeared in the log. Enable it again, and they were back. Additional confusion was caused by the fact that the domain had "debian" in its name, but it was somewhere obscure. So why was logcheck causing a lookup for that domain? This did confuse me enough to waste some time looking for some silly weird (default) configuration problem in some package. Enter spamassassin. Apparently that was parsing the message body, recognized "somedomain.org" as a host name, and proceded to do a DNS lookup as validity check. So we have the following loop, started off by something causing an initial DNS lookup for the domain, which fails and gets logged: Duh. I remember struggling with probably the same problem a couple of years ago, but then it was a lot more severe: masses of repeating DNS errors for obscure domains. At that time I failed to get to the bottom of it and ended up just ignoring the errors by adding the following option in my bind9 configuration: 
logging  
    category lame-servers   null;  ;
 ;
Anyway, now I just no longer pass logcheck mails through spamassassin. (Although filtering out these DNS errors in bind9 can be perfectly valid.)

Frans Pop: The case of the self-perpetuating DNS-errors

Ingredients: The last couple of days I've been plagued by some DNS errors that kept showing up in the logcheck mails for my home server which I was busy migrating from one box to another, doing an upgrade from etch/i386 to lenny/amd64 at the same time. So, plenty of stuff going on to confuse the issue. I kept getting the following messages every hour (anonymized): 
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
The times were fairly regular: once just before the hour, most 2 minutes after. I fetch mail at around that time, but also at other times, so possible but unlikely. The 2 minutes after was the first real clue: some cron job maybe? After disabling logcheck the messages no longer appeared in the log. Enable it again, and they were back. Additional confusion was caused by the fact that the domain had "debian" in its name, but it was somewhere obscure. So why was logcheck causing a lookup for that domain? This did confuse me enough to waste some time looking for some silly weird (default) configuration problem in some package. Enter spamassassin. Apparently that was parsing the message body, recognized "somedomain.org" as a host name, and proceded to do a DNS lookup as validity check. So we have the following loop, started off by something causing an initial DNS lookup for the domain, which fails and gets logged: Duh. I remember struggling with probably the same problem a couple of years ago, but then it was a lot more severe: masses of repeating DNS errors for obscure domains. At that time I failed to get to the bottom of it and ended up just ignoring the errors by adding the following option in my bind9 configuration: 
logging  
    category lame-servers   null;  ;
 ;
Anyway, now I just no longer pass logcheck mails through spamassassin. (Although filtering out these DNS errors in bind9 can be perfectly valid.)

Next.

Previous.