Search Results: "ag"

31 December 2008

Joachim Breitner: Handling explicit and implicit recusion in Haskell data

For a while I have been pondering over a problem that arises when your functionally written program has some state with cross references for example a list of users, each of which uses a number of computers, and a list of computers, each having an owner. Implicit referencing For doing queries on such data, it would be convenient if every reference is just the referenced object itself. Although we would visualize this as a graph, semantically, it is more like an infinite tree. This is possible in Haskell, due to laziness, and if you create the data structure cleverly, it even uses constant memory, no matter how deep you enter this infinite tree (a recent post of mine talks about this). A possible data definition would be:
data User = User  
userName :: String,
uses :: [Computer]

data Computer = Computer
computerName :: String,
owner :: User -- references the Users

data State = State [User] [Computer]

testState = let user = User "Conrad" [computer]
computer = Computer "Z3" user
in State [user] [computer]
Explicit referencing But such a representation is very unsuitable for updates (at least I can t think if a nice way of updating such a graph without breaking the internal cross-references) and serialization, which is a must for a HAppS based application. So what one would probably start with is this data structure:
data User = User  
userId :: Int,
userName :: String,
uses :: [Int] -- references the Computers

data Computer = Computer
computerId :: Int,
computerName :: String,
owner :: Int -- references the Users

data State = State [User] [Computer]

testState = State
[User 0 "Conrad" [1]]
[Computer 1 "Z3" 0]
I think the semantics of this are clear. Note that the referencing is currently not type-safe, but this can be provided by phantom types. Maybe I ll write more about that later. Now imaging you want to display the information about the first computer with your web application. You extract the information with let State _ cs = testState in head cs and pass that to your templating engine. But what if your template wants to display the name of the owner? It only has access to his userId. You would either need to know what information the template will ever need, extract that from the state beforehand and pass it along, or give the template access to the whole state. In that case, though, there has to be lookup-logic in your output code, which is also not nice. Woudln t it be nice if you could, in your application logic, work with the explicit references, which are easy to modify and store, but somehow turn that into the implicit referencing? Duplicated representation One way would be to have two unrelated sets of data structures, ExplicitState, ExplicitUser, ExplicitComputer, which use explicit identifiers to reference each other, and ImplicitState,... which are defined as the first representation of our state. It is then mostly trivial to write a function that converts ExplicitState to ImplicitState. The big disadvantage of this is that you have to maintain these two different hierarchies. It also means that every function on the state has to be defined twice, which often almost identical code. Clearly, this is not desirable. Annotated representation It would be more elegant if the state is stored in one data type that, controlled by a type parameter, comes in the one or the other representation. To do that, we need two types: One that contains a value, and one that contains just a reference:
newtype Id v = Id v deriving (Show, Typeable, Data)
newtype Ref v = Ref Int deriving (Show, Typeable, Data)
Then we need to adjust our data definitions, to make use of these. (I ll leave out the names, to keep the code smaller)
data User ref = User  
userId :: Int,
uses :: [ref (Computer ref)]

data Computer ref = Computer
computerId :: Int,
owner :: ref (User ref)

data State ref = State [User ref] [Computer ref]
Here we introduce a type parameter ref , which will later be either Id or Ref. Note that now a reference also states the object it is a reference for, which greatly increases type safety. Functions on these data types that don t work with the references will be polymorphic in the ref type parameter, so only need to be written once. A User Id is a complete user with all related data, while User Ref is a user with only references. And a Ref (User Ref) is reference to a user, which contains references... Not so kind kinds Did you notice the lack of a deriving clause? Our data structures have the relatively peculiar kind ((* -> *) -> *), which makes it hard for the compiler to derive instances for things like Show. But we already know that we will only use Id or Ref for the type variable, so we can use ghc s StandaloneDeriving language extension and have these instances created:
deriving instance Show (User Id)
deriving instance Show (User Ref)
deriving instance Show (Computer Id)
deriving instance Show (Computer Ref)
deriving instance Show (State Id)
deriving instance Show (State Ref)
Toggling a type parameter The next step is to write the conversion function. It will have type
unrefState :: State Ref -> State Id
For that, and for later, we need lookup functions:
unrefUserRef :: State Id -> Ref (User Ref) -> Id (User Id)
unrefUserRef (State l _) (Ref i) = Id $ fromJust $
find (\u@(User i' _) -> i == i') l
unrefComputerRef :: State Id -> Ref (Computer Ref) -> Id (Computer Id)
unrefComputerRef (State _ l) (Ref i) = Id $ fromJust $
find (\u@(Computer i' _) -> i == i') l
These expect a State (with implicit referencing) and a reference, and look up this reference. The function unrefState then looks like this:
unrefState :: State Ref -> State Id
unrefState (State us cs) =
let unrefState = State (map (unrefUser unrefState) us)
(map (unrefComp unrefState) cs)
in unrefState
where unrefUser :: State Id -> User Ref -> User Id
unrefUser s (User i refs) = User i (map (unrefComputerRef s) refs)

unrefComp :: State Id -> Computer Ref -> Computer Id
unrefComp s (Computer i ref) = Computer i (unrefUserRef s ref)
Note how we tie the knot in the let expression. This is the trick that ensures constant memory consumption, because every reference points back to the same place in memory. Satisfied already? So what do we have? We have no duplication of data types, we can write general functions, and we can resolve the explicit referencing. We can also easily write functions like unrefUser :: State Ref -> User Ref -> User Id, which transform just a part of the state. But writing unrefState is very tedious when the State becomes more complex. Each of the other unrefSomething functions are very similar, but need to be written anyways. This is unsatisfactory. What we want, is a generic function, something like
gunref :: State Ref -> a Ref -> a Id
which, given a state with explicit references, replaces all explicit references in the first argument (which could be State Ref or User Ref or anything like that) with implicit ones. This function can not exist, because we would not know anything about a and b. But maybe we can do this:
gunref :: (Data (a Id), Data (a Ref)) =>  State Ref -> a Ref -> a Id
Typeable and Data Before being able to do so, we need to derive Data for our types. We can start with
deriving instance Data (User Id)
deriving instance Data (User Ref)
deriving instance Data (Computer Id)
deriving instance Data (Computer Ref)
deriving instance Data (State Id)
deriving instance Data (State Ref
but that will complain about missing Typeable instances. Unfortunately, ghc s deriver for Typeable (even the stand-alone-one), does not handle our peculiar kind, so we need to do it by hand. With some help from quicksilver on #haskell, I got it to work:
instance Typeable1 ref => Typeable (User ref) where
typeOf _ = mkTyConApp (mkTyCon "User") [typeOf1 (undefined :: ref ())]
instance Typeable1 ref => Typeable (Computer ref) where
typeOf _ = mkTyConApp (mkTyCon "Computer") [typeOf1 (undefined :: ref ())]
instance Typeable1 ref => Typeable (State ref) where
typeOf _ = mkTyConApp (mkTyCon "State") [typeOf1 (undefined :: ref ())]
everywhere is not enough Turning to the documentation of Data.Generics, I notice with some disappointment that there is no function that is able to change a type they all seem to replace a value by another value of the same type. But the functions gfoldl and gunfold sounded like they could be used for this. Warning: What comes now is a very non-haskellish hack that subverts the type system, just to get the job done. Please read it with a healthy portion of distrust. If you know of a cleaner way of doing that, please tell me! Wrapped Data I want to do some heavy type hackery, so I need to disable haskell s type system. There is Data.Dynamic, but not even that is enough, as we need to carry a type s Data instance around as well. So let s wrap that up:
data AData where AData :: Data a => a -> AData
instance Show AData where show (AData a) = "<" ++ show (typeOf a) ++ ">"

fromADataE :: forall b. Data b => AData -> b
fromADataE (AData d) = case cast d of
Just v -> v
Nothing -> error $ "Type error, trying to convert " ++
show (typeOf d) ++ " to " ++
show (typeOf (undefined :: b)) ++ "."
There is also a function that converts an AData back to a normal type, if possible. If it s not possible, then there is a bug in our code, so we give an informative error message. AData transformers Similar to everywhere, we want to have transformers that combinable. They need to have the change to convert an arbitrary value, but also signal that they could not convert something (and this something has to be recursed into). I came up with this:
type ADataT = AData -> Maybe AData

extADT :: forall a b. (Data a, Data b) => ADataT -> (a -> b) -> ADataT
extADT at t a@(AData v) = case cast v of
Just x -> Just (AData (t x))
Nothing -> at a

doNothing :: ADataT
doNothing = const Nothing
ADataT is the type for such a transformer. doNothing will not transform anything, and extADT can be used to add any function to the list of tried transformers, in the spirit of extT. The ugly part To apply such a transformer, I want to use this function, which I ll describe in the code comments:
everywhereADT :: forall a b. (Data a, Data b) => ADataT -> a -> b

-- first check if we can transform this value already
everywhereADT f v = case f (AData v) of
-- if so, coerce it to the users requested type, which hopefully goes well
Just r -> fromADataE r
-- if not, we need to recurse into the data structure
Nothing -> recursed

-- for that, we first need to figure out the arguments to the
-- constructor. We store them in the untyped list
where args :: [AData]
(Const args) = gfoldl k z v
-- gfoldl lets us have a look at each argument. We wrap it in AData
-- and append it to the list
k (Const args) arg = Const (AData arg : args)
z start = Const []

-- We need the data constructor of the input data type. If the user did not want
-- it to be transformed, it will be the same
c = toConstr v

-- To give better error messages, we make sure that the outmost type constructor
-- of the requested type actually has the data constructor we were given. Otherwise
-- gunfold will complain in a non-helpful way
input_type = dataTypeRep (constrType c)
output_type = dataTypeRep (dataTypeOf (undefined :: b))

recursed = if input_type /= output_type
then error $ "Can not convert <" ++ show input_type ++ ">"++
" to <" ++ show output_type ++ ">."
-- the types match, so we assemble the output type, using gunfold
else snd (gunfold k' z' c)
k' :: forall b r . Data b => ([AData], b -> r) -> ([AData],r)

-- we start by reversing the input list
z' start = (reverse args,start)

-- then we call us recursively on the argument and feed the result
-- to the output constructor
k' ((AData a) : args, append) = (args, append (everywhereADT f a))

-- Used for folding (we don t need to retain the original constructor)
data Const a b = Const a
What a beast! But surprisingly, it works. Here are some examples. Note that I always have to specify the requested output type:
bool2Int :: Bool -> Int
bool2Int False = 0
bool2Int True = 1

*Main> everywhereADT (doNothing extADT bool2Int) True :: Int
1
*Main> everywhereADT (doNothing extADT bool2Int) True :: ()
*** Exception: Type error, trying to convert Int to ().
*Main> everywhereADT (doNothing extADT bool2Int) (True,False) :: (Int,Int)
(1,0)
*Main> everywhereADT (doNothing extADT bool2Int) ([True,False],True,()) :: ([Int],Int,())
([1,0],1,())
*Main> everywhereADT (doNothing extADT bool2Int) ([True,False],True,()) :: [()]
*** Exception: Can not convert to .
*Main> everywhereADT (doNothing extADT bool2Int) [True] :: [Bool]
[*** Exception: Type error, trying to convert Int to Bool.
I hope this code does not inflict too much pain on any Haskell-loving reader. I know I violated the language, but I didn t know how to do it better (at least not without using Template Haskell). I also know that this is not very good performance wide: Every single value in the input will be deconstructed, type-compared several times and re-assembled. If that is an issue, then this function should only be used for prototyping. Almost done To apply this to our state, we only need to glue it to our lookup functions from above:
gunref :: (Data (a Id), Data (a Ref)) =>  State Ref -> a Ref -> a Id
gunref s w = let unrefState = gunref' unrefState s
in gunref' unrefState w

gunref' :: (Data (a Id), Data (a Ref)) => State Id -> a Ref -> a Id
gunref' unrefState = everywhereADT unref'
where unref' = doNothing extADT
unrefUserRef unrefState extADT
unrefComputerRef unrefState
Now we have a generic unreferencer. I set the type a bit more specific than necessary, to make it safe to use (under the assumption that the list of lookup functions is complete and will not leave any Ref in the output).
*Main> testState 
State [User userId = 0, uses = [Ref 1] ] [Computer computerId = 1, owner = Ref 0 ]
*Main> gunref testState testState
State [User userId = 0, uses = [Id (Computer computerId = 1, owner = Id (User userId = 0,
uses = [Id (Computer computerId = 1, owner = Id (User userId = 0, uses = ..
Oh, and by the way, if you want to test this code, you ll need at least:
 -# LANGUAGE DeriveDataTypeable, FlexibleInstances, GADTs,
FlexibleContexts, StandaloneDeriving, ScopedTypeVariables #-

Romain Beauxis: Fun with liquidsoap and video..

The other day, I wanted to prepare some videos of my favorite reggae and soul tunes for uploading them to YouTube.. My goal was very simple: prepare a video with a static image and the music I wanted to upload.. After briefly digging for a simple software to do that, which I could not found, I said "hey, why not doing it with liquidsoap !". Well, that is fairly easy ! Video support in liquidsoap is very young, but the amount of things that can be done is already quite important, so I though this would be an interesting example.. All you have to do is to prepare a PPN version of the picture, using convert /path/to/image.jpeg /path/to/image.ppn and have the mp3 or ogg file in an other place... From the scripting point of view, the only difficult thing is to have liquidsoap stop at the end of the song.. As liquidsoap is mainly a streaming language, it is not designed to be able to stop easily. It should be made better soon, but for now I had to use a special trick: compose sequentially the song with a blank source. On the blank source, I add special metadata such that the output file is reopened with a new title when the song has ended. Eventually, the shutdown is triggered when adding the new metadata on the blank source. Here is the code:
# Log to stdout
set("log.file",false)
set("log.stdout",true)
# Disable real-time processing, to process with the maximun speed
set("root.sync",false)
# Enable video
set("frame.video.channels",1)

# The video source using a static image
s = video.image(width=240,height=240,x=40,"/tmp/bla.ppm")
# The audio song.
audio = single("/tmp/bla.mp3")
# The special metadata function, that
# also triggers the shutdown..
def end_meta(_) =
shutdown ()
[("title","prout")]
end
# This is the blank source
bl = map_metadata(insert_missing=true,update=false,end_meta,blank())
# And the final source: the sequence song and blank added
# with the video of the static image..
final = add([sequence([audio,bl]),s])
# Output to a theora file, reopen on new metadata..
output.file.theora(reopen_on_metadata=true,"/tmp/$(title).ogv",
final)
The result is just what I wanted (surprisingly !): The other videos can be found here.

Fabio Tranchitella: year.stop(); year = Year(2009); year.start();

A few hours and this year will be over. Instead of making a summary of 2008, I feel the need to list a few of my goals for 2009:

This list is not exhaustive, but writing down these items helps me. After a so-so 2008, I'm looking forward for a new exciting year: I feel very motivated and I'm sure that 2009 will be positive and rich of satisfactions.

Happy New Year to everybody, may the Force be with you.

Russell Coker: Links December 2008

A teacher in Arizona steals Linux CDs from a student and then accuses a Linux distributor of being a criminal [1]. Even though she had used Linux in the past she didn t believe that software was free. Of course that implies that in the past she had performed actions that she believed were criminal. Neat Little Mac Apps interviews Marshall Kirk McKusick - he describes how the BSD Daemon logo was designed and one of his most significant bugs [2]. OurDelta.org offers MySQL builds with some extra features and support [3]. I was recommended to use their builds by Arjen Lentz of Open Query [4], as one of my clients is going to use the services of Open Query it seems best to use the Our Delta builds if only to get better support from Open Query. The extra features in the Our Delta builds seem interesting, but I m not sure that my client needs any of them at this time. The Global Guerilla blog reports on a man who single-handedly invaded the most heavily guarded power station in Britain and shut it down to protest against new coal power stations [5]. The entire blog is worth reading, the author has a lot of interesting ideas. PhpMyVisites is a free web site analytics system that competes with Google Analytics [6]. I haven t implemented it yet, but it looks promising. It seems that PhpMyVisites is not being updated any more (not even security updates) and the replacement is Piwik [11]. Andrew Lahde was a fund manager who made significant amounts of money by betting on the inability of US mortgagees to repay their debts, he wrote an interesting goodbye letter (Telegraph.co.uk) [7]. He now has a Wikipedia page which gives some interesting background to his career [8]. An Employee of the Financial Times is famous for flaming Andrew [9], I have submitted a comment pointing out that being famous for flaming someone who is more successful than yourself is nothing to be proud of and suggesting that he advocate his own political views when criticising those of others - I doubt that it will get through moderation. It s a pity that Andrew doesn t have a blog, I would like to read more from him. At CCC a paper by Alexander Sotirov, Marc Stevens, Jacob Appelbaum, Arjen Lenstra, David Molnar, Dag Arne Osvik, and Benne de Weger on how to crack the PKI infrastructure used for SSL signing was presented [10]. The root cause is some CAs still using MD5 even though it was broken a long time ago. Updated to note that Piwik is the replacement for PhpMyVisites.

Neil Williams: OpenID.com https FAIL

*.myopenid.com - page security information.
Equifax Secure Global eBusiness CA-1
PKCS #1 MD5 With RSA Encryption

Same for https://www.myopenid.com/signin_password and the same for the myOpenID.com SSL certificates that you actually use with myOpenID.com - all now known to be vulnerable to spoofing. I don't think I'll be using those any more.

SSL spoofing via MD5 certification

CA's using only MD5 to sign the certificate

Oops.

It depends on how you use OpenID I suppose, but if known weak certificates are removed from browsers in order to protect users from possible SSL spoofing, current myOpenID.com certificates will simply stop working.

Not really a surprise considering how Canonical came to exist but, Launchpad OpenID has got it right. Launchpad itself doesn't use a CA that relies solely on the weak MD5. Maybe I'll start using the Launchpad OpenID instead. (Launchpad OpenID still works with the Alioth OpenID support). However, it would be nice if the Launchpad OpenID could provide SSL certificates for users to import into browsers and then I could log into various sites directly. I did like that functionality with myopenid.com but those certificates have had to be revoked now.

Gold stars for Ubuntu and the Alioth admins. A pile of stinky brown stuff for the myopenid.com team.

Update:
Maybe it's time for Alioth to support OpenID more fully? Or there again, maybe not. After a little reading up on the multiple failings of openid in general, I'm going to leave openid as it is (mostly disabled in my case) and use it only for stuff that really doesn't matter much. Using openid for important stuff just doesn't make any sense.

30 December 2008

David Moreno: Introducing Feedbag: Feed auto-discovery Ruby library/tool

Last week, I spent some time building a good (that I liked) feed auto-discovery tool to use in Ruby for other project I m building, rFeed. I liked CPAN s Feed::Find, and at some point I made a wrapper class to run a Perl script using such module, however, I wasn t happy by mixing it all. So, Feedbag was born:
>> require "rubygems"
=> true
>> require "feedbag"
=> true
>> Feedbag.find "log.damog.net"
=> ["http://feeds.feedburner.com/TeoremaDelCerdoInfinito",
 "http://log.damog.net/comments/feed/"]
>> planet_feeds = Feedbag.find("planet.debian.org")
[ ... ]
>> planet_feeds.first(3)
=> ["http://planet.debian.org/rss10.xml",
 "http://planet.debian.org/rss20.xml",
 "http://planet.debian.org/atom.xml"]
>> planet_feeds.size
=> 104
>>
It makes smart use of relative and absolute bases, hrefs, links, content types, etc. It is also a single Ruby file, so you can grab it and use it on your application. Plus, it only requires Hpricot as dependency. It can find all feeds linked on a web page, but it will return the most important at the beginning of the resulting array, so you will have the important one on the first results (see example above with Planet Debian). Synopsis, README and a brief tutorial have been placed at axiombox.com/feedbag. You can also take a look at the git repo, hosted in GitHub.

Neil Williams: Comparing Packages files

Something that's bothered me and Emdebian in general for months is that it should be simple to compare two repositories and work out which packages need to migrate, which are behind etc. using perl. apt-cross has had rough support for this for some time but it isn't general - it relies on the apt perl bindings.

Recently, the shell scripts for Emdebian Grip became too complex to maintain long term - and far too slow. The main reason was the difficulty in getting all the relevant repository data into memory within a shell scripts. So, I turned to perl and the result is the tenth package to be derived from emdebian-tools source: libdebian-packages-compare-perl - Debian::Packages::Compare - emdebian repository comparison support. So far, it is meant to run on the server because various elements of meta-data need to be read from the repository configuration and that means relying on reprepro. In time, I hope to extend it so that this meta-data can be set arbitrarily so that the module only requires a pair of Packages files - at which point it can be extended to handle a Packages.gz file too (it already looks for and handles the Sources.gz file). The module includes bespoke handling for Emdebian TDebs and the locale root layout.

It reads the Packages file, create a hash of Package and version - once per arch, one per suite, one pair per repo. The underlying libparse-debian-packages-perl is a very simple module, there probably isn't any point putting extra data into the comparison hashes, get the data necessary and load the full Packages data separately using get_single_package.

This module is currently tied to the repository layout used by reprepro in order to identify the architecture list and various other pieces of meta-data. In time, functions can be added to provide such lists.

The module expects to find all repositories beneath a single base directory:

$base/$repo_name/conf/distributions

If $base is undefined or if the repository directories cannot be found, subsequent functions each return undef.

etc.

Example code:

use Debian::Packages::Compare;

my $base = '/path/to/repositories/'
&set_base($base);

my $debu = &read_packages ('unstable', 'filter');
my $gripu = &read_packages ('unstable', 'grip');
my %debianunstable = %$debu if (defined $debu);
my %gripunstable = %$gripu if (defined $gripu);
foreach my $pkg (sort keys %debianunstable)

if ($debianunstable $pkg 'source' ne $gripunstable $pkg 'source' )

# $pkg is either not in grip repository or is a different version




The main features for subsequent versions will involve more verbose error handling.

Currently just in Emdebian SVN, I'll be releasing to Emdebian fairly soon. If there's interest, I could put it into experimental but I think it needs that meta-data handling code first.

Debian::Packages::Compare.pm.

David Welton: Rant: Ubuntu, Google, J2ME

Long day, lots of broken stuff. Rant time:
  • Ubuntu's Intrepid Ibex has way too many regressions. I've mentioned this before talking about wireless, but also on the sidelines are bluetooth and my laptop's multimedia keys. Other things are probably slipping my mind, but that's what's bugging me today. I've use Linux for a while, and am used to not always having everything working just right, but "it's never worked" is less annoying than "it used to work but this release broke it". Also, I bought this computer from Dell because it shipped with Ubuntu. I would have expected the Ubuntu guys to have a few around themselves to test on prior to release.
  • J2ME marketing: http://blogs.sun.com/hinkmond/entry/getjar_3rd_annual_mobile_awards GetJar is a nice service, and I put my own ShopList app there, but you can't seriously compare it to the Apple or Android stores without insulting my intelligence. What percentage of people with J2ME capable phones actually use GetJar? Is it on their phone when they turn it on? Can developers actually sell stuff there?
  • Google has started dumping lots of my mail in the spam folder since I switched the domain over to a new server. I can't believe they're dumping so much non spam mail, and yet can't figure out that I have never, not once, received an email written in Chinese characters that I actually wanted to read.
Add in some bugs and broken stuff of my own, and it's made for a frustrating day.

David Welton: Rant: Ubuntu, Google, J2ME

Long day, lots of broken stuff. Rant time:
  • Ubuntu's Intrepid Ibex has way too many regressions. I've mentioned this before talking about wireless, but also on the sidelines are bluetooth and my laptop's multimedia keys. Other things are probably slipping my mind, but that's what's bugging me today. I've use Linux for a while, and am used to not always having everything working just right, but "it's never worked" is less annoying than "it used to work but this release broke it". Also, I bought this computer from Dell because it shipped with Ubuntu. I would have expected the Ubuntu guys to have a few around themselves to test on prior to release.
  • J2ME marketing: http://blogs.sun.com/hinkmond/entry/getjar_3rd_annual_mobile_awards GetJar is a nice service, and I put my own ShopList app there, but you can't seriously compare it to the Apple or Android stores without insulting my intelligence. What percentage of people with J2ME capable phones actually use GetJar? Is it on their phone when they turn it on? Can developers actually sell stuff there?
  • Google has started dumping lots of my mail in the spam folder since I switched the domain over to a new server. I can't believe they're dumping so much non spam mail, and yet can't figure out that I have never, not once, received an email written in Chinese characters that I actually wanted to read.
Add in some bugs and broken stuff of my own, and it's made for a frustrating day.

Joerg Jaspert: Fuck it! What could possibly go wrong?

Time for another ftpmaster blog. It s not like we stopped doing things. :) It seems we, somehow, got a pretty active coder (yay), Michael Casadevall. In the last few days I merged multiple code changes from him. The first was a different output format for our NEW queue overview page. It is now also available in 822 Format for all those people that want to use the data in their tools. Like QA for example. Today got a second change touching this, we now not only list NEW/BYHAND in there, but also our incoming (accepted) and Proposed-Updates queue. There is also a patch waiting to fix up the pretty weird code thats around the DM stuff. Will fix some of the many bugs and weirdnesses in the current implementation. And it seems he is right now trying to get rid of apt-ftparchive, the beast that makes our dinstall run take so long. Oh we will be so happy when we got rid of it. :) Also, we do have two new people as trainees for ftpteam. If that works out, we will have yet another 2 new members in our team. Yay.

Steve Kemp: When the light is green, the trap is clean.

Recently the Debian Administration website has been besieged by spammers submitting bogus comments. I also run a couple of blogs which are constantly having random gibberish submitted to them - although it is less of a problem there because I approve them manually due to the offline nature of my blog, and the blog compiler I use. Regardless I figured it was time to do something about it, in part because Chris Searle suggested that I should be able to query arbitrary text for spammyness due to my experience with fighting email spam. (As it turns out that is not so useful experience. Comments and emails are not the same. The same header-trickery you see in bogus email, false EHLOs, etc, are not useful concepts to carry accross.) Anyway we were saying, I was going to do something? So I did. My solution? An XML-RPC server which you submit your comments to. This will return either:
OKThe comment is clean.
SPAM:[reason]The comment is spam, and the optional reason explains further
The server itself is very simple, and it comes with a collection of simple plugins which do the job of testing different things. More plugins will almost certainly be added over time - so far I've seen the lotsaurls plugin capturing most of the SPAM, although the DNSRBL lookup is also working nicely having captured live spam without my touching it. Huzzah. & etc.. If there is any interest feel free to let me know, although as the server is open, and the client side usage is documented I imagine other people could run with it easily enough too - avoiding me becoming a single point of failure. For reference my initial change to hook this up live on the site was only a few lines of code - although later I did need to make some additional changes to make it cope with failures, & etc. ObFilm: Ghostbusters

Julien Danjou: Rants about Lua

I've started using Lua some months ago, while looking for a more powerful way to configure awesome. At this time, around March 2008, Lua seemed to be the best language to integrate inside the core system of awesome. I still think that Lua was a good choice, but after 8 months, it shows some important drawbacks. I'll try to keep my explanation simple and to make you understand everything, even if you do not know Lua. I refer here to Lua 5.1 Design flaws Length operator Lua has a length operator on its objects, known as #. It can be used to get the size of various objects.
return #"lol" 3
This operator works for table, string, etc It's possible to define this operator by setting a __len meta-method on a userdata value. The problem is that you cannot redefine it on string or table objects, see:
> a =   "hello", "world"  
> return #a
2
> setmetatable(a,   __len = function () return 18 end  )
> return #a
2
Indeed, looking at the Lua core code:
     case OP_LEN:   
       const TValue *rb = RB(i); 
       switch (ttype(rb))   
         case LUA_TTABLE:   
           setnvalue(ra, cast_num(luaH_getn(hvalue(rb)))); 
           break; 
             
         case LUA_TSTRING:   
           setnvalue(ra, cast_num(tsvalue(rb)->len)); 
           break; 
             
         default:    /* try metamethod */ 
           Protect( 
             if (!call_binTM(L, rb, luaO_nilobject, ra, TM_LEN)) 
               luaG_typeerror(L, rb, "get length of"); 
           )   
             
           
       continue; 
      
You clearly see that tables and strings always use the internal length operator, never the __len meta-method. That's for me a design problem, which will cause more trouble. We'll see later. __index and __newindex metamethods Lua defines two useful metamethods, which are __index and __newindex. Both can be set on a table or any other object. __index will be called upon each read access to an undefined key on an object, and __newindex upon each write access. Example
> a =  
> setmetatable(a,   __index = myindexfunction, __newindex = mynewindexfunction  ) -- function are not defined, this is just an example
> a[1] = "hello" -- This will call __newindex metamethod
> return a[2] -- This will call __index metamethods
> return a[1] -- This will NOT call __index
The last line does not call __index meta-method because a[1] does exists. This is a problem when you want to use table as object, because sometimes you want to monitor access to the table elements. This can be easily worked around using a proxy system: you don't store things in the table you manipulate, but in another table.
> a =  
> realtable =  
> setmetatable(a,   __index = myindexfunction, __newindex = mynewindexfunction  ) -- function are not defined, this is just an example
Where meta-methods are something like:
function myindexfunction(table, key)
   return realtable[key]
end
function mynewindexfunction(table, key, value)
   realtable[key] = value
end
This way, our a table will always be empty, and realtable will have the data. At every read or write access to a, the meta-methods will be called. This is very convenient and widely used hack. But this has serious drawbacks: as we saw before, the length operator (#) cannot be redefined on a table. That means #a will always be 0, and you cannot get the table length anymore, except by defining another method or a special attribute. Also, Lua has several functions in the table library that are used to manipulate table in a easy way. The problem is that standard functions like table.insert or table.remove do raw accesses to the table. Meaning that if you do table.insert(a, 1, 1) it will insert the value 1 at the key 1 into a, without calling the __newindex meta-methods, breaking all your beautiful object-oriented model. Another solution is to use a userdata object, like done in the Lua newproxy function (which is under-documented). The problem is that it breaks all the other functions that are waiting for a table as argument, because they see a userdata, not a table. So this time table.insert is now more usable, which somehow fixes the problem, but not in the right way IMHO. However, this allows to use the __len meta-method. Development model The development model of Lua is, from my point of view, non-existent. There is no public version control system repository available, so there's no chance to really contribute to Lua. It seems only a defined set of people work on it and therefore, the development system is very closed to my eyes, in comparison of usual projects. Conclusion I still think Lua is a good choice, because it is very easy to integrate into any C program, and to expand to fulfill your needs. However, some bad design choices were made, and the poor and closed development model chosen does not allow to have a good overview of the future of Lua. This has been well stated by the authors themselves.

Junichi Uekawa: Using Multicore CPUs with pbuilder.

Using Multicore CPUs with pbuilder. I did have distcc and ccache set up on my system, but I didn't have multiple build setup on my system. Strange. dpkg-buildpakage takes '-jX' option for parallel builds, and pbuilder supports --debbuildopts for passing down build options. I usually pass on '--debbuildopts "-I -i "', so I can just change that to --debbuildopts "-I -i -j5 " to make it build in 5 parallel mode.

29 December 2008

John Goerzen: A Day with Jacob

Old train! Smoke inside. That s what Jacob said today when he saw an old steam locomotive at the old Union Station in Wichita, KS. Never mind that it s been sitting there for decades he s got that concept of smoke coming out of steam engines down, and apparently realizes that for smoke to come out of them, it must start out inside them. But I don t think I ve ever heard anyone talk about a steam engine in a more cute way. Two is a fun age. Today I needed to return a power supply to Circuit City, so Jacob and I made a day of it. He came with me to town (yes, us country folk go to town ). We returned the power supply, then ate lunch at a Mexican place. Jacob would call it a chips and salsa place. Today he drank from a straw for the first time, and loved it so much that I think he got more liquid than food in him over lunch. Oh well. After that, we went to the eye cream (ice cream) store - Cold Stone Creamery. Whenever Jacob hears talk of Wichita, he comments with something like Go ichita. Eat eye cream. Go town. Jacob ate his entire kid-sized chocolate ice cream with sprinkles. He loves sprinkles. While we were eating ice cream, I asked him, After you are done, would you like to see skyscrapers or trains? NO! Eat eye cream! I don t think he quite got the after bit, so I asked him again in the car after we were done eating. His eyes lit up. SEE KYCRAPER!! So we drove to downtown Wichita and saw the skyscrapers. (Yes, I am aware that other cities have much taller buildings. It doesn t matter when you re 2.) We got out of the car and he got to touch a few of them. In fact, the only way I got him back into the car without a meltdown was by pointing out the train crossing above the road on a bridge, and the fact that he d still be able to see it from in the car. That, of course, led to us needing to see trains. I drove him up the ramp to be trackside by the old Wichita Union Station. There weren t any trains going past right then, but he did get to see the locomotives at the Great Plains Transportation Museum through the fence. We started to head home, but he really wanted to see the train again. So we stopped at the park in Newton to see the old ATSF locomotive there. By that point, it was waaaay past naptime and we narrowly averted a fit when I informed him that no, we could not go inside the adjacent grain elevators. So all told, I think Jacob had a fun day with his dad. Now, in case there is any doubt out there about what kind of child Jacob is, I should point out the things he notices in church lately. First off, when we get into the sanctuary, we sit in the back. But first thing he said lately when we walked in was TWO THERMOSTATS! Yes, he saw two thermostats at opposite ends of the room up front, pointed right at them, and was very excited. He also notices the organ speakers ( oran peaker ), big cross , ceiling fans, and now furnace vents . Yes, furnace vents. At the Christmas Eve service, there was a candlelight part. He loved it. Daddy hold fire! A few people nearby smiled. While we sang Silent Night, he really wanted to blow out the candle. When it was time, I said OK Jacob, now you can blow it out. A very excited Fffffffffffffffff followed, and several people nearby were chuckling by then. I also let Jacob put the candle in the box on our way out. He said, I put fire in box. I did it!

Dirk Eddelbuettel: RQuantLib 0.2.10

Earlier this month and following the release of QuantLib 0.9.7, I updated RQuantLib to version 0.2.10. For once, there were no changes required to keep up with QuantLib. Rather, changes were internal as Rcpp had been spun off into its own package. As Rcpp is now in Debian, RQuantLib itself was also updated in Debian following the earlier upload to the CRAN network.

Lior Kaplan: Speed Up Multiple SSH Connections to the Same Server


Don Marti writes at linuxjournal about howto Speed Up Multiple SSH Connections to the Same Server. This seems very usefull - so enjoy. Posted in System Administration Tagged: ssh

Adrian von Bidder: Milk

A blog entry I wanted to write quite some time now... Depicted on the right is a liter carton of organic whole milk of one of the two big retailers here in Switzerland. At the bottom is the red area again, enlarged. For those who don't read french, italian or german: it's an indication of the ingredients ( Ingredients: whole milk ), and a warning for people with allergies ( Contains milk .) No doubt information like this is an essential contribution to lowering health costs by making people chose healthier foods and being aware of what they eat.

Clint Adams: Oh, the triage

Thanks to Richard Hartmann, the number of bugs on the zsh package is down to 85. That's only about 100 too many.

David Moreno: Good looking Irb

Irb is a pain to work with when you don t know it enough. Fortunately, it can be configured extensively enough to make your Ruby interactive sessions much smoother. Pocahontas asked me to post my Irb configuration:
# ~/.irbrc
IRB.conf[:AUTO_INDENT] = true
IRB.conf[:USE_READLINE] = true
IRB.conf[:SAVE_HISTORY] = 5000
IRB.conf[:HISTORY_FILE] = "# ENV['HOME'] /.irb_history"
IRB.conf[:PROMPT_MODE] = :SIMPLE
require 'irb/completion'
require 'irb/ext/save-history'
require 'pp'
# load rubygems and wirble
require 'rubygems' rescue nil
require 'wirble'
require 'utility_belt'
# load wirble
Wirble.init
Wirble.colorize
Which makes Irb to look much much better: irb But it is not only coloring the features you are getting, but also Readline support, command history saving, tab completion, libraries loaded by default, persistent history with Wirble, etc, etc. as you can see yourself on the very self-explanatory configuration file. Enjoy. Update: I have to thank my co-worker Galin for pointing me most of these neat features.

Wouter Verhelst: Why GNOME sucks

A recent post by Adrian von Bidder got me to reread the "Printing dialogs and GNOME" thread that Linus started on the GNOME mailinglists a few years ago. I don't know how or why I missed it back then, but I did see this little gem in a post by Mike Shaver:
You and I are not interesting GNOME users, in part because we already _are_ GNOME users. The interesting GNOME users are the ones that haven't tried it yet, and we need to figure out why they haven't. There are lots of things that I want in GNOME, but if they aren't going to bring a hundred thousand of my closest friends along as well, I would hope that nobody goes out of their way to do them.
That is precisely what is wrong with the GNOME people. While encouraging new users to try and use GNOME certainly is a good and worthy goal, it is this mentality of 'only the new users matter, the existing ones can go f*ck themselves' which will ultimately drive away users. Of course this post is a few years old, and I'm not involved in GNOME development in any way, so this mentality might have gone away now; but I don't hold my breath.

Next.

Previous.