Since July, I have been aware of an ugly problem with
propellor. Certain
propellor configurations could have a bug. I've tried to solve the problem at
least a half-dozen times without success; it's eaten several weekends.
Today I finally managed to fix propellor so it's impossible to write code that
has the bug, bending the Haskell type checker to my will with the power of
GADTs and type-level functions.
the bug
Code with the bug looked innocuous enough. Something like this:
foo :: Property
foo = property "foo" $
unlessM (liftIO $ doesFileExist "/etc/foo") $ do
bar <- liftIO $ readFile "/etc/foo.template"
ensureProperty $ setupFoo bar
The problem comes about because some properties in propellor have Info
associated with them. This is used by propellor to introspect over the
properties of a host, and do things like set up DNS, or decrypt
private data used by the property.
At the same time, it's useful to let a Property internally decide to
run some other Property. In the example above, that's the
ensureProperty
line, and the
setupFoo Property is run only sometimes, and is
passed data that is read from the filesystem.
This makes it very hard, indeed probably impossible for Propellor to
look inside the monad, realize that
setupFoo is being used, and add
its Info to the host.
Probably,
setupFoo doesn't have Info associated with it -- most
properties do not. But, it's hard to tell, when writing such a Property
if it's safe to use ensureProperty. And worse,
setupFoo could later
be changed to have Info.
Now, in most languages, once this problem was noticed, the solution would
probably be to make
ensureProperty notice when it's called on a Property
that has Info, and print a warning message. That's Good Enough in a sense.
But it also really stinks as a solution. It means that building propellor
isn't good enough to know you have a working system; you have to let it
run on each host, and watch out for warnings. Ugh, no!
the solution
This screams for GADTs. (Well, it did once I learned how what GADTs are
and what they can do.)
With GADTs,
Property NoInfo and
Property HasInfo can be separate data
types. Most functions will work on either type (
Property i) but
ensureProperty can be limited to only accept a
Property NoInfo.
data Property i where
IProperty :: Desc -> ... -> Info -> Property HasInfo
SProperty :: Desc -> ... -> Property NoInfo
data HasInfo
data NoInfo
ensureProperty :: Property NoInfo -> Propellor Result
Then the type checker can detect the bug, and refuse to compile it.
Yay!
Except ...
Property combinators
There are a lot of Property combinators in propellor. These combine
two or more properties in various ways. The most basic one is
requires,
which only runs the first Property after the second one has successfully
been met.
So, what's it's type when used with GADT Property?
requires :: Property i1 -> Property i2 -> Property ???
It seemed I needed some kind of type class, to vary the return type.
class Combine x y r where
requires :: x -> y -> r
Now I was able to write 4 instances of
Combines, for each combination
of 2 Properties with HasInfo or NoInfo.
It type checked. But, type inference was busted. A simple expression
like
foo requires bar
blew up:
No instance for (Requires (Property HasInfo) (Property HasInfo) r0)
arising from a use of requires'
The type variable r0' is ambiguous
Possible fix: add a type signature that fixes these type variable(s)
Note: there is a potential instance available:
instance Requires
(Property HasInfo) (Property HasInfo) (Property HasInfo)
-- Defined at Propellor/Types.hs:167:10
To avoid that, it needed ":: Property HasInfo" appended --
I didn't want the user to need to write that.
I got stuck here for an long time, well over a month.
type level programming
Finally today I realized that I could fix this with a little type-level
programming.
class Combine x y where
requires :: x -> y -> CombinedType x y
Here
CombinedType is a type-level function, that calculates the type that
should be used for a combination of types x and y. This turns out to be really
easy to do, once you get your head around type level functions.
type family CInfo x y
type instance CInfo HasInfo HasInfo = HasInfo
type instance CInfo HasInfo NoInfo = HasInfo
type instance CInfo NoInfo HasInfo = HasInfo
type instance CInfo NoInfo NoInfo = NoInfo
type family CombinedType x y
type instance CombinedType (Property x) (Property y) = Property (CInfo x y)
And, with that change, type inference worked again! \o/
(Bonus: I added some more intances of CombinedType for combining
things like RevertableProperties, so propellor's property
combinators got more powerful too.)
Then I just had to make a massive pass over all of Propellor,
fixing the types of each Property to be Property NoInfo or Property HasInfo.
I frequently picked the wrong one, but the type checker was able to detect
and tell me when I did.
A few of the type signatures got slightly complicated, to provide the type
checker with sufficient proof to do its thing...
before :: (IsProp x, Combines y x, IsProp (CombinedType y x)) => x -> y -> CombinedType y x
before x y = (y requires x) describe (propertyDesc x)
onChange
:: (Combines (Property x) (Property y))
=> Property x
=> Property y
=> CombinedType (Property x) (Property y)
onChange = -- 6 lines of code omitted
fallback :: (Combines (Property p1) (Property p2)) => Property p1 -> Property p2 -> Property (CInfo p1 p2)
fallback = -- 4 lines of code omitted
.. This mostly happened in property combinators, which is an acceptable
tradeoff, when you consider that the type checker is now being used to prove
that propellor can't have this bug.
Mostly, things went just fine. The only other annoying thing was that some
things use a
[Property], and since a haskell list can only contain a
single type, while Property Info and Property NoInfo are two different
types, that needed to be dealt with. Happily, I was able to extend
propellor's existing
(&) and
(!) operators to work in this situation,
so a list can be constructed of properties of several different types:
propertyList "foos" $ props
& foo
& foobar
! oldfoo
conclusion
The resulting 4000 lines of changes will be in the next release of
propellor. Just as soon as I test that it always generates the same Info
as before, and perhaps works when I run it. (eep)
These uses of GADTs and type families are not new; this is merely the first
time I used them. It's another Haskell leveling up for me.
Anytime you can identify a class of bugs that can impact a complicated code
base, and rework the code base to completely avoid that class of bugs, is a
time to celebrate!
About a year and a half after I started writing the openstack-debian-images package, I m very happy to announce to everyone that, thanks to Steve McIntyre s help, the official OpenStack Debian image is now generated at the same time as the official Debian CD ISO images. If you are a cloud user, if you use OpenStack on a private cloud, or if you are a public cloud operator, then you may want to download the weekly build of the OpenStack image from here:
I go to the gym every couple of days. I lift things up, then put them down, and sometimes I repeat this process another 30 times. When I'm done I write down what I've done, how many times I did the lifty-droppy thing, and so on.
I want to see pretty graphs. I want to have records of different things. I guess I just need some simple text-boxes:
I install the
I've done a bunch of hacking on the 
