Narabu is a new intraframe video codec. You may or may not want to read
part 1
first.
The GPU, despite being extremely more flexible than it was fifteen years
ago, is still a very different beast from your CPU, and not all problems
map well to it performance-wise. Thus, before designing a codec, it's
useful to know what our platform looks like.
A GPU has lots of special functionality for graphics (well, duh), but we'll
be concentrating on the
compute shader subset in this context, ie., we
won't be drawing any polygons. Roughly, a GPU (as I understand it!) is built
up about as follows:
A GPU contains 1 20 cores; NVIDIA calls them SMs (shader multiprocessors),
Intel calls them subslices. (Trivia: A typical mid-range Intel GPU contains two cores,
and thus is designated GT2.) One such core usually runs the same program,
although on different data; there are exceptions, but typically, if your
program can't fill an entire core with parallelism, you're wasting energy.
Each core, in addition to tons (thousands!) of registers, also has some
shared memory (also called local memory sometimes, although that term
is overloaded), typically 32 64 kB, which you can think of in two ways:
Either as a sort-of explicit L1 cache, or as a way to communicate
internally on a core. Shared memory is a limited, precious resource in
many algorithms.
Each core/SM/subslice contains about 8 execution units (Intel
calls them EUs, NVIDIA/AMD calls them something else) and some memory
access logic. These multiplex a bunch of threads (say, 32) and run in
a round-robin-ish fashion. This means that a GPU can handle memory stalls
much better than a typical CPU, since it has so many streams to pick from;
even though each thread runs in-order, it can just kick off an operation
and then go to the next thread while the previous one is working.
Each execution unit has a bunch of ALUs (typically 16) and executes code in a SIMD
fashion. NVIDIA calls these ALUs CUDA cores , AMD calls them stream
processors . Unlike on CPU, this SIMD has full scatter/gather support
(although sequential access, especially in certain patterns, is
much more efficient
than random access), lane enable/disable so it can work with conditional
code, etc.. The typically fastest operation is a 32-bit float muladd;
usually that's single-cycle. GPUs
love 32-bit FP code. (In fact, in some
GPU languages, you won't even have 8-, 16-bit or 64-bit types. This is
annoying, but not the end of the world.)
The vectorization is not exposed to the user in typical code (GLSL has some
vector types, but they're usually just broken up into scalars, so that's a
red herring), although in some programming languages you can get to swizzle
the SIMD stuff internally to gain advantage of that (there's also schemes for
broadcasting bits by voting etc.). However, it is crucially important to
performance; if you have divergence within a warp, this means the GPU needs
to execute both sides of the if. So less divergent code is good.
Such a SIMD group is called a warp by NVIDIA (I don't know if the others have
names for it). NVIDIA has SIMD/warp width always 32; AMD used to be 64 but
is now 16. Intel supports 4 32 (the compiler will autoselect based on a bunch of
factors), although 16 is the most common.
The upshot of all of this is that you need massive amounts of parallelism
to be able to get useful performance out of a CPU. A rule of thumb is that
if you could have launched about a thousand threads for your problem on CPU,
it's a good fit for a GPU, although this is of course just a guideline.
There's a ton of APIs available to write compute shaders. There's CUDA (NVIDIA-only, but the
dominant player), D3D compute (Windows-only, but multi-vendor),
OpenCL (multi-vendor, but highly variable implementation quality),
OpenGL compute shaders (all platforms except macOS, which has too old drivers),
Metal (Apple-only) and probably some that I forgot. I've chosen to go for
OpenGL compute shaders since I already use OpenGL shaders a lot, and this
saves on interop issues. CUDA probably is more mature, but my laptop is
Intel. :-) No matter which one you choose, the programming model looks very
roughly like this pseudocode:
for (size_t workgroup_idx = 0; workgroup_idx < NUM_WORKGROUPS; ++workgroup_idx) // in parallel over cores
char shared_mem[REQUESTED_SHARED_MEM]; // private for each workgroup
for (size_t local_idx = 0; local_idx < WORKGROUP_SIZE; ++local_idx) // in parallel on each core
main(workgroup_idx, local_idx, shared_mem);
except in reality, the indices will be split in x/y/z for your convenience
(you control all six dimensions, of course), and if you haven't asked for too
much shared memory, the driver can silently make larger workgroups if it
helps increase parallelity (this is totally transparent to you). main()
doesn't return anything, but you can do reads and writes as you wish;
GPUs have large amounts of memory these days, and staggering amounts of
memory bandwidth.
Now for the bad part: Generally, you will have no debuggers, no way of
logging and no real profilers (if you're lucky, you can get to know how long
each compute shader invocation takes, but not what takes time within the
shader itself). Especially the latter is maddening; the only real recourse
you have is some timers, and then placing timer probes or trying to comment
out sections of your code to see if something goes faster. If you don't
get the answers you're looking for, forget printf you need to set up a
separate buffer, write some numbers into it and pull that buffer down to
the GPU. Profilers are an essential part of optimization, and I had really
hoped the world would be more mature here by now. Even CUDA doesn't give
you all that much insight sometimes I wonder if all of this is because
GPU drivers and architectures are
meant to be shrouded in mystery for
competitiveness reasons, but I'm honestly not sure.
So that's it for a crash course in GPU architecture. Next time, we'll start
looking at the Narabu codec itself.
As you may be aware I have
a bit of a problem with Single Board Computers in that I have a lot of them. Keeping them organised has turned into a bit of a problem.
I designed clip cases for many of these systems giving me a higher storage density on my rack shelves and
built a power supply to reduce the cabling complexity. These helped but I still ended up with a cluttered shelf full of SBC.
I decided I would make a rack enclosure to hold the SBC, I was limited to material I could easily CNC machine which limited me to acrylic plastics or wood.
Initially I started with the idea of housing the individual boards in a
toast rack arrangement. This would mean that the enclosure would have to be at least
2U high to fit the boards all the existing cases would have to be discarded. This approach was dropped when the drawbacks of having no flexibility and only being able to fit the units that were selected at design time became apparent (connector cutouts and mounting hole placement.
Instead I changed course to try and incorporate the existing cases which already solved the differing connector and mounting placement problem and gave me a uniform size to consider. Once I had this approach the design came pretty quickly. I used a
tube girder construction 1U in height to get as much strength as possible from the 3mm acrylic plastic I would use.
The design was simply laser cut from sheet stock and fastened together with M3 nut and bolts. Once I corrected the initial design errors (I managed to get almost every important dimension wrong on the first attempt) the result was a success.
The prototype is a variety of colours because
makespace ran out of suitably sized clear acrylic stock but the colouring has no effect on the result other than aesthetical. The structure gives a great deal of rigidity and there is no sagging or warping, indeed testing on the prototype got to almost 50Kg loading without a failure (one end clamped and the other end loaded at 350mm distance)
I added some simple rotating latches at the front which keep the modules held in place and allow units to be removed quickly if necessary.
Overall this project was successful and I can now pack five SBC per U neatly. It does limit me to using systems cased in my "slimline" designs (68x30x97mm) which currently means the
Raspberry Pi B+ style and the
Orange Pi PC.
Once small drawback is access to I/O and power connectors. These need to be right angled and must be unplugged before unit removal which can be a little fiddly. Perhaps a toast rack design of cases would have given easier connector access but I am happy with this trade off of space for density.
As usual the design files are
freely available, perhaps they could be useful as a basis for other laser cut rack enclosure designs.
Debian Jessie has been released on April 25th, 2015. This has opened the
Stretch development cycle. Reactions to the idea of making Debian
build reproducibly have
been pretty enthusiastic. As the pace is now likely to be even faster,
let's see if we can keep everyone up-to-date on the developments.
Before the release of Jessie
The
story goes back a long
way
but a
formal announcement to the
project
has only been sent in February 2015.
Since then, too much work has happened to make a complete report, but
to give some highlights:
- New variations are now tested: umask, kernel version, domain name,
and timezone. We might only be missing CPU type and current date now.
- Many improvements to the test system on
jenkins.debian.net
and the pages showing the
results.
- Now not only packages from
unstable are tested but also those in testing
and experimental.
- When rescheduling packages for testing, the build products can be
kept and the IRC channel gets a notification when its over.
- binutils version
2.25-6 is now built with
the
--enable-deterministic-archives flag. Making ar, strip and
others create deterministic static
libraries.
- Number of identified issues has grown from about 80 to 123 today.
Lunar did a pretty
improvised lightning
talk
during the
Mini-DebConf in
Lyon.
This past week
It seems changes were pilling behind the curtains given the amount of activity
that happened in just one week.
Toolchain fixes
- Niels Thykier uploaded
debhelper/9.20150501
which includes fixes to
dh_makeshlibs
(#774100), dh_icons
(#774102), dh_usrlocal
(#775020). Patches written
by Lunar.
- Helmut Grohne uploaded
doxygen/1.8.9.1-3 which
will not generate timestamps in HTML by default. Kudos to
akira for bringing the issue
upstream.
- Kenneth J. Pronovici uploaded
epydoc/3.0.1+dfsg-6
adding a
--no-include-build-time option.
Patch by Jelmer
Vernooij.
- David Pr vot uploaded
php-apigen/2.8.1+dfsg-2
which now has reproducible output.
- C dric Boutillier uploaded
ruby-prawn/2.0.1+dfsg-1
which now produce a deterministic output when using gradients.
Patch by Lunar.
- Jelmer Vernooij uploaded
samba/2:4.1.17+dfsg-4 which
contains a patch by Matthieu Patou making the output of
pidl
(from libparse-pidl-perl) reproducible.
- Dmitry Shachnev uploaded
sphinx/1.3.1-1 in experimental
which should produce deterministic output. The original
patch from Chris Lamb has
inspired the upstream
fix.
- gregor herrmann uploaded
libextutils-depends-perl/0.404-1
which makes ExtUtils::Depends output deterministic. Original
patch by Reiner Herrmann.
- Niko Tyni uploaded perl/5.20.2-4
which makes the output of Pod::Man reproducible. Nice team work
visible on #780259.
We also rebased the
experimental
version
of
debhelper twice to merge
the latest set of changes.
Lunar submitted a
patch to add a
-creation-date to
genisoimage.
Reiner Herrmann opened
#783938 to
request making
-notimestamp the default behavior for
javadoc.
Juan Picca submitted a
patch to add a
--use-date flag to
texi2html.
Packages fixed
The following packages became reproducible due to changes of their
build dependencies:
apport,
batctl,
cil,
commons-math3,
devscripts,
disruptor,
ehcache,
ftphs,
gtk2hs-buildtools,
haskell-abstract-deque,
haskell-abstract-par,
haskell-acid-state,
haskell-adjunctions,
haskell-aeson,
haskell-aeson-pretty,
haskell-alut,
haskell-ansi-terminal,
haskell-async,
haskell-attoparsec,
haskell-augeas,
haskell-auto-update,
haskell-binary-conduit,
haskell-hscurses,
jsch,
ledgersmb,
libapache2-mod-auth-mellon,
libarchive-tar-wrapper-perl,
libbusiness-onlinepayment-payflowpro-perl,
libcapture-tiny-perl,
libchi-perl,
libcommons-codec-java,
libconfig-model-itself-perl,
libconfig-model-tester-perl,
libcpan-perl-releases-perl,
libcrypt-unixcrypt-perl,
libdatetime-timezone-perl,
libdbd-firebird-perl,
libdbix-class-resultset-recursiveupdate-perl,
libdbix-profile-perl,
libdevel-cover-perl,
libdevel-ptkdb-perl,
libfile-tail-perl,
libfinance-quote-perl,
libformat-human-bytes-perl,
libgtk2-perl,
libhibernate-validator-java,
libimage-exiftool-perl,
libjson-perl,
liblinux-prctl-perl,
liblog-any-perl,
libmail-imapclient-perl,
libmocked-perl,
libmodule-build-xsutil-perl,
libmodule-extractuse-perl,
libmodule-signature-perl,
libmoosex-simpleconfig-perl,
libmoox-handlesvia-perl,
libnet-frame-layer-ipv6-perl,
libnet-openssh-perl,
libnumber-format-perl,
libobject-id-perl,
libpackage-pkg-perl,
libpdf-fdf-simple-perl,
libpod-webserver-perl,
libpoe-component-pubsub-perl,
libregexp-grammars-perl,
libreply-perl,
libscalar-defer-perl,
libsereal-encoder-perl,
libspreadsheet-read-perl,
libspring-java,
libsql-abstract-more-perl,
libsvn-class-perl,
libtemplate-plugin-gravatar-perl,
libterm-progressbar-perl,
libterm-shellui-perl,
libtest-dir-perl,
libtest-log4perl-perl,
libtext-context-eitherside-perl,
libtime-warp-perl,
libtree-simple-perl,
libwww-shorten-simple-perl,
libwx-perl-processstream-perl,
libxml-filter-xslt-perl,
libxml-writer-string-perl,
libyaml-tiny-perl,
mupen64plus-core,
nmap,
openssl,
pkg-perl-tools,
quodlibet,
r-cran-rjags,
r-cran-rjson,
r-cran-sn,
r-cran-statmod,
ruby-nokogiri,
sezpoz,
skksearch,
slurm-llnl,
stellarium.
The following packages became reproducible after getting fixed:
- berkeley-abc/1.01+20141105hg5b5af75+dfsg-3 uploaded by Ruben Undheim, original patch by Johann Klammer.
- binutils/2.25-7 by Matthias Klose.
- bitmap-mule/8.5+0.20030825.0433-16 uploaded by Tatsuya Kinoshita, original patch by Chris Lamb.
- blobby/1.0-2 by Felix Geyer.
- codelite/7.0+dfsg-2 by James Cowgill.
- conkeror/1.0~~pre-1+git150409-1 by Axel Beckert.
- convmv/1.15-1 by Christian Perrier.
- cpl/6.6~b-1 by Ole Streicher.
- deheader/1.1-2 by Reiner Herrmann.
- dict-foldoc/20150318-1 uploaded by Iustin Pop, original patch by Chris Lamb.
- ding/1.8-1 by Roland Rosenfeld.
- doc-rfc/20150425-1 by Iustin Pop.
- doxygen/1.8.9.1-3 by Helmut Grohne.
- eureka/1.07-1 by Fabian Greffrath.
- eximdoc4/4.85-2 uploaded by Andreas Metzler, original patch by Chris Lamb.
- flashproxy/1.7-3 by Ximin Luo.
- heimdal/1.6~rc2+dfsg-10 by Jelmer Vernooij.
- init-system-helpers/1.23 by Martin Pitt original patch by Lunar.
- ldb/2:1.1.20-2 by Jelmer Vernooij.
- libalien-wxwidgets-perl/0.67+dfsg-1 uploaded by gregor herrmann, original patch by Chris Lamb.
- libcairo-perl/1.105-1 by intrigeri.
- libclass-methodmaker-perl/2.24-1 uploaded by gregor herrmann, original patch by Chris Lamb.
- libcommon-sense-perl/3.73-3 uploaded by gregor herrmann, original patch by Chris Lamb.
- libelixirfm-perl/1.1.976-4 uploaded by gregor herrmann, original patch by Chris Lamb.
- libgnome2-perl/1.045-3 by intrigeri.
- libjs-jcrop/0.9.12+dfsg-2 uploaded by David Pr vot, original patch by Chris Lamb.
- liblas/1.8.0-2 by Bas Couwenberg.
- libopengl-perl/0.6704+dfsg-1 uploaded by gregor herrmann, original patch by Chris Lamb.
- libparse-recdescent-perl/1.967009+dfsg-2 uploaded by gregor herrmann, original patch by Reiner Herrmann.
- librasterlite/1.1g-5 by Bas Couwenberg.
- libterm-size-perl-perl/0.029-2 uploaded by gregor herrmann, original patch by Chris Lamb.
- libvdpau/1.1-1 by Andreas Beckmann.
- libxml-sax-expatxs-perl/1.33-2 uploaded by gregor herrmann, original patch by Chris Lamb.
- lynx-cur/2.8.9dev4-2 by Axel Beckert.
- mapcache/1.2.1-3 by Bas Couwenberg.
- mdbtools/0.7.1-4 by Jean-Michel Nirgal Vourg re.
- mopidy uploaded by Stein Magnus Jodal, fixed by upstream.
- objenesis/2.1-1 by Markus Koschany.
- opendkim/2.10.1-2 uploaded by Scott Kitterman, original patch by Reiner Herrmann.
- pktools/2.6.3-1 by Bas Couwenberg.
- posh/0.12.4 uploaded by Clint Adams, original patch by Chris Lamb.
- prboom-plus/2:2.5.1.4~svn4425+dfsg1-1 by Fabian Greffrath.
- qlandkartegt/1.8.1+ds-1 by Bas Couwenberg.
- readosm/1.0.0d-1~exp2 by Bas Couwenberg.
- robocode/1.9.2.4-1 by Markus Koschany.
- ruby-prawn/2.0.1+dfsg-1 uploaded by C dric Boutillier, original patch by Lunar.
- sane-backends/1.0.25+git20150425-1 by J rg Frings-F rst.
- scoop/0.7.1-3 by Daniel Stender.
- springlobby/0.218-1 by Markus Koschany.
- subvertpy/0.9.2-1 by Jelmer Vernooij.
- t-prot/3.4-2 by Axel Beckert.
- talloc/2.1.2-3 by Jelmer Vernooij.
- tdb/1.3.4-2 by Jelmer Vernooij, patch now applied by upstream.
- tk-html3/3.0~fossil20110109-5 by Ole Streicher.
- tox/1.9.2-2 uploaded by Barry Warsaw original patch by Reiner Herrmann.
- trove3/3.0.3-2 by Erich Schubert.
- txt2man/1.5.6-2 uploaded by Joao Eriberto Mota Filho, initial patch by Jonathan Wiltshire.
- units/2.11-2 by Stephen Kitt.
- win32-loader/0.7.10 upload by Didier Raboud, original patch by Lunar.
- zec/0.12-3 uploaded by Clint Adams, original patch by Chris Lamb.
- zomg/0.8-1 uploaded by Clint Adams, original patch by Chris Lamb.
Some uploads fixed some reproducibility issues but not all of them:
Patches submitted which did not make their way to the archive yet:
Improvements to reproducible.debian.net
Mattia Rizzolo has been working on compressing logs using gzip to save
disk space. The web server would uncompress them on-the-fly for clients
which does not accept gzip content.
Mattia Rizzolo worked on a new page
listing various
breakage: missing
or bad
debbindiff output, missing build logs, unavailable build
dependencies.
Holger Levsen added a new execution environment to run
debbindiff using
dependencies from
testing. This is required for packages built with
GHC as the compiler only understands interfaces built by the same
version.
debbindiff development
Version 17 has been uploaded to unstable. It now supports comparing
ISO9660 images, dictzip files and should compare identical files much
faster.
Documentation update
Various small updates and fixes to the pages about
PDF produced by LaTeX,
DVI produced by
LaTeX,
static
libraries,
Javadoc,
PE
binaries,
and
Epydoc.
Package reviews
Known issues have been tagged when known to be deterministic as some
might unfortunately not show up on every single build.
For example, two new issues have been identified by building with one
timezone in April and one in May.
RD
and
help2man add current month and
year to the documentation they are producing.
1162 packages have been removed and 774 have been added in
the past week. Most of them are the work of proper automated
investigation done by Chris West.
Summer of code
Finally, we learned that both akira and Dhole were
accepted for
this Google Summer of
Code.
Let's welcome them!
They have until May 25th before coding officialy begins. Now is the good time
to help them feel more comfortable by sharing all these little bits of
knowledge on how Debian works.
Patti Smith is right, more than any other US city I have visited, Portland feels different. Although living in Cambridge, which sometimes feels like where books were invented, might give me a warped sense of a place.
I have visited Portland a few times previously and I feel comfortable every time I arrive at PDX. Sure the place still suffers from the american obsession with the car but similar to New York you can rely on public transport to get about.
On this occasion my visit was for the Debian Conference which i was excited to attend having missed the previous one in Switzerland. This time the conference has changed its format to being 10 days long and mixing the developer time in with the more formal sessions.
The opening session gave Steve McIntyre and myself the opportunity to
present a small token of our appreciation to Russ. The keynote speakers that afternoon were all very interesting both Stefano Zacchiroli and Gabriella Coleman giving food for thought on two very different subjects.
Several conferences in the past have experienced issues with sponsored accommodation and food, I am very pleased to report that both were very good this time. The room I was in had a small kitchen area, en-suite bathroom, desks and most importantly comfortable beds.
The food provision was in the form of a buffet in the Ondine facility. The menu was not greatly varied but catered to all requirements including vegetarian and gluten free diets.
Some of us went on a visit to the
Evergreen air and space museum to look at some rare aircraft and rockets. I can thoroughly recommend a visit if you are in the area.
These are just the highlights of the week though, the time in the hack-labs was productive with several practical achievements Including:
- Uploading new packages reducing the bug count
- Sorting out getting an updated key into the Debian keyring.
Overall I had a thoroughly enjoyable time and got a lot out of the conference this year. The new format suited me surprisingly well and as usual the social side was as valuable as the practical.
I hope the organisers have recovered enough to appreciate just how good a job they did and not get hung up on the small number of things that went wrong when the majority of things went perfectly to plan.
2013 is nearly all finished up and so I thought I'd spend a little time
writing up what was noteable in the last twelve months. When I did so I
found an unfinished draft from the year before. It would be a shame for
it to go to waste, so here it is.
2012 was an interesting year in many respects with personal
highs
and
lows. Every year I see a lots of "round-up"-style blog posts
on the web, titled things like "2012 in music", which attempt to summarize the
highlights of the year in that particular context.
Here's JWZ's
effort, for example.
Often they are prefixed with statements like "2012 was a strong year for music"
or whatever. For me, 2012 was not a particularly great year. I discovered quite
a lot of stuff that I love that was new to me, but not new in any other sense.
In Music, there were a bunch of come-back albums that made the headlines. I
picked up both of Orbital's
Wonky and Brian Eno's
Lux (debatably a
comeback: his first ambient record since 1983, his first solo effort since
2005, but his fourth collaborative effort on Warp in the naughties). I've
enjoyed them both, but I've already forgotten
Wonky and I still haven't fully
embraced
Lux (and
On Land has not been knocked from the top spot when I
want to listen to ambience.) There was also Throbbing Gristle's (or
X-TG)
final effort, a semi/post-TG, partly posthumous double-album swan song effort
which, even more than
Lux, I still haven't fully digested. In all honesty I
think it was eclipsed by the surprise one-off release of a live recording of a
TG side project featuring Nik Void of
Factory Floor: Carter Tutti Void's
Transverse, which is excellent. Ostensibly a four-track release, there's a
studio excerpt
V4 studio (Slap 1) which is available from (at least)
Amazon.
There's also a much more obscure fifth "unreleased" track
cruX which I managed
to "buy" from one of the web shops for zero cost.
The other big musical surprise for me last year was Beth Jeans Houghton and the
Hooves of Destiny:
Yours Truly, Cellophane Nose. I knew nothing of BJH,
although it turns out I've heard some of her singles repeatedly on Radio 6, but
her band's guitarist Ed Blazey and his partner lived in the flat below me
briefly. In that time I managed to get to the pub with him just once, but he
kindly gave me a copy of their album on 12" afterwards. It reminds me a bit of
Goldfrapp circa "Seventh Tree": I really like it and I'm looking forward to
whatever they do next.
Reznor's
How To Destroy Angels squeezed out
An Omen EP which failed to set
my world on fire as a coherent collection, despite a few strong songs
individually.
In movies, sadly once again I'd say most of the things I recall seeing would be
"also rans".
Prometheus was a disappointment, although I will probably rewatch
it in 2D at least once. The final Batman was fun although not groundbreaking to
me and it didn't surpass Ledger's efforts in
The Dark Knight. Inception remains
my favourite Nolan by a long shot.
Looper is perhaps the stand-out, not least
because it came from nowhere and I managed to avoid any hype.
In games, I moaned about having
moaning about too many
games, most of which are much older than 2012. I started
Borderlands 2 after enjoying
Borderlands (disqualified on age grounds) but
to this day haven't persued it much further. I mostly played the two similar
meta-games: The Playstation Plus
download free games in a fixed time period
and the more sporadic but bountiful
humble bundle whack-a-mole. More on
these another time.
In reading, as is typical I mostly read stuff that was not written in 2012.
Of that which was, Charles Stross's
The Apocalypse Codex was an
improvement over
The Fuller Memorandum which I did not enjoy much, but in
general I'm finding I much prefer Stross's older work to his newer; David
Byrne's
How Music Works was my first (and currently last) Google Books ebook
purchase, and I read it entirely on a Nexus 7. I thoroughly enjoyed the book
but the experience has not made a convert of me away from paper. He leans
heavily on his own experiences which is inevitable but fortunately they are
wide and numerous. Iain Banks'
Stonemouth was an enjoyable romp around a
fictional Scottish town (one which, I am reliably informed, is incredibly
realistical rendered). One of his "mainstream" novels, It avoided a particular
plot pattern that I've grown to dread with Banks, much to my suprise (and
pleasure). Finally, the stand-out pleasant surprise novel of the year was
Pratchett and Baxter's
The Long Earth. With a plot device not unlike Banks'
Transition or Stross's
Family Trade series, the pair managed to write a
journey-book capturing the sense-of-wonder that these multiverse plots are
good for. (Or perhaps I have a weakness for them). It's hard to find the lines
between Baxter and Pratchett's writing, but the debatably-reincarnated Tibetan
Monk-cum-Artificial Intelligence 'Lobsang' must surely be Pratchett's.
Pratchett managed to squeeze out another non-Discworld novel (
Dodger) as well
as a long-overdue short story collection, although I haven't read either of
them yet.
On to 2013's write-up...
Whups, I seem to have built a configuration management system this evening!
Propellor has similar goals to chef or puppet or ansible, but with an approach
much more like
slaughter.
Except it's configured by writing Haskell code.
The name is because propellor ensures that a system is configured with the
desired PROPerties, and also because it kind of pulls system configuration
along after it. And you may not want to stand too close.
Disclaimer: I'm not really a sysadmin, except for on the scale of "diffuse
administration of every Debian machine on planet earth or nearby", and so I
don't really understand configuration management. (Well, I did write
debconf, which claims to be the "Debian Configuration Management system"..
But I didn't understand configuration management back then either.)
So, propellor makes some perhaps wacky choices. The least of these
is that it's built from
a git repository
that any (theoretical) other users will fork and modify; a cron job can
re-make it from time to time and pull down configuration changes, or
something can be run to push changes.
A really simple configuration for a Tor bridge server
using propellor looks something like this:
main = ensureProperties
[ Apt.stdSourcesList Apt.Stable onChange Apt.upgrade
, Apt.removed ["exim4"] onChange Apt.autoRemove
, Hostname.set "bridget"
, Ssh.uniqueHostKeys
, Tor.isBridge
]
Since it's just haskell code, it's "easy" to refactor out common
configurations for classes of servers, etc. Or perhaps integrate
reclass? I don't know. I'm happy
with just pure functions and type-safe refactorings of my configs, I
think.
Properties are also written in Haskell of course. This one ensures that
all the packages in a list are installed.
installed :: [Package] -> Property
installed ps = check (isInstallable ps) go
where
go = runApt $ [Param "-y", Param "install"] ++ map Param ps
Here's one that ensures the hostname is set to the desired value,
which shows how to specify content for a file, and also how to run
another action if a change needed to be made to satisfy a property.
set :: HostName -> Property
set hostname = "/etc/hostname" File.hasContent [hostname]
onChange cmdProperty "hostname" [Param hostname]
Here's part of a custom one that I use to check out a user's
home directory from git. Shows how to make a property require
that some other property is satisfied first, and how to test
if a property has already been satisfied.
installedFor :: UserName -> Property
installedFor user = check (not <$> hasGitDir user) $
Property ("githome " ++ user) (go =<< homedir user)
requires Apt.installed ["git", "myrepos"]
where
go ... -- 12 lines elided
I'm about 37% happy with the overall approach to listing properties
and combining properties into larger properties etc. I think that some
unifying insight is missing -- perhaps there should be a Property monad?
But as long as it yields a list of properties, any smarter thing should
be able to be built on top of this.
Propellor is 564 lines of code, including 25 or so built-in properties like
the examples above. It took around 4 hours to build.
I'm pretty sure it was easier to write it than it would have been to look
into ansible and salt and slaughter (and also liw's human-readable configuration
language whose name I've forgotten) in enough detail to pick one, and learn
how its configuration worked, and warp it into something close to how I
wanted this to work.
I think that's interesting.. It's partly about NIH and
I-want-everything-in-Haskell, but it's also about a complicated system
that is a lot of things to a lot of people --
of the kind I see when I look at ansible -- vs the tools and experience
to build just the thing you want without the cruft. Nice to have the latter!
Review:
City of Diamond, by Jane Emerson
| Publisher: |
DAW |
| Copyright: |
March 1996 |
| ISBN: |
0-88677-704-6 |
| Format: |
Mass market |
| Pages: |
624 |
The setting is deep into the star-faring future of humanity. Humans have
settled many other worlds, thanks largely to technology from a race named
the Curosa. There is a thriving network of settlement based around a sort
of warp conduit network. But, standing apart from most human settlements,
are three huge ships: the City of Opal, the City of
Pearl, and the City of Diamond. Their inhabitants are devout
members of the Redemptionist faith, a merging of Christian (primarily
Catholic) belief and Curosa religion. Their ships are gifts from the
now-departed Curosa, featuring drives unknown to the rest of humanity that
allows them to ignore the normal transport network. They are meant to
witness and spread their religion, but now, six centuries later, they
primarily trade.
Much of this is not apparent at the start.
City of Diamond opens
with Adrian Mercati, chosen heir of the current dying Protector, dodging
people in what seems to be an oddly-organized city. Impudent,
occasionally flippant, utterly charming, and in love with danger, he
inherits the position of Protector shortly after we meet him. Emerson
slowly reveals more of the nature of the ship while Adrian is shocking
(and charming) the nobility and the church, and consorting with demons.
Or, at least, one demon: Tal, an entirely human-appearing creature except
for his eyes, who the Redemptionists claim is born without a soul.
Slowly, it becomes clear that he's actually the offspring of an alien and
a human, a mating of species whose children cannot socialize as human and
behave in ways that seem like human sociopaths. Tal is no exception, but
his interests align with Adrian's for the time.
To those two viewpoint characters, Emerson adds four more: a low-rank
administrator named Spider who works for Tal after Tal saved him from
death by recycling, Adrian's politically-arranged bride-to-be from the
rival (and even more religious) ship
City of Opal, a guard from a
French-descended slum on
City of Opal, and a dangerously honorable
and literal mercenary from an off-shoot culture of humanity that's now
only a legend. This is a book that tends to sprawl. There's quite a lot
going on, starting with the negotiations between
City of Diamond
and
City of Opal to put to rest the remnants of a civil war and
leading to the search for a legendary Curosa artifact on the planet
they're both visiting. All of the characters have their own agendas,
their own history, and their own goals, and are pursuing them in parallel
and occasionally in opposition. Emerson takes the time to let all of this
build.
Despite that, there is very little in this book that drags or bores, and
that's due primarily to the strength of the characters. Every one of the
viewpoint characters is unique, bringing a different perspective to the
story, and every one of them changes and grow over the course of the
story. They also play off of each other in delightful ways. This is a
book that makes full use of the fact that three characters make seven
character interactions. For example, Tal by himself is logical but just
slightly askew, Tal seen by Spider is uncomfortable and worrisome, Tal and
Adrian is a dance of mutual amusement and respect, and Tal and Keylinn
finding points of commonality is just delightful. I did wish that there
was a bit more Keylinn and Tal and a bit less of Spider, but once Emerson
built her world, introduced her characters, and got the story moving,
there aren't many weak pieces.
Part of the fun is figuring out the world background. There are almost no
infodumps here; Emerson just starts telling stories of characters in the
surroundings in which they find themselves, and the reader is left to
piece things together from background description, the occasional
explanation from one character to another, and parallels with
long-standing SF tropes. It's not a puzzle book, but it is a book of
where the reader is dropped as an alien into a fully-developed culture.
In both that way and in the use of nobility, religion, and formal
hierarchy against a starfaring backdrop,
City of Diamond reminds me
of the sort of classic space opera that led up to
Star Wars but
isn't seen as much these days.
It's also
funny. I was unsurprised to discover that Emerson is a
pseudonym for Doris Egan, who (in addition to writing some other SF) is an
occasional writer and producer for
House. The humor isn't entirely
obvious at the start, although Adrian frequently provokes a grin from the
start of the book, but Tal's perceptive but obliviously blunt comments
become more and more fun as the book progresses, and both Iolanthe
(Adrian's bride) and Keylinn (the mercenary, sort of) are a delight. I
want to see more of both of them, but sadly the book ends just as they
(particularly Iolanthe) are hitting their strides.
And that's the biggest problem with
City of Diamond: it's a failed
(or at least postponed) trilogy. As with Walter Jon Williams's
Metropolitan and
City on Fire, there were supposed to be
more books, but apparently either the publisher lost interest or other
projects took precedence. The story does come to something of a
conclusion, but most of the major story arcs aren't resolved. Emerson is
still bringing more
guns onto the set
late in the book. None of the villains truly get their just desserts, nor
do most of the character arcs fully resolve, although friendships and more
have formed satisfyingly by the end of the book. There is obviously far
more to both these characters and this universe, including an entire ship
that was never visited and an alien whose presence should prove startling.
This is a great book: warm, funny, full of interesting characters, and
with a nicely complex background universe. If it were complete in itself,
or if the sequels had materialized, I would recommend it unreservedly. As
is, you will have to weigh your dislike of unfinished stories against its
stand-alone merits. But I enjoyed it a great deal.
Rating: 7 out of 10
Some years ago I came across a really peculiar newsgroup post. It was not about technicalities of any sort. It was about history. A beautifully written history of computers. From the Turing machine to the Free Software world, the original author managed to capture all the important events of the computer revolution with a great deal of humor. Re-posting it here, it is just brilliant.
The Gospel of Tux (v1.0)
In the beginning Turing created the Machine.
And the Machine was crufty and bodacious, existing in theory only. And von Neumann looked upon the Machine, and saw that it was crufty. He divided the Machine into two Abstractions, the Data and the Code, and yet the two were one Architecture. This is a great Mystery, and the beginning of wisdom.
And von Neumann spoke unto the Architecture, and blessed it, saying, Go forth and replicate, freely exchanging data and code, and bring forth all manner of devices unto the earth. And it was so, and it was cool. The Architecture prospered and was implemented in hardware and software. And it brought forth many Systems unto the earth.
The first Systems were mighty giants; many great works of renown did they accomplish. Among them were Colossus, the codebreaker; ENIAC, the targeter; EDSAC and MULTIVAC and all manner of froody creatures ending in AC, the experimenters; and SAGE, the defender of the sky and father of all networks. These were the mighty giants of old, the first children of Turing, and their works are written in the Books of the Ancients. This was the First Age, the age of Lore.
Now the sons of Marketing looked upon the children of Turing, and saw that they were swift of mind and terse of name and had many great and baleful attributes. And they said unto themselves, Let us go now and make us Corporations, to bind the Systems to our own use that they may bring us great fortune. With sweet words did they lure their customers, and with many chains did they bind the Systems, to fashion them after their own image. And the sons of Marketing fashioned themselves Suits to wear, the better to lure their customers, and wrote grave and perilous Licenses, the better to bind the Systems. And the sons of Marketing thus became known as Suits, despising and being despised by the true Engineers, the children of von Neumann.
And the Systems and their Corporations replicated and grew numerous upon the earth. In those days there were IBM and Digital, Burroughs and Honeywell, Unisys and Rand, and many others. And they each kept to their own System, hardware and software, and did not interchange, for their Licences forbade it. This was the Second Age, the age of Mainframes.
Now it came to pass that the spirits of Turing and von Neumann looked upon the earth and were displeased. The Systems and their Corporations had grown large and bulky, and Suits ruled over true Engineers. And the Customers groaned and cried loudly unto heaven, saying, Oh that there would be created a System mighty in power, yet small in size, able to reach into the very home! And the Engineers groaned and cried likewise, saying, Oh, that a deliverer would arise to grant us freedom from these oppressing Suits and their grave and perilous Licences, and send us a System of our own, that we may hack therein! And the spirits of Turing and von Neumann heard the cries and were moved, and said unto each other, Let us go down and fabricate a Breakthrough, that these cries may be stilled.
And that day the spirits of Turing and von Neumann spake unto Moore of Intel, granting him insight and wisdom to understand the future. And Moore was with chip, and he brought forth the chip and named it 4004. And Moore did bless the Chip, saying, Thou art a Breakthrough; with my own Corporation have I fabricated thee. Thou thou art yet as small as a dust mote, yet shall thou grow and replicate unto the size of a mountain, and conquer all before thee. This blessing I give unto thee: every eighteen months shall thou double in capacity, until the end of the age. This is Moore s Law, which endures unto this day.
And the birth of 4004 was the beginning of the Third Age, the age of Microchips. And as the Mainframes and their Systems and Corporations had flourished, so did the Microchips and their Systems and Corporations. And their lineage was on this wise:
Moore begat Intel. Intel begat Mostech, Zilog and Atari. Mostech begat 6502, and Zilog begat Z80. Intel also begat 8800, who begat Altair; and 8086, mother of all PCs. 6502 begat Commodore, who begat PET and 64; and Apple, who begat 2. (Apple is the great Mystery, the Fruit that was devoured, yet bloomed again.) Atari begat 800 and 1200, masters of the game, who were destroyed by Sega and Nintendo. Xerox begat PARC. Commodore and PARC begat Amiga, creator of fine arts; Apple and PARC begat Lisa, who begat Macintosh, who begat iMac. Atari and PARC begat ST, the music maker, who died and was no more. Z80 begat Sinclair the dwarf, TRS-80 and CP/M, who begat many machines, but soon passed from this world. Altair, Apple and Commodore together begat Microsoft, the Great Darkness which is called Abomination, Destroyer of the Earth, the Gates of Hell.
Now it came to pass in the Age of Microchips that IBM, the greatest of the Mainframe Corporations, looked upon the young Microchip Systems and was greatly vexed. And in their vexation and wrath they smote the earth and created the IBM PC. The PC was without sound and colour, crufty and bodacious in great measure, and its likeness was a tramp, yet the Customers were greatly moved and did purchase the PC in great numbers. And IBM sought about for an Operating System Provider, for in their haste they had not created one, nor had they forged a suitably grave and perilous License, saying, First we will build the market, then we will create a new System, one in our own image, and bound by our Licence. But they reasoned thus out of pride and not wisdom, not forseeing the wrath which was to come.
And IBM came unto Microsoft, who licensed unto them QDOS, the child of CP/M and 8086. (8086 was the daughter of Intel, the child of Moore). And QDOS grew, and was named MS-DOS. And MS-DOS and the PC together waxed mighty, and conquered all markets, replicating and taking possession thereof, in accordance with Moore s Law. And Intel grew terrible and devoured all her children, such that no chip could stand before her. And Microsoft grew proud and devoured IBM, and this was a great marvel in the land. All these things are written in the Books of the Deeds of Microsoft.
In the fullness of time MS-DOS begat Windows. And this is the lineage of Windows: CP/M begat QDOS. QDOS begat DOS 1.0. DOS 1.0 begat DOS 2.0 by way of Unix. DOS 2.0 begat Windows 3.11 by way of PARC and Macintosh. IBM and Microsoft begat OS/2, who begat Windows NT and Warp, the lost OS of lore. Windows 3.11 begat Windows 95 after triumphing over Macintosh in a mighty Battle of Licences. Windows NT begat NT 4.0 by way of Windows 95. NT 4.0 begat NT 5.0, the OS also called Windows 2000, The Millenium Bug, Doomsday, Armageddon, The End Of All Things.
Now it came to pass that Microsoft had waxed great and mighty among the Microchip Corporations; mighter than any of the Mainframe Corporations before it had it waxed. And Gates heart was hardened, and he swore unto his Customers and their Engineers the words of this curse:
Children of von Neumann, hear me. IBM and the Mainframe Corporations bound thy forefathers with grave and perilous Licences, such that ye cried unto the spirits of Turing and von Neumann for deliverance. Now I say unto ye: I am greater than any Corporation before me. Will I loosen your Licences? Nay, I will bind thee with Licences twice as grave and ten times more perilous than my forefathers. I will engrave my Licence on thy heart and write my Serial Number upon thy frontal lobes. I will bind thee to the Windows Platform with cunning artifices and with devious schemes. I will bind thee to the Intel Chipset with crufty code and with gnarly APIs. I will capture and enslave thee as no generation has been enslaved before. And wherefore will ye cry then unto the spirits of Turing, and von Neumann, and Moore? They cannot hear ye. I am become a greater Power than they. Ye shall cry only unto me, and shall live by my mercy and my wrath. I am the Gates of Hell; I hold the portal to MSNBC and the keys to the Blue Screen of Death. Be ye afraid; be ye greatly afraid; serve only me, and live.
And the people were cowed in terror and gave homage to Microsoft, and endured the many grave and perilous trials which the Windows platform and its greatly bodacious Licence forced upon them. And once again did they cry to Turing and von Neumann and Moore for a deliverer, but none was found equal to the task until the birth of Linux.
These are the generations of Linux:
SAGE begat ARPA, which begat TCP/IP, and Aloha, which begat Ethernet. Bell begat Multics, which begat C, which begat Unix. Unix and TCP/IP begat Internet, which begat the World Wide Web. Unix begat RMS, father of the great GNU, which begat the Libraries and Emacs, chief of the Utilities. In the days of the Web, Internet and Ethernet begat the Intranet LAN, which rose to renown among all Corporations and prepared the way for the Penguin.
And Linus and the Web begat the Kernel through Unix. The Kernel, the Libraries and the Utilities together are the Distribution, the one Penguin in many forms, forever and ever praised.
Now in those days there was in the land of Helsinki a young scholar named Linus the Torvald. Linus was a devout man, a disciple of RMS and mighty in the spirit of Turing, von Neumann and Moore. One day as he was meditating on the Architecture, Linus fell into a trance and was granted a vision. And in the vision he saw a great Penguin, serene and well-favoured, sitting upon an ice floe eating fish. And at the sight of the Penguin Linus was deeply afraid, and he cried unto the spirits of Turing, von Neumann and Moore for an interpretation of the dream.
And in the dream the spirits of Turing, von Neumann and Moore answered and spoke unto him, saying, Fear not, Linus, most beloved hacker. You are exceedingly cool and froody. The great Penguin which you see is an Operating System which you shall create and deploy unto the earth. The ice-floe is the earth and all the systems thereof, upon which the Penguin shall rest and rejoice at the completion of its task. And the fish on which the Penguin feeds are the crufty Licensed codebases which swim beneath all the earth s systems. The Penguin shall hunt and devour all that is crufty, gnarly and bodacious; all code which wriggles like spaghetti, or is infested with blighting creatures, or is bound by grave and perilous Licences shall it capture. And in capturing shall it replicate, and in replicating shall it document, and in documentation shall it bring freedom, serenity and most cool froodiness to the earth and all who code therein.
Linus rose from meditation and created a tiny Operating System Kernel as the dream had foreshewn him; in the manner of RMS, he released the Kernel unto the World Wide Web for all to take and behold. And in the fulness of Internet Time the Kernel grew and replicated, becoming most cool and exceedingly froody, until at last it was recognised as indeed a great and mighty Penguin, whose name was Tux. And the followers of Linus took refuge in the Kernel, the Libraries and the Utilities; they installed Distribution after Distribution, and made sacrifice unto the GNU and the Penguin, and gave thanks to the spirits of Turing, von Neumann and Moore, for their deliverance from the hand of Microsoft. And this was the beginning of the Fourth Age, the age of Open Source.
Now there is much more to be said about the exceeding strange and wonderful events of those days; how some Suits of Microsoft plotted war upon the Penguin, but were discovered on a Halloween Eve; how Gates fell among lawyers and was betrayed and crucified by his former friends, the apostles of Media; how the mercenary Knights of the Red Hat brought the gospel of the Penguin into the halls of the Corporations; and even of the dispute between the brethren of Gnome and KDE over a trollish Licence. But all these things are recorded elsewhere, in the Books of the Deeds of the Penguin and the Chronicles of the Fourth Age, and I suppose if they were all narrated they would fill a stack of DVDs as deep and perilous as a Usenet Newsgroup.
Now may you code in the power of the Source; may the Kernel, the Libraries and the Utilities be with you, throughout all Distributions, until the end of the Epoch. Amen.
A full month or more has past since the last upload of TeX Live, so it was high time to prepare a new package. Nothing spectacular here I have to say, two small bugs fixed and the usual long list of updates and new packages.
From the new packages I found 
Over a hundred years ago, Russian leaders proposed a treaty 
The 
One thing that s always bugged me is this schism between graphic designers and hackers.
Seeing as how most graphic designers learn how to code even if it s just JavaScript / CoffeeScript, HTML5, CSS3 / LessCSS or ActionScript / Flash.
Almost no hackers learn how to design.
And that s a problem.
As a result, I ve started a real effort to learn the basics / fundamentals of design, in hopes that I can better communicate with my peers, create things that don t look like a 1992 timewarp and be able to finish a project and feel proud of what it is, as it is, even without a designer s help.
I can only hope that it goes well, and others do the same.
