This is a terse guide on bootstrapping virtual-machine images for
OpenStack infrastructure, with the goal of adding
continuous-integration support for new platforms. It might also be
handy if you are trying to replicate the upstream CI environment.
It covers deployment to
Rackspace for
testing; Rackspace is one of the major providers of capacity for the
OpenStack Infrastructure project, so it makes a good place to start
when building up your support story.
Firstly, get an Ubuntu Trusty environment to build the image in (other
environments, like CentOS or Fedora,
probably work -- but take this
step to minimise differences to what the automated machinery of what
upstream does). You want a fair bit of memory, and plenty of
disk-space.
The tool used for building virtual-machine images is
diskimage-builder. In
short, it takes a series of
elements, which are really just scripts
run in different phases of the build process.
I'll describe building a Fedora image, since that's been my focus
lately. We will use a
fedora-minimal element -- this means the
system is bootstrapped from nothing inside a
chroot environment,
before eventually being turned into a virtual-machine image (contrast
this to the
fedora element, which bases itself off the
qcow2
images provided by the official Fedora cloud project). Thus you'll
need a few things installed on the Ubuntu host to deal with
bootstrapping a Fedora chroot
apt-get install yum yum-utils python-lzma
You will hit stuff like that
python-lzma dependency on this
road-less-travelled -- technically it is a bug that
yum packages on
Ubuntu don't depend on it; without it you will get strange
yum
errors about unsupported compression.
At this point, you can bootstrap your
diskimage-builder environment.
You probably want
diskimage-builder from
git, and then build up a
virtualenv for your support bits and pieces.
git clone git://git.openstack.org/openstack/diskimage-builder
virtualenv dib_env
. dib_env/bin/activate
pip install dib-utils
is a small part of
diskimage-builder that is split-out;
don't think too much about it.
While
diskimage-builder is responsible for the creation of the basic
image, there are a number of elements provided by the OpenStack
project-config repository that bootstrap the OpenStack environment.
This does
a lot of stuff, including caching all
git trees (so CI
jobs aren't cloning everything constantly) and running puppet setup.
git clone git://git.openstack.org/openstack-infra/project-config
There's one more trick required for building the
VHD images that
Rackspace requires; make sure you install the patched
vhd-util as
described in the
script help.
At this point, you can probably build an image. Here's something
approximating what you will want to do
break=after-error \
TMP_DIR=~/tmp \
ELEMENTS_PATH=~/project-config/nodepool/elements \
DIB_DEV_USER_PASSWORD="password" DIB_DEV_USER_PWDLESS_SUDO=1 \
DISTRO=23 \
./bin/disk-image-create -x --no-tmpfs -t vhd \
fedora-minimal vm devuser simple-init \
openstack-repos puppet nodepool-base node-devstack
To break some of this down
- break= will help you debug failing builds by dropping you into a
shell when one of the element parts fail.
- TMP_DIR should be set to somewhere with plenty of space; a
default /tmp with tmpfs is probably restricted to a couple of
gigabytes; not enough for a build.
- ELEMENTS_PATH will add the OpenStack specific elements
- DIB_DEV_USER_* flags will create a devuser login with a password
and sudo access. This is really important as it is most likely
you'll boot up fairly broken the first time, and you need a way to
log-in (this is not used in "production").
- DISTRO in this case says to build a Fedora 23 based-image, but
will vary depending on what you are trying to do.
- disk-image-create finally creates the image. We are telling it to
create a vhd based image, using fedora-minimal in this case. For
configuration we are using simple-init, which uses the glean project to configure
networking from a configuration-drive.
- By default, this will install glean from pypi. However,
adding DIB_INSTALL_TYPE_simple_init=repo would modify the
install to use the git source. This is handy if you have
changes in there that are not released to pypi yet.
This goes and does its thing; it will take about 20 minutes.
Depending on how far your platform diverges from the existing support,
it will require
a lot of work to get everything working so you can
get an image out the other side. To see a rough example of what
should be happening, see the logs of the
official image builds that happen for a variety of
platforms.
At some point, you should get a file
image.vhd which is now ready
to be deployed. The only reasonable way to do this is with
shade. You can quickly grab
this into the
virtualenv we created before
Now you'll need to setup a
clouds.yaml file to give yourself the
permissions to upload the image. It should look something like
clouds:
rax:
profile: rackspace
auth:
username: your_rax_username
password: your_rax_password
project_id: your_rax_accountnum
regions:
- IAD
You should know your user-name and password (whatever you log into the
website with), and when you login to Rackspace your
project_id
value is listed in the drop-down box labeled with your username as
Account #.
shade has no UI as such, so a simple script will
do the upload.
import shade
shade.simple_logging(debug=True)
cloud = shade.openstack_cloud(cloud='rax')
image = cloud.create_image('image-name', filename='image.vhd', wait=True)
Now wait -- this will also take a while. Even after upload it takes a
fair while to process (you will see the
shade debug output looping
around some
glance calls seeing if it is ready). If everything
works, the script should return and you should be able to see the new
image in the output of
nova list-images.
At this point, you're ready to try booting! One caveat is that the
Rackspace web interface does not seem to give you the option to boot
with a
configuration drive
available to the host, essential for
simple-init to bring up the
network. So boot via the API with something like
nova boot --flavor=2 --image=image-uuid --config-drive 1 test-image
This
should build and boot your image! This will allow you to open
a whole new door of debugging to get your image booting correctly.
You can now iterate by rebuilding/uploading/booting as required. Note
these are pretty big images, and uploaded in broken-up swift files, so
I find
swift delete --all helpful to reset between builds
(obviously, I have nothing else in
swift that I want to keep).
The Rackspace java-based console UI is rather annoying; it cuts itself
off every time the host reboots. This makes it quite difficult to
catch the early bootup, or modify grub options via the boot-loader,
etc. You might need to fiddle timeout options, etc in the image
build.
If you've managed to get your image booting and listening on the
network, you're a good-deal of the way towards having your platform
supported in upstream OpenStack CI. At this point, you likely want to
peruse the
nodepool configuration
and get an official build happening here. Once that is up, you can
start the process of adding jobs that use your platform to test!
Don't worry, there's plenty more that can go wrong from here -- but
you're on the way! OpenStack infra is a very dynamic environment,
which many changes in progress; so in general,
#openstack-infra on
freenode is going to be a great place to start looking for help.
Before I disappear into the Japanese winter holidays, here the Christmas update of all TeX Live packages. Nothing spectacular here, just the usual bunch of updates of loads of packages. There is a bug with respect to a file move from one package to another, will be fixed in an upload soon.
Updated packages
apnum, appendix, archaeologie, babel, babel-french, babel-greek, babel-hungarian, bangorcsthesis, beamer-verona, bhcexam, bibarts, bidi, bxjscls, chemfig, commado, comprehensive, context, cslatex, csplain, ctanify, doclicense, dvips, ejpecp, exsheets, fbb, fibeamer, fithesis, fontools, francais-bst, gitinfo2, glossaries, gnuplottex, greek-fontenc, hyph-utf8, indextools, ksp-thesis, l3build, l3experimental, l3kernel, l3packages, latexcourse-rug, lollipop, lstbayes, lualibs, luaotfload, luatexja, luatexko, luatodonotes, make4ht, mcf2graph, media9, mex, mfirstuc, mhchem, mptopdf, nameauth, nevelok, newtx, nicetext, nucleardata, ocgx2, pageslts, petri-nets, phonrule, pkuthss, powerdot, proofread, proposal, pst-labo, pst-solides3d, reledmac, sapthesis, serbian-lig, showlabels, t2, tcolorbox, tempora, testhyphens, tetex, tex4ebook, tex4ht, texlive-scripts, texsis, thuthesis, toptesi, tudscr, ucharcat, versonotes, xcharter, xepersian, xifthen, xindy, xint.
New packages
beamertheme-detlevcm, beamertheme-metropolis, beamertheme-phnompenh, beamer-verona, bitpattern, carbohydrates, delimseasy, drawmatrix, einfuehrung2, ellipse, ffslides, gitlog, greektonoi, ksp-thesis, longfbox, options, simpler-wick, unicode-data.
Enjoy.
Performance for patching compressed files
Contents files are usually compressed with gzip, and kept compressed locally because they are about 500 MB uncompressed and only 30MB compressed. I profiled this, and it turns out there is not much we can do about it: The majority of the time is spent inside zlib, mostly combining CRC checksums:
Going forward, I think a reasonable option might be to recompress Contents files using lzo they will be a bit bigger (50MB instead of 30MB), but lzo is about 6 times as fast (compressing a 430MB Contents file took 1 second instead of 6).
(This is just a copy of 
Git is the clear winner now, with the migration rate increasing recently.
Packaging helpers
As usual, the number of packages using CDBS is quite stable. The number of packages using traditional debhelper might soon be lower than those using CDBS.
Patch systems and packaging formats
Kurt has trouble keeping his mouth shut. This became widely apparent
when he was expelled from Catholic school for telling a visiting
dignitary to go fuck himself.
WB used to preach angrily against casual sex.
Kurt enlisted in the Army, and due to his high IQ he ended up in the
Army Security Agency. After training, he was stationed in Germany
where he performed signals intelligence functions like direction
finding and passing on information to the CIA and NSA.
One day WB shagged a boy with a leather hat.
Contrary to what one might assume, Kurt had awareness of consequences.
He refused officer training because that would have extended his
commitment from 4 to 6 years. He purposefully flunked his French
language proficiency exam. He wanted there to be no chance of him
getting transferred to Vietnam.
The following day, WB found herself confronted and asked to explain
why she had had a change of heart about casual sex. Oh, it wasn't
casual sex, she explained. He came over and said, I've been dying
to make love with you for the past three hours, so it was special,
she clarified. Thus the nature of the past communication failure
became clear to everyone but WB.
Despite all his efforts to avoid Vietnam, Kurt, of course, had trouble
keeping his mouth shut. So in 1967 he pissed off the wrong person and
found himself having to choose between Leavenworth and Saigon. He
chose life of alcoholism over prison.
It has become almost tradition for me, so yes, I'm attending 
After a mildly ridiculous amount of effort I made a 
Today I spend some time to go through the lists.debian.org bugreports.
In consequence I created three new lists:
Siri and I are on a journey through India and Nepal,
with the aim of learning about needs of Debian derivatives,
to improve Debian and encourage closer integration.
This week I'll be mostly doing drive-by bug-reporting.
As with last year we start by using the Debian Code Search, to look
for obviously broken patterns such as "