QEMU 2.5 has just been released, with
a lot of new features. As with the previous release, we have also created a
video changelog.
I plan to write a few blog posts explaining some of the things I have been working on. In this one I m going to talk about how to control the size of the qcow2 L2 cache. But first, let s see why that cache is useful.
The qcow2 file format
qcow2 is the main format for disk images used by QEMU. One of the features of this format is that its size grows on demand, and the disk space is only allocated when it is actually needed by the virtual machine.
A qcow2 file is organized in units of constant size called clusters. The virtual disk seen by the guest is also divided into guest clusters of the same size. QEMU defaults to 64KB clusters, but a different value can be specified when creating a new image:
qemu-img create -f qcow2 -o cluster_size=128K hd.qcow2 4G
In order to map the virtual disk as seen by the guest to the qcow2 image in the host, the qcow2 image contains a set of tables organized in a two-level structure. These are called the L1 and L2 tables.
There is one single L1 table per disk image. This table is small and is always kept in memory.
There can be many L2 tables, depending on how much space has been allocated in the image. Each table is one cluster in size. In order to read or write data to the virtual disk, QEMU needs to read its corresponding L2 table to find out where that data is located. Since reading the table for each I/O operation can be expensive, QEMU keeps a cache of L2 tables in memory to speed up disk access.
The L2 cache can have a dramatic impact on performance. As an example, here s the number of I/O operations per second that I get with random read requests in a fully populated 20GB disk image:
| L2 cache size |
Average IOPS |
| 1 MB |
5100 |
| 1,5 MB |
7300 |
| 2 MB |
12700 |
| 2,5 MB |
63600 |
If you re using an older version of QEMU you might have trouble getting the most out of the qcow2 cache because of
this bug, so either upgrade to at least QEMU 2.3 or apply this patch.
(in addition to the L2 cache, QEMU also keeps a refcount cache. This is used for cluster allocation and internal snapshots, but I m not covering it in this post. Please refer to the qcow2 documentation if you want to know more about refcount tables)
Understanding how to choose the right cache size
In order to choose the cache size we need to know how it relates to the amount of allocated space.
The amount of virtual disk that can be mapped by the L2 cache (in bytes) is:
disk_size = l2_cache_size * cluster_size / 8
With the default values for cluster_size (64KB) that is
disk_size = l2_cache_size * 8192
So in order to have a cache that can cover
n GB of disk space with the default cluster size we need
l2_cache_size = disk_size_GB * 131072
QEMU has a default L2 cache of 1MB (1048576 bytes) so using the formulas we ve just seen we have
1048576 / 131072 = 8 GB of virtual disk covered by that cache. This means that if the size of your virtual disk is larger than 8 GB you can speed up disk access by increasing the size of the L2 cache. Otherwise you ll be fine with the defaults.
How to configure the cache size
Cache sizes can be configured using the
-drive option in the command-line, or the
blockdev-add QMP command.
There are three options available, and all of them take bytes:
- l2-cache-size: maximum size of the L2 table cache
- refcount-cache-size: maximum size of the refcount block cache
- cache-size: maximum size of both caches combined
There are two things that need to be taken into account:
- Both the L2 and refcount block caches must have a size that is a multiple of the cluster size.
- If you only set one of the options above, QEMU will automatically adjust the others so that the L2 cache is 4 times bigger than the refcount cache.
This means that these three options are equivalent:
-drive file=hd.qcow2,l2-cache-size=2097152
-drive file=hd.qcow2,refcount-cache-size=524288
-drive file=hd.qcow2,cache-size=2621440
Although I m not covering the refcount cache here, it s worth noting that it s used much less often than the L2 cache, so it s perfectly reasonable to keep it small:
-drive file=hd.qcow2,l2-cache-size=4194304,refcount-cache-size=262144
Reducing the memory usage
The problem with a large cache size is that it obviously needs more memory. QEMU has a separate L2 cache for each qcow2 file, so if you re using many big images you might need a considerable amount of memory if you want to have a reasonably sized cache for each one. The problem gets worse if you add backing files and snapshots to the mix.
Consider this scenario:
Here,
hd0 is a fully populated disk image, and
hd1 a freshly created image as a result of a
snapshot operation. Reading data from this virtual disk will fill up the L2 cache of
hd0, because that s where the actual data is read from. However
hd0 itself is read-only, and if you write data to the virtual disk it will go to the active image,
hd1, filling up its L2 cache as a result. At some point you ll have in memory cache entries from
hd0 that you won t need anymore because all the data from those clusters is now retrieved from
hd1.
Let s now create a new live snapshot:
Now we have the same problem again. If we write data to the virtual disk it will go to
hd2 and its L2 cache will start to fill up. At some point a significant amount of the data from the virtual disk will be in
hd2, however the L2 caches of
hd0 and
hd1 will be full as a result of the previous operations, even if they re no longer needed.
Imagine now a scenario with several virtual disks and a long chain of qcow2 images for each one of them. See the problem?
I wanted to improve this a bit so I was working on a new setting that allows the user to reduce the memory usage by cleaning unused cache entries when they are not being used.
This new setting is available in QEMU 2.5, and is called
cache-clean-interval . It defines an interval (in seconds) after which all cache entries that haven t been accessed are removed from memory.
This example removes all unused cache entries every 15 minutes:
-drive file=hd.qcow2,cache-clean-interval=900
If unset, the default value for this parameter is 0 and it disables this feature.
Further information
In this post I only intended to give a brief summary of the qcow2 L2 cache and how to tune it in order to increase the I/O performance, but it is by no means an exhaustive description of the disk format.
If you want to know more about the qcow2 format here s a few links:
Acknowledgments
My work in QEMU is sponsored by
Outscale and has been made possible by
Igalia and the invaluable help of the QEMU development team.
Enjoy QEMU 2.5!
A while back I wrote about some issues with converting
a two-disk RAID system to a one-disk system, but just to recap:
I woke up this morning and realized that for the first time since
Today I was wondering about converting a pdf made from scan of a book
into djvu, hopefully to reduce the size, without too much loss of
quality. My initial experiments with
Performance for patching compressed files
Contents files are usually compressed with gzip, and kept compressed locally because they are about 500 MB uncompressed and only 30MB compressed. I profiled this, and it turns out there is not much we can do about it: The majority of the time is spent inside zlib, mostly combining CRC checksums:
Going forward, I think a reasonable option might be to recompress Contents files using lzo they will be a bit bigger (50MB instead of 30MB), but lzo is about 6 times as fast (compressing a 430MB Contents file took 1 second instead of 6).
What happened in the 
Portability toward other systems has been improved: old versions of GNU diff are now supported (Mike McQuaid), suggestion of the appropriate locale is now the more generic 
Just to give a brief recap of why I'm writing this post, I've to
describe a brief encounter I had with systemd which rendered my system
unbootable. I first felt its systemd's problem but later figured out
it was my mistake. So below is brief story divided into 2 sections
Problem and Solution. Here Problem describes issue I faced with
systemd and Solution discusses the .mount and .automount suffix
files used by systemd.
Following on 
It s been a while since my last what s been happening behind the scenes e-mail so I m here to report on what has been happening within the GNOME Infrastructure, its future plans and my personal sensations about a challenge that started around three (3) years ago when 
My monthly report covers a large part of what I have been doing in the free software world. I write it for 
