yeahuh / rpms / qemu-kvm

Forked from rpms/qemu-kvm 2 years ago
Clone
902636
From af4d66e07c86d7593f7d18ae4b6a2151123b529b Mon Sep 17 00:00:00 2001
902636
From: Eric Blake <eblake@redhat.com>
902636
Date: Tue, 2 Jun 2020 02:34:17 +0100
902636
Subject: [PATCH 12/26] qcow2: Expose bitmaps' size during measure
902636
902636
RH-Author: Eric Blake <eblake@redhat.com>
902636
Message-id: <20200602023420.2133649-10-eblake@redhat.com>
902636
Patchwork-id: 97072
902636
O-Subject: [RHEL-AV-8.2.1 qemu-kvm PATCH 09/12] qcow2: Expose bitmaps' size during measure
902636
Bugzilla: 1779893 1779904
902636
RH-Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
902636
RH-Acked-by: Max Reitz <mreitz@redhat.com>
902636
RH-Acked-by: Kevin Wolf <kwolf@redhat.com>
902636
902636
It's useful to know how much space can be occupied by qcow2 persistent
902636
bitmaps, even though such metadata is unrelated to the guest-visible
902636
data.  Report this value as an additional QMP field, present when
902636
measuring an existing image and output format that both support
902636
bitmaps.  Update iotest 178 and 190 to updated output, as well as new
902636
coverage in 190 demonstrating non-zero values made possible with the
902636
recently-added qemu-img bitmap command (see 3b51ab4b).
902636
902636
The new 'bitmaps size:' field is displayed automatically as part of
902636
'qemu-img measure' any time it is present in QMP (that is, any time
902636
both the source image being measured and destination format support
902636
bitmaps, even if the measurement is 0 because there are no bitmaps
902636
present).  If the field is absent, it means that no bitmaps can be
902636
copied (source, destination, or both lack bitmaps, including when
902636
measuring based on size rather than on a source image).  This behavior
902636
is compatible with an upcoming patch adding 'qemu-img convert
902636
--bitmaps': that command will fail in the same situations where this
902636
patch omits the field.
902636
902636
The addition of a new field demonstrates why we should always
902636
zero-initialize qapi C structs; while the qcow2 driver still fully
902636
populates all fields, the raw and crypto drivers had to be tweaked to
902636
avoid uninitialized data.
902636
902636
Consideration was also given towards having a 'qemu-img measure
902636
--bitmaps' which errors out when bitmaps are not possible, and
902636
otherwise sums the bitmaps into the existing allocation totals rather
902636
than displaying as a separate field, as a potential convenience
902636
factor.  But this was ultimately decided to be more complexity than
902636
necessary when the QMP interface was sufficient enough with bitmaps
902636
remaining a separate field.
902636
902636
See also: https://bugzilla.redhat.com/1779904
902636
902636
Reported-by: Nir Soffer <nsoffer@redhat.com>
902636
Signed-off-by: Eric Blake <eblake@redhat.com>
902636
Message-Id: <20200521192137.1120211-3-eblake@redhat.com>
902636
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
902636
(cherry picked from commit 5d72c68b49769c927e90b78af6d90f6a384b26ac)
902636
902636
Signed-off-by: Danilo C. L. de Paula <ddepaula@redhat.com>
902636
902636
Conflicts:
902636
	block/crypto.c - commit a9da6e49 not present (no measure support)
902636
	docs/tools/qemu-img.rst - changes in qemu-img.texi instead
902636
Signed-off-by: Eric Blake <eblake@redhat.com>
902636
902636
Signed-off-by: Danilo C. L. de Paula <ddepaula@redhat.com>
902636
---
902636
 block/qcow2-bitmap.c             | 36 ++++++++++++++++++++++++++++++
902636
 block/qcow2.c                    | 14 +++++++++---
902636
 block/qcow2.h                    |  2 ++
902636
 block/raw-format.c               |  2 +-
902636
 qapi/block-core.json             | 16 +++++++++-----
902636
 qemu-img.c                       |  3 +++
902636
 qemu-img.texi                    |  7 ++++++
902636
 tests/qemu-iotests/178.out.qcow2 | 16 ++++++++++++++
902636
 tests/qemu-iotests/190           | 47 ++++++++++++++++++++++++++++++++++++++--
902636
 tests/qemu-iotests/190.out       | 27 ++++++++++++++++++++++-
902636
 10 files changed, 158 insertions(+), 12 deletions(-)
902636
902636
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
902636
index cbac905..10d1297 100644
902636
--- a/block/qcow2-bitmap.c
902636
+++ b/block/qcow2-bitmap.c
902636
@@ -1766,3 +1766,39 @@ bool qcow2_supports_persistent_dirty_bitmap(BlockDriverState *bs)
902636
 
902636
     return s->qcow_version >= 3;
902636
 }
902636
+
902636
+/*
902636
+ * Compute the space required for bitmaps in @bs.
902636
+ *
902636
+ * The computation is based as if copying to a new image with the
902636
+ * given @cluster_size, which may differ from the cluster size in @bs.
902636
+ */
902636
+uint64_t qcow2_get_persistent_dirty_bitmap_size(BlockDriverState *bs,
902636
+                                                uint32_t cluster_size)
902636
+{
902636
+    uint64_t bitmaps_size = 0;
902636
+    BdrvDirtyBitmap *bm;
902636
+    size_t bitmap_dir_size = 0;
902636
+
902636
+    FOR_EACH_DIRTY_BITMAP(bs, bm) {
902636
+        if (bdrv_dirty_bitmap_get_persistence(bm)) {
902636
+            const char *name = bdrv_dirty_bitmap_name(bm);
902636
+            uint32_t granularity = bdrv_dirty_bitmap_granularity(bm);
902636
+            uint64_t bmbytes =
902636
+                get_bitmap_bytes_needed(bdrv_dirty_bitmap_size(bm),
902636
+                                        granularity);
902636
+            uint64_t bmclusters = DIV_ROUND_UP(bmbytes, cluster_size);
902636
+
902636
+            /* Assume the entire bitmap is allocated */
902636
+            bitmaps_size += bmclusters * cluster_size;
902636
+            /* Also reserve space for the bitmap table entries */
902636
+            bitmaps_size += ROUND_UP(bmclusters * sizeof(uint64_t),
902636
+                                     cluster_size);
902636
+            /* And space for contribution to bitmap directory size */
902636
+            bitmap_dir_size += calc_dir_entry_size(strlen(name), 0);
902636
+        }
902636
+    }
902636
+    bitmaps_size += ROUND_UP(bitmap_dir_size, cluster_size);
902636
+
902636
+    return bitmaps_size;
902636
+}
902636
diff --git a/block/qcow2.c b/block/qcow2.c
902636
index 36b0f7d..dbd870a 100644
902636
--- a/block/qcow2.c
902636
+++ b/block/qcow2.c
902636
@@ -4751,16 +4751,24 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
902636
         required = virtual_size;
902636
     }
902636
 
902636
-    info = g_new(BlockMeasureInfo, 1);
902636
+    info = g_new0(BlockMeasureInfo, 1);
902636
     info->fully_allocated =
902636
         qcow2_calc_prealloc_size(virtual_size, cluster_size,
902636
                                  ctz32(refcount_bits)) + luks_payload_size;
902636
 
902636
-    /* Remove data clusters that are not required.  This overestimates the
902636
+    /*
902636
+     * Remove data clusters that are not required.  This overestimates the
902636
      * required size because metadata needed for the fully allocated file is
902636
-     * still counted.
902636
+     * still counted.  Show bitmaps only if both source and destination
902636
+     * would support them.
902636
      */
902636
     info->required = info->fully_allocated - virtual_size + required;
902636
+    info->has_bitmaps = version >= 3 && in_bs &&
902636
+        bdrv_supports_persistent_dirty_bitmap(in_bs);
902636
+    if (info->has_bitmaps) {
902636
+        info->bitmaps = qcow2_get_persistent_dirty_bitmap_size(in_bs,
902636
+                                                               cluster_size);
902636
+    }
902636
     return info;
902636
 
902636
 err:
902636
diff --git a/block/qcow2.h b/block/qcow2.h
902636
index ceb1ceb..3297e6b 100644
902636
--- a/block/qcow2.h
902636
+++ b/block/qcow2.h
902636
@@ -768,6 +768,8 @@ int qcow2_co_remove_persistent_dirty_bitmap(BlockDriverState *bs,
902636
                                             const char *name,
902636
                                             Error **errp);
902636
 bool qcow2_supports_persistent_dirty_bitmap(BlockDriverState *bs);
902636
+uint64_t qcow2_get_persistent_dirty_bitmap_size(BlockDriverState *bs,
902636
+                                                uint32_t cluster_size);
902636
 
902636
 ssize_t coroutine_fn
902636
 qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
902636
diff --git a/block/raw-format.c b/block/raw-format.c
902636
index 93b25e1..4bb54f4 100644
902636
--- a/block/raw-format.c
902636
+++ b/block/raw-format.c
902636
@@ -346,7 +346,7 @@ static BlockMeasureInfo *raw_measure(QemuOpts *opts, BlockDriverState *in_bs,
902636
                             BDRV_SECTOR_SIZE);
902636
     }
902636
 
902636
-    info = g_new(BlockMeasureInfo, 1);
902636
+    info = g_new0(BlockMeasureInfo, 1);
902636
     info->required = required;
902636
 
902636
     /* Unallocated sectors count towards the file size in raw images */
902636
diff --git a/qapi/block-core.json b/qapi/block-core.json
902636
index a64ad81..2893209 100644
902636
--- a/qapi/block-core.json
902636
+++ b/qapi/block-core.json
902636
@@ -689,18 +689,24 @@
902636
 # efficiently so file size may be smaller than virtual disk size.
902636
 #
902636
 # The values are upper bounds that are guaranteed to fit the new image file.
902636
-# Subsequent modification, such as internal snapshot or bitmap creation, may
902636
-# require additional space and is not covered here.
902636
+# Subsequent modification, such as internal snapshot or further bitmap
902636
+# creation, may require additional space and is not covered here.
902636
 #
902636
-# @required: Size required for a new image file, in bytes.
902636
+# @required: Size required for a new image file, in bytes, when copying just
902636
+#            allocated guest-visible contents.
902636
 #
902636
 # @fully-allocated: Image file size, in bytes, once data has been written
902636
-#                   to all sectors.
902636
+#                   to all sectors, when copying just guest-visible contents.
902636
+#
902636
+# @bitmaps: Additional size required if all the top-level bitmap metadata
902636
+#           in the source image were to be copied to the destination,
902636
+#           present only when source and destination both support
902636
+#           persistent bitmaps. (since 5.1)
902636
 #
902636
 # Since: 2.10
902636
 ##
902636
 { 'struct': 'BlockMeasureInfo',
902636
-  'data': {'required': 'int', 'fully-allocated': 'int'} }
902636
+  'data': {'required': 'int', 'fully-allocated': 'int', '*bitmaps': 'int'} }
902636
 
902636
 ##
902636
 # @query-block:
902636
diff --git a/qemu-img.c b/qemu-img.c
902636
index 11a4537..b57856e 100644
902636
--- a/qemu-img.c
902636
+++ b/qemu-img.c
902636
@@ -5212,6 +5212,9 @@ static int img_measure(int argc, char **argv)
902636
     if (output_format == OFORMAT_HUMAN) {
902636
         printf("required size: %" PRIu64 "\n", info->required);
902636
         printf("fully allocated size: %" PRIu64 "\n", info->fully_allocated);
902636
+        if (info->has_bitmaps) {
902636
+            printf("bitmaps size: %" PRIu64 "\n", info->bitmaps);
902636
+        }
902636
     } else {
902636
         dump_json_block_measure_info(info);
902636
     }
902636
diff --git a/qemu-img.texi b/qemu-img.texi
902636
index abf2771..3670b96 100644
902636
--- a/qemu-img.texi
902636
+++ b/qemu-img.texi
902636
@@ -576,6 +576,7 @@ The following fields are reported:
902636
 @example
902636
 required size: 524288
902636
 fully allocated size: 1074069504
902636
+bitmaps size: 0
902636
 @end example
902636
 
902636
 The @code{required size} is the file size of the new image.  It may be smaller
902636
@@ -586,6 +587,12 @@ been written to all sectors.  This is the maximum size that the image file can
902636
 occupy with the exception of internal snapshots, dirty bitmaps, vmstate data,
902636
 and other advanced image format features.
902636
 
902636
+The @code{bitmaps size} is the additional size required in order to
902636
+copy bitmaps from a source image in addition to the guest-visible
902636
+data; the line is omitted if either source or destination lacks
902636
+bitmap support, or 0 if bitmaps are supported but there is nothing to
902636
+copy.
902636
+
902636
 @item snapshot [--object @var{objectdef}] [--image-opts] [-U] [-q] [-l | -a @var{snapshot} | -c @var{snapshot} | -d @var{snapshot}] @var{filename}
902636
 
902636
 List, apply, create or delete snapshots in image @var{filename}.
902636
diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
902636
index 345eab3..b9ed41b 100644
902636
--- a/tests/qemu-iotests/178.out.qcow2
902636
+++ b/tests/qemu-iotests/178.out.qcow2
902636
@@ -37,6 +37,7 @@ qemu-img: The image size is too large (try using a larger cluster size)
902636
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
902636
 required size: 196608
902636
 fully allocated size: 196608
902636
+bitmaps size: 0
902636
 
902636
 converted image file size in bytes: 196608
902636
 
902636
@@ -45,6 +46,7 @@ converted image file size in bytes: 196608
902636
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
902636
 required size: 393216
902636
 fully allocated size: 1074135040
902636
+bitmaps size: 0
902636
 wrote 512/512 bytes at offset 512
902636
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
902636
 wrote 65536/65536 bytes at offset 65536
902636
@@ -53,6 +55,7 @@ wrote 64512/64512 bytes at offset 134217728
902636
 63 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
902636
 required size: 589824
902636
 fully allocated size: 1074135040
902636
+bitmaps size: 0
902636
 
902636
 converted image file size in bytes: 524288
902636
 
902636
@@ -60,6 +63,7 @@ converted image file size in bytes: 524288
902636
 
902636
 required size: 524288
902636
 fully allocated size: 1074135040
902636
+bitmaps size: 0
902636
 
902636
 converted image file size in bytes: 458752
902636
 
902636
@@ -67,16 +71,19 @@ converted image file size in bytes: 458752
902636
 
902636
 required size: 1074135040
902636
 fully allocated size: 1074135040
902636
+bitmaps size: 0
902636
 
902636
 == qcow2 input image and LUKS encryption ==
902636
 
902636
 required size: 2686976
902636
 fully allocated size: 1076232192
902636
+bitmaps size: 0
902636
 
902636
 == qcow2 input image and preallocation (human) ==
902636
 
902636
 required size: 1074135040
902636
 fully allocated size: 1074135040
902636
+bitmaps size: 0
902636
 
902636
 converted image file size in bytes: 1074135040
902636
 
902636
@@ -87,6 +94,7 @@ wrote 8388608/8388608 bytes at offset 0
902636
 8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
902636
 required size: 8716288
902636
 fully allocated size: 8716288
902636
+bitmaps size: 0
902636
 
902636
 converted image file size in bytes: 8716288
902636
 
902636
@@ -173,6 +181,7 @@ qemu-img: The image size is too large (try using a larger cluster size)
902636
 
902636
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
902636
 {
902636
+    "bitmaps": 0,
902636
     "required": 196608,
902636
     "fully-allocated": 196608
902636
 }
902636
@@ -183,6 +192,7 @@ converted image file size in bytes: 196608
902636
 
902636
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
902636
 {
902636
+    "bitmaps": 0,
902636
     "required": 393216,
902636
     "fully-allocated": 1074135040
902636
 }
902636
@@ -193,6 +203,7 @@ wrote 65536/65536 bytes at offset 65536
902636
 wrote 64512/64512 bytes at offset 134217728
902636
 63 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
902636
 {
902636
+    "bitmaps": 0,
902636
     "required": 589824,
902636
     "fully-allocated": 1074135040
902636
 }
902636
@@ -202,6 +213,7 @@ converted image file size in bytes: 524288
902636
 == qcow2 input image with internal snapshot (json) ==
902636
 
902636
 {
902636
+    "bitmaps": 0,
902636
     "required": 524288,
902636
     "fully-allocated": 1074135040
902636
 }
902636
@@ -211,6 +223,7 @@ converted image file size in bytes: 458752
902636
 == qcow2 input image and a backing file (json) ==
902636
 
902636
 {
902636
+    "bitmaps": 0,
902636
     "required": 1074135040,
902636
     "fully-allocated": 1074135040
902636
 }
902636
@@ -218,6 +231,7 @@ converted image file size in bytes: 458752
902636
 == qcow2 input image and LUKS encryption ==
902636
 
902636
 {
902636
+    "bitmaps": 0,
902636
     "required": 2686976,
902636
     "fully-allocated": 1076232192
902636
 }
902636
@@ -225,6 +239,7 @@ converted image file size in bytes: 458752
902636
 == qcow2 input image and preallocation (json) ==
902636
 
902636
 {
902636
+    "bitmaps": 0,
902636
     "required": 1074135040,
902636
     "fully-allocated": 1074135040
902636
 }
902636
@@ -237,6 +252,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=8388608
902636
 wrote 8388608/8388608 bytes at offset 0
902636
 8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
902636
 {
902636
+    "bitmaps": 0,
902636
     "required": 8716288,
902636
     "fully-allocated": 8716288
902636
 }
902636
diff --git a/tests/qemu-iotests/190 b/tests/qemu-iotests/190
902636
index eb766ad..5084ccd 100755
902636
--- a/tests/qemu-iotests/190
902636
+++ b/tests/qemu-iotests/190
902636
@@ -2,7 +2,7 @@
902636
 #
902636
 # qemu-img measure sub-command tests on huge qcow2 files
902636
 #
902636
-# Copyright (C) 2017 Red Hat, Inc.
902636
+# Copyright (C) 2017-2020 Red Hat, Inc.
902636
 #
902636
 # This program is free software; you can redistribute it and/or modify
902636
 # it under the terms of the GNU General Public License as published by
902636
@@ -42,7 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
902636
 _supported_fmt qcow2
902636
 _supported_proto file
902636
 
902636
-echo "== Huge file =="
902636
+echo "== Huge file without bitmaps =="
902636
 echo
902636
 
902636
 IMGOPTS='cluster_size=2M' _make_test_img 2T
902636
@@ -51,6 +51,49 @@ $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG"
902636
 $QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG"
902636
 $QEMU_IMG measure -O qcow2 -o cluster_size=2M -f qcow2 "$TEST_IMG"
902636
 
902636
+echo
902636
+echo "== Huge file with bitmaps =="
902636
+echo
902636
+
902636
+$QEMU_IMG bitmap --add --granularity 512 -f qcow2 "$TEST_IMG" b1
902636
+$QEMU_IMG bitmap --add -g 2M -f qcow2 "$TEST_IMG" b2
902636
+
902636
+# No bitmap without a source
902636
+$QEMU_IMG measure -O qcow2 --size 10M
902636
+# No bitmap output, since raw does not support it
902636
+$QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG"
902636
+# No bitmap output, since no bitmaps on raw source. Munge required size, as
902636
+# some filesystems store the qcow2 file with less sparseness than others
902636
+$QEMU_IMG measure -O qcow2 -f raw "$TEST_IMG" |
902636
+    sed '/^required size:/ s/[0-9][0-9]*/SIZE/'
902636
+# No bitmap output, since v2 does not support it
902636
+$QEMU_IMG measure -O qcow2 -o compat=0.10 -f qcow2 "$TEST_IMG"
902636
+
902636
+# Compute expected output: bitmap clusters + bitmap tables + bitmaps directory
902636
+echo
902636
+val2T=$((2*1024*1024*1024*1024))
902636
+cluster=$((64*1024))
902636
+b1clusters=$(( (val2T/512/8 + cluster - 1) / cluster ))
902636
+b2clusters=$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster ))
902636
+echo expected bitmap $((b1clusters * cluster +
902636
+                        (b1clusters * 8 + cluster - 1) / cluster * cluster +
902636
+                        b2clusters * cluster +
902636
+                        (b2clusters * 8 + cluster - 1) / cluster * cluster +
902636
+                        cluster))
902636
+$QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG"
902636
+
902636
+# Compute expected output: bitmap clusters + bitmap tables + bitmaps directory
902636
+echo
902636
+cluster=$((2*1024*1024))
902636
+b1clusters=$(( (val2T/512/8 + cluster - 1) / cluster ))
902636
+b2clusters=$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster ))
902636
+echo expected bitmap $((b1clusters * cluster +
902636
+                        (b1clusters * 8 + cluster - 1) / cluster * cluster +
902636
+                        b2clusters * cluster +
902636
+                        (b2clusters * 8 + cluster - 1) / cluster * cluster +
902636
+                        cluster))
902636
+$QEMU_IMG measure --output=json -O qcow2 -o cluster_size=2M -f qcow2 "$TEST_IMG"
902636
+
902636
 # success, all done
902636
 echo "*** done"
902636
 rm -f $seq.full
902636
diff --git a/tests/qemu-iotests/190.out b/tests/qemu-iotests/190.out
902636
index d001942..ed9d821 100644
902636
--- a/tests/qemu-iotests/190.out
902636
+++ b/tests/qemu-iotests/190.out
902636
@@ -1,11 +1,36 @@
902636
 QA output created by 190
902636
-== Huge file ==
902636
+== Huge file without bitmaps ==
902636
 
902636
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2199023255552
902636
 required size: 2199023255552
902636
 fully allocated size: 2199023255552
902636
 required size: 335806464
902636
 fully allocated size: 2199359062016
902636
+bitmaps size: 0
902636
 required size: 18874368
902636
 fully allocated size: 2199042129920
902636
+bitmaps size: 0
902636
+
902636
+== Huge file with bitmaps ==
902636
+
902636
+required size: 327680
902636
+fully allocated size: 10813440
902636
+required size: 2199023255552
902636
+fully allocated size: 2199023255552
902636
+required size: SIZE
902636
+fully allocated size: 17170432
902636
+required size: 335806464
902636
+fully allocated size: 2199359062016
902636
+
902636
+expected bitmap 537198592
902636
+required size: 335806464
902636
+fully allocated size: 2199359062016
902636
+bitmaps size: 537198592
902636
+
902636
+expected bitmap 545259520
902636
+{
902636
+    "bitmaps": 545259520,
902636
+    "required": 18874368,
902636
+    "fully-allocated": 2199042129920
902636
+}
902636
 *** done
902636
-- 
902636
1.8.3.1
902636