Blame SOURCES/kvm-Add-functional-acceptance-tests-infrastructure.patch

26ba25
From 687a123ab2165fa3adf9e3469577c22008125270 Mon Sep 17 00:00:00 2001
26ba25
From: Yash Mankad <ymankad@redhat.com>
26ba25
Date: Wed, 12 Dec 2018 00:14:35 +0000
26ba25
Subject: [PATCH 07/13] Add functional/acceptance tests infrastructure
26ba25
MIME-Version: 1.0
26ba25
Content-Type: text/plain; charset=UTF-8
26ba25
Content-Transfer-Encoding: 8bit
26ba25
26ba25
RH-Author: Yash Mankad <ymankad@redhat.com>
26ba25
Message-id: <b1ac81d7ccd9e43f36a74dc6a008974484508c7f.1544573601.git.ymankad@redhat.com>
26ba25
Patchwork-id: 83432
26ba25
O-Subject: [RHEL-8.0 qemu-kvm PATCH v2 1/7] Add functional/acceptance tests infrastructure
26ba25
Bugzilla: 1655807
26ba25
RH-Acked-by: Eduardo Habkost <ehabkost@redhat.com>
26ba25
RH-Acked-by: John Snow <jsnow@redhat.com>
26ba25
RH-Acked-by: Philippe Mathieu-Daudé <philmd@redhat.com>
26ba25
26ba25
From: Cleber Rosa <crosa@redhat.com>
26ba25
26ba25
This patch adds the very minimum infrastructure necessary for writing
26ba25
and running functional/acceptance tests, including:
26ba25
26ba25
 * Documentation
26ba25
 * The avocado_qemu.Test base test class
26ba25
 * One example tests (version.py)
26ba25
26ba25
Additional functionality is expected to be added along the tests that
26ba25
require them.
26ba25
26ba25
Signed-off-by: Cleber Rosa <crosa@redhat.com>
26ba25
Message-Id: <20180530184156.15634-2-crosa@redhat.com>
26ba25
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
26ba25
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
26ba25
[ehabkost: fix typo on testing.rst]
26ba25
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
26ba25
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
26ba25
(cherry picked from commit c3d7e8c90db208b1d876f8d6458c2dfca169137f)
26ba25
Signed-off-by: Yash Mankad <ymankad@redhat.com>
26ba25
26ba25
Signed-off-by: Danilo C. L. de Paula <ddepaula@redhat.com>
26ba25
---
26ba25
 docs/devel/testing.rst                    | 192 ++++++++++++++++++++++++++++++
26ba25
 tests/acceptance/README.rst               |  10 ++
26ba25
 tests/acceptance/avocado_qemu/__init__.py |  54 +++++++++
26ba25
 tests/acceptance/version.py               |  24 ++++
26ba25
 4 files changed, 280 insertions(+)
26ba25
 create mode 100644 tests/acceptance/README.rst
26ba25
 create mode 100644 tests/acceptance/avocado_qemu/__init__.py
26ba25
 create mode 100644 tests/acceptance/version.py
26ba25
26ba25
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
26ba25
index 0ca1a2d..f33e5a8 100644
26ba25
--- a/docs/devel/testing.rst
26ba25
+++ b/docs/devel/testing.rst
26ba25
@@ -484,3 +484,195 @@ supported. To start the fuzzer, run
26ba25
 
26ba25
 Alternatively, some command different from "qemu-img info" can be tested, by
26ba25
 changing the ``-c`` option.
26ba25
+
26ba25
+Acceptance tests using the Avocado Framework
26ba25
+============================================
26ba25
+
26ba25
+The ``tests/acceptance`` directory hosts functional tests, also known
26ba25
+as acceptance level tests.  They're usually higher level tests, and
26ba25
+may interact with external resources and with various guest operating
26ba25
+systems.
26ba25
+
26ba25
+These tests are written using the Avocado Testing Framework (which must
26ba25
+be installed separately) in conjunction with a the ``avocado_qemu.Test``
26ba25
+class, implemented at ``tests/acceptance/avocado_qemu``.
26ba25
+
26ba25
+Tests based on ``avocado_qemu.Test`` can easily:
26ba25
+
26ba25
+ * Customize the command line arguments given to the convenience
26ba25
+   ``self.vm`` attribute (a QEMUMachine instance)
26ba25
+
26ba25
+ * Interact with the QEMU monitor, send QMP commands and check
26ba25
+   their results
26ba25
+
26ba25
+ * Interact with the guest OS, using the convenience console device
26ba25
+   (which may be useful to assert the effectiveness and correctness of
26ba25
+   command line arguments or QMP commands)
26ba25
+
26ba25
+ * Interact with external data files that accompany the test itself
26ba25
+   (see ``self.get_data()``)
26ba25
+
26ba25
+ * Download (and cache) remote data files, such as firmware and kernel
26ba25
+   images
26ba25
+
26ba25
+ * Have access to a library of guest OS images (by means of the
26ba25
+   ``avocado.utils.vmimage`` library)
26ba25
+
26ba25
+ * Make use of various other test related utilities available at the
26ba25
+   test class itself and at the utility library:
26ba25
+
26ba25
+   - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
26ba25
+   - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
26ba25
+
26ba25
+Installation
26ba25
+------------
26ba25
+
26ba25
+To install Avocado and its dependencies, run:
26ba25
+
26ba25
+.. code::
26ba25
+
26ba25
+  pip install --user avocado-framework
26ba25
+
26ba25
+Alternatively, follow the instructions on this link:
26ba25
+
26ba25
+  http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#installing-avocado
26ba25
+
26ba25
+Overview
26ba25
+--------
26ba25
+
26ba25
+This directory provides the ``avocado_qemu`` Python module, containing
26ba25
+the ``avocado_qemu.Test`` class.  Here's a simple usage example:
26ba25
+
26ba25
+.. code::
26ba25
+
26ba25
+  from avocado_qemu import Test
26ba25
+
26ba25
+
26ba25
+  class Version(Test):
26ba25
+      """
26ba25
+      :avocado: enable
26ba25
+      :avocado: tags=quick
26ba25
+      """
26ba25
+      def test_qmp_human_info_version(self):
26ba25
+          self.vm.launch()
26ba25
+          res = self.vm.command('human-monitor-command',
26ba25
+                                command_line='info version')
26ba25
+          self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)')
26ba25
+
26ba25
+To execute your test, run:
26ba25
+
26ba25
+.. code::
26ba25
+
26ba25
+  avocado run version.py
26ba25
+
26ba25
+Tests may be classified according to a convention by using docstring
26ba25
+directives such as ``:avocado: tags=TAG1,TAG2``.  To run all tests
26ba25
+in the current directory, tagged as "quick", run:
26ba25
+
26ba25
+.. code::
26ba25
+
26ba25
+  avocado run -t quick .
26ba25
+
26ba25
+The ``avocado_qemu.Test`` base test class
26ba25
+-----------------------------------------
26ba25
+
26ba25
+The ``avocado_qemu.Test`` class has a number of characteristics that
26ba25
+are worth being mentioned right away.
26ba25
+
26ba25
+First of all, it attempts to give each test a ready to use QEMUMachine
26ba25
+instance, available at ``self.vm``.  Because many tests will tweak the
26ba25
+QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
26ba25
+is left to the test writer.
26ba25
+
26ba25
+At test "tear down", ``avocado_qemu.Test`` handles the QEMUMachine
26ba25
+shutdown.
26ba25
+
26ba25
+QEMUMachine
26ba25
+~~~~~~~~~~~
26ba25
+
26ba25
+The QEMUMachine API is already widely used in the Python iotests,
26ba25
+device-crash-test and other Python scripts.  It's a wrapper around the
26ba25
+execution of a QEMU binary, giving its users:
26ba25
+
26ba25
+ * the ability to set command line arguments to be given to the QEMU
26ba25
+   binary
26ba25
+
26ba25
+ * a ready to use QMP connection and interface, which can be used to
26ba25
+   send commands and inspect its results, as well as asynchronous
26ba25
+   events
26ba25
+
26ba25
+ * convenience methods to set commonly used command line arguments in
26ba25
+   a more succinct and intuitive way
26ba25
+
26ba25
+QEMU binary selection
26ba25
+~~~~~~~~~~~~~~~~~~~~~
26ba25
+
26ba25
+The QEMU binary used for the ``self.vm`` QEMUMachine instance will
26ba25
+primarily depend on the value of the ``qemu_bin`` parameter.  If it's
26ba25
+not explicitly set, its default value will be the result of a dynamic
26ba25
+probe in the same source tree.  A suitable binary will be one that
26ba25
+targets the architecture matching host machine.
26ba25
+
26ba25
+Based on this description, test writers will usually rely on one of
26ba25
+the following approaches:
26ba25
+
26ba25
+1) Set ``qemu_bin``, and use the given binary
26ba25
+
26ba25
+2) Do not set ``qemu_bin``, and use a QEMU binary named like
26ba25
+   "${arch}-softmmu/qemu-system-${arch}", either in the current
26ba25
+   working directory, or in the current source tree.
26ba25
+
26ba25
+The resulting ``qemu_bin`` value will be preserved in the
26ba25
+``avocado_qemu.Test`` as an attribute with the same name.
26ba25
+
26ba25
+Attribute reference
26ba25
+-------------------
26ba25
+
26ba25
+Besides the attributes and methods that are part of the base
26ba25
+``avocado.Test`` class, the following attributes are available on any
26ba25
+``avocado_qemu.Test`` instance.
26ba25
+
26ba25
+vm
26ba25
+~~
26ba25
+
26ba25
+A QEMUMachine instance, initially configured according to the given
26ba25
+``qemu_bin`` parameter.
26ba25
+
26ba25
+qemu_bin
26ba25
+~~~~~~~~
26ba25
+
26ba25
+The preserved value of the ``qemu_bin`` parameter or the result of the
26ba25
+dynamic probe for a QEMU binary in the current working directory or
26ba25
+source tree.
26ba25
+
26ba25
+Parameter reference
26ba25
+-------------------
26ba25
+
26ba25
+To understand how Avocado parameters are accessed by tests, and how
26ba25
+they can be passed to tests, please refer to::
26ba25
+
26ba25
+  http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#accessing-test-parameters
26ba25
+
26ba25
+Parameter values can be easily seen in the log files, and will look
26ba25
+like the following:
26ba25
+
26ba25
+.. code::
26ba25
+
26ba25
+  PARAMS (key=qemu_bin, path=*, default=x86_64-softmmu/qemu-system-x86_64) => 'x86_64-softmmu/qemu-system-x86_64
26ba25
+
26ba25
+qemu_bin
26ba25
+~~~~~~~~
26ba25
+
26ba25
+The exact QEMU binary to be used on QEMUMachine.
26ba25
+
26ba25
+Uninstalling Avocado
26ba25
+--------------------
26ba25
+
26ba25
+If you've followed the installation instructions above, you can easily
26ba25
+uninstall Avocado.  Start by listing the packages you have installed::
26ba25
+
26ba25
+  pip list --user
26ba25
+
26ba25
+And remove any package you want with::
26ba25
+
26ba25
+  pip uninstall <package_name>
26ba25
diff --git a/tests/acceptance/README.rst b/tests/acceptance/README.rst
26ba25
new file mode 100644
26ba25
index 0000000..89260fa
26ba25
--- /dev/null
26ba25
+++ b/tests/acceptance/README.rst
26ba25
@@ -0,0 +1,10 @@
26ba25
+============================================
26ba25
+Acceptance tests using the Avocado Framework
26ba25
+============================================
26ba25
+
26ba25
+This directory contains functional tests, also known as acceptance
26ba25
+level tests.  They're usually higher level, and may interact with
26ba25
+external resources and with various guest operating systems.
26ba25
+
26ba25
+For more information, please refer to ``docs/devel/testing.rst``,
26ba25
+section "Acceptance tests using the Avocado Framework".
26ba25
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
26ba25
new file mode 100644
26ba25
index 0000000..1e54fd5
26ba25
--- /dev/null
26ba25
+++ b/tests/acceptance/avocado_qemu/__init__.py
26ba25
@@ -0,0 +1,54 @@
26ba25
+# Test class and utilities for functional tests
26ba25
+#
26ba25
+# Copyright (c) 2018 Red Hat, Inc.
26ba25
+#
26ba25
+# Author:
26ba25
+#  Cleber Rosa <crosa@redhat.com>
26ba25
+#
26ba25
+# This work is licensed under the terms of the GNU GPL, version 2 or
26ba25
+# later.  See the COPYING file in the top-level directory.
26ba25
+
26ba25
+import os
26ba25
+import sys
26ba25
+
26ba25
+import avocado
26ba25
+
26ba25
+SRC_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
26ba25
+SRC_ROOT_DIR = os.path.abspath(os.path.dirname(SRC_ROOT_DIR))
26ba25
+sys.path.append(os.path.join(SRC_ROOT_DIR, 'scripts'))
26ba25
+
26ba25
+from qemu import QEMUMachine
26ba25
+
26ba25
+def is_readable_executable_file(path):
26ba25
+    return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
26ba25
+
26ba25
+
26ba25
+def pick_default_qemu_bin():
26ba25
+    """
26ba25
+    Picks the path of a QEMU binary, starting either in the current working
26ba25
+    directory or in the source tree root directory.
26ba25
+    """
26ba25
+    arch = os.uname()[4]
26ba25
+    qemu_bin_relative_path = os.path.join("%s-softmmu" % arch,
26ba25
+                                          "qemu-system-%s" % arch)
26ba25
+    if is_readable_executable_file(qemu_bin_relative_path):
26ba25
+        return qemu_bin_relative_path
26ba25
+
26ba25
+    qemu_bin_from_src_dir_path = os.path.join(SRC_ROOT_DIR,
26ba25
+                                              qemu_bin_relative_path)
26ba25
+    if is_readable_executable_file(qemu_bin_from_src_dir_path):
26ba25
+        return qemu_bin_from_src_dir_path
26ba25
+
26ba25
+
26ba25
+class Test(avocado.Test):
26ba25
+    def setUp(self):
26ba25
+        self.vm = None
26ba25
+        self.qemu_bin = self.params.get('qemu_bin',
26ba25
+                                        default=pick_default_qemu_bin())
26ba25
+        if self.qemu_bin is None:
26ba25
+            self.cancel("No QEMU binary defined or found in the source tree")
26ba25
+        self.vm = QEMUMachine(self.qemu_bin)
26ba25
+
26ba25
+    def tearDown(self):
26ba25
+        if self.vm is not None:
26ba25
+            self.vm.shutdown()
26ba25
diff --git a/tests/acceptance/version.py b/tests/acceptance/version.py
26ba25
new file mode 100644
26ba25
index 0000000..13b0a74
26ba25
--- /dev/null
26ba25
+++ b/tests/acceptance/version.py
26ba25
@@ -0,0 +1,24 @@
26ba25
+# Version check example test
26ba25
+#
26ba25
+# Copyright (c) 2018 Red Hat, Inc.
26ba25
+#
26ba25
+# Author:
26ba25
+#  Cleber Rosa <crosa@redhat.com>
26ba25
+#
26ba25
+# This work is licensed under the terms of the GNU GPL, version 2 or
26ba25
+# later.  See the COPYING file in the top-level directory.
26ba25
+
26ba25
+
26ba25
+from avocado_qemu import Test
26ba25
+
26ba25
+
26ba25
+class Version(Test):
26ba25
+    """
26ba25
+    :avocado: enable
26ba25
+    :avocado: tags=quick
26ba25
+    """
26ba25
+    def test_qmp_human_info_version(self):
26ba25
+        self.vm.launch()
26ba25
+        res = self.vm.command('human-monitor-command',
26ba25
+                              command_line='info version')
26ba25
+        self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)')
26ba25
-- 
26ba25
1.8.3.1
26ba25