yeahuh / rpms / qemu-kvm

Forked from rpms/qemu-kvm 2 years ago
Clone

Blame SOURCES/kvm-s390-doc-detailed-specifications-for-AP-virtualizati.patch

26ba25
From 3caa3a2cfbb83be5f52484a0542edc36cfac7b66 Mon Sep 17 00:00:00 2001
26ba25
From: Thomas Huth <thuth@redhat.com>
26ba25
Date: Mon, 15 Oct 2018 10:19:31 +0100
26ba25
Subject: [PATCH 6/6] s390: doc: detailed specifications for AP virtualization
26ba25
26ba25
RH-Author: Thomas Huth <thuth@redhat.com>
26ba25
Message-id: <1539598771-16223-7-git-send-email-thuth@redhat.com>
26ba25
Patchwork-id: 82699
26ba25
O-Subject: [RHEL-8 qemu-kvm PATCH 6/6] s390: doc: detailed specifications for AP virtualization
26ba25
Bugzilla: 1508142
26ba25
RH-Acked-by: David Hildenbrand <david@redhat.com>
26ba25
RH-Acked-by: Cornelia Huck <cohuck@redhat.com>
26ba25
RH-Acked-by: Jens Freimann <jfreimann@redhat.com>
26ba25
26ba25
From: Tony Krowiak <akrowiak@linux.ibm.com>
26ba25
26ba25
This patch provides documentation describing the AP architecture and
26ba25
design concepts behind the virtualization of AP devices. It also
26ba25
includes an example of how to configure AP devices for exclusive
26ba25
use of KVM guests.
26ba25
26ba25
Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
26ba25
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
26ba25
Tested-by: Pierre Morel <pmorel@linux.ibm.com>
26ba25
Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
26ba25
Message-Id: <20181010170309.12045-7-akrowiak@linux.ibm.com>
26ba25
Signed-off-by: Cornelia Huck <cohuck@redhat.com>
26ba25
(cherry picked from commit 694a8d703bfe06226a0574f5ec4af17a2b7060ef)
26ba25
Signed-off-by: Danilo C. L. de Paula <ddepaula@redhat.com>
26ba25
---
26ba25
 MAINTAINERS      |   2 +
26ba25
 docs/vfio-ap.txt | 825 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
26ba25
 2 files changed, 827 insertions(+)
26ba25
 create mode 100644 docs/vfio-ap.txt
26ba25
26ba25
diff --git a/MAINTAINERS b/MAINTAINERS
26ba25
index 99694d8..9b74756 100644
26ba25
--- a/MAINTAINERS
26ba25
+++ b/MAINTAINERS
26ba25
@@ -96,6 +96,7 @@ F: include/hw/watchdog/wdt_diag288.h
26ba25
 F: pc-bios/s390-ccw/
26ba25
 F: pc-bios/s390-ccw.img
26ba25
 F: target/s390x/
26ba25
+F: docs/vfio-ap.txt
26ba25
 K: ^Subject:.*(?i)s390x?
26ba25
 T: git git://github.com/cohuck/qemu.git s390-next
26ba25
 L: qemu-s390x@nongnu.org
26ba25
@@ -1164,6 +1165,7 @@ F: hw/s390x/ap-bridge.c
26ba25
 F: include/hw/s390x/ap-device.h
26ba25
 F: include/hw/s390x/ap-bridge.h
26ba25
 F: hw/vfio/ap.c
26ba25
+F: docs/vfio-ap.txt
26ba25
 L: qemu-s390x@nongnu.org
26ba25
 
26ba25
 vhost
26ba25
diff --git a/docs/vfio-ap.txt b/docs/vfio-ap.txt
26ba25
new file mode 100644
26ba25
index 0000000..1233968
26ba25
--- /dev/null
26ba25
+++ b/docs/vfio-ap.txt
26ba25
@@ -0,0 +1,825 @@
26ba25
+Adjunct Processor (AP) Device
26ba25
+=============================
26ba25
+
26ba25
+Contents:
26ba25
+=========
26ba25
+* Introduction
26ba25
+* AP Architectural Overview
26ba25
+* Start Interpretive Execution (SIE) Instruction
26ba25
+* AP Matrix Configuration on Linux Host
26ba25
+* Starting a Linux Guest Configured with an AP Matrix
26ba25
+* Example: Configure AP Matrices for Three Linux Guests
26ba25
+
26ba25
+Introduction:
26ba25
+============
26ba25
+The IBM Adjunct Processor (AP) Cryptographic Facility is comprised
26ba25
+of three AP instructions and from 1 to 256 PCIe cryptographic adapter cards.
26ba25
+These AP devices provide cryptographic functions to all CPUs assigned to a
26ba25
+linux system running in an IBM Z system LPAR.
26ba25
+
26ba25
+On s390x, AP adapter cards are exposed via the AP bus. This document
26ba25
+describes how those cards may be made available to KVM guests using the
26ba25
+VFIO mediated device framework.
26ba25
+
26ba25
+AP Architectural Overview:
26ba25
+=========================
26ba25
+In order understand the terminology used in the rest of this document, let's
26ba25
+start with some definitions:
26ba25
+
26ba25
+* AP adapter
26ba25
+
26ba25
+  An AP adapter is an IBM Z adapter card that can perform cryptographic
26ba25
+  functions. There can be from 0 to 256 adapters assigned to an LPAR depending
26ba25
+  on the machine model. Adapters assigned to the LPAR in which a linux host is
26ba25
+  running will be available to the linux host. Each adapter is identified by a
26ba25
+  number from 0 to 255; however, the maximum adapter number allowed is
26ba25
+  determined by machine model. When installed, an AP adapter is accessed by
26ba25
+  AP instructions executed by any CPU.
26ba25
+
26ba25
+* AP domain
26ba25
+
26ba25
+  An adapter is partitioned into domains. Each domain can be thought of as
26ba25
+  a set of hardware registers for processing AP instructions. An adapter can
26ba25
+  hold up to 256 domains; however, the maximum domain number allowed is
26ba25
+  determined by machine model. Each domain is identified by a number from 0 to
26ba25
+  255. Domains can be further classified into two types:
26ba25
+
26ba25
+    * Usage domains are domains that can be accessed directly to process AP
26ba25
+      commands
26ba25
+
26ba25
+    * Control domains are domains that are accessed indirectly by AP
26ba25
+      commands sent to a usage domain to control or change the domain; for
26ba25
+      example, to set a secure private key for the domain.
26ba25
+
26ba25
+* AP Queue
26ba25
+
26ba25
+  An AP queue is the means by which an AP command-request message is sent to an
26ba25
+  AP usage domain inside a specific AP. An AP queue is identified by a tuple
26ba25
+  comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
26ba25
+  APQI corresponds to a given usage domain number within the adapter. This tuple
26ba25
+  forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
26ba25
+  instructions include a field containing the APQN to identify the AP queue to
26ba25
+  which the AP command-request message is to be sent for processing.
26ba25
+
26ba25
+* AP Instructions:
26ba25
+
26ba25
+  There are three AP instructions:
26ba25
+
26ba25
+  * NQAP: to enqueue an AP command-request message to a queue
26ba25
+  * DQAP: to dequeue an AP command-reply message from a queue
26ba25
+  * PQAP: to administer the queues
26ba25
+
26ba25
+  AP instructions identify the domain that is targeted to process the AP
26ba25
+  command; this must be one of the usage domains. An AP command may modify a
26ba25
+  domain that is not one of the usage domains, but the modified domain
26ba25
+  must be one of the control domains.
26ba25
+
26ba25
+Start Interpretive Execution (SIE) Instruction
26ba25
+==============================================
26ba25
+A KVM guest is started by executing the Start Interpretive Execution (SIE)
26ba25
+instruction. The SIE state description is a control block that contains the
26ba25
+state information for a KVM guest and is supplied as input to the SIE
26ba25
+instruction. The SIE state description contains a satellite control block called
26ba25
+the Crypto Control Block (CRYCB). The CRYCB contains three fields to identify
26ba25
+the adapters, usage domains and control domains assigned to the KVM guest:
26ba25
+
26ba25
+* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
26ba25
+  to the KVM guest. Each bit in the mask, from left to right, corresponds to
26ba25
+  an APID from 0-255. If a bit is set, the corresponding adapter is valid for
26ba25
+  use by the KVM guest.
26ba25
+
26ba25
+* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
26ba25
+  assigned to the KVM guest. Each bit in the mask, from left to right,
26ba25
+  corresponds to  an AP queue index (APQI) from 0-255. If a bit is set, the
26ba25
+  corresponding queue is valid for use by the KVM guest.
26ba25
+
26ba25
+* The AP Domain Mask field is a bit mask that identifies the AP control domains
26ba25
+  assigned to the KVM guest. The ADM bit mask controls which domains can be
26ba25
+  changed by an AP command-request message sent to a usage domain from the
26ba25
+  guest. Each bit in the mask, from left to right, corresponds to a domain from
26ba25
+  0-255. If a bit is set, the corresponding domain can be modified by an AP
26ba25
+  command-request message sent to a usage domain.
26ba25
+
26ba25
+If you recall from the description of an AP Queue, AP instructions include
26ba25
+an APQN to identify the AP adapter and AP queue to which an AP command-request
26ba25
+message is to be sent (NQAP and PQAP instructions), or from which a
26ba25
+command-reply message is to be received (DQAP instruction). The validity of an
26ba25
+APQN is defined by the matrix calculated from the APM and AQM; it is the
26ba25
+cross product of all assigned adapter numbers (APM) with all assigned queue
26ba25
+indexes (AQM). For example, if adapters 1 and 2 and usage domains 5 and 6 are
26ba25
+assigned to a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for
26ba25
+the guest.
26ba25
+
26ba25
+The APQNs can provide secure key functionality - i.e., a private key is stored
26ba25
+on the adapter card for each of its domains - so each APQN must be assigned to
26ba25
+at most one guest or the linux host.
26ba25
+
26ba25
+   Example 1: Valid configuration:
26ba25
+   ------------------------------
26ba25
+   Guest1: adapters 1,2  domains 5,6
26ba25
+   Guest2: adapter  1,2  domain 7
26ba25
+
26ba25
+   This is valid because both guests have a unique set of APQNs: Guest1 has
26ba25
+   APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7).
26ba25
+
26ba25
+   Example 2: Valid configuration:
26ba25
+   ------------------------------
26ba25
+   Guest1: adapters 1,2 domains 5,6
26ba25
+   Guest2: adapters 3,4 domains 5,6
26ba25
+
26ba25
+   This is also valid because both guests have a unique set of APQNs:
26ba25
+      Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
26ba25
+      Guest2 has APQNs (3,5), (3,6), (4,5), (4,6)
26ba25
+
26ba25
+   Example 3: Invalid configuration:
26ba25
+   --------------------------------
26ba25
+   Guest1: adapters 1,2  domains 5,6
26ba25
+   Guest2: adapter  1    domains 6,7
26ba25
+
26ba25
+   This is an invalid configuration because both guests have access to
26ba25
+   APQN (1,6).
26ba25
+
26ba25
+AP Matrix Configuration on Linux Host:
26ba25
+=====================================
26ba25
+A linux system is a guest of the LPAR in which it is running and has access to
26ba25
+the AP resources configured for the LPAR. The LPAR's AP matrix is
26ba25
+configured via its Activation Profile which can be edited on the HMC. When the
26ba25
+linux system is started, the AP bus will detect the AP devices assigned to the
26ba25
+LPAR and create the following in sysfs:
26ba25
+
26ba25
+/sys/bus/ap
26ba25
+... [devices]
26ba25
+...... xx.yyyy
26ba25
+...... ...
26ba25
+...... cardxx
26ba25
+...... ...
26ba25
+
26ba25
+Where:
26ba25
+    cardxx     is AP adapter number xx (in hex)
26ba25
+....xx.yyyy    is an APQN with xx specifying the APID and yyyy specifying the
26ba25
+               APQI
26ba25
+
26ba25
+For example, if AP adapters 5 and 6 and domains 4, 71 (0x47), 171 (0xab) and
26ba25
+255 (0xff) are configured for the LPAR, the sysfs representation on the linux
26ba25
+host system would look like this:
26ba25
+
26ba25
+/sys/bus/ap
26ba25
+... [devices]
26ba25
+...... 05.0004
26ba25
+...... 05.0047
26ba25
+...... 05.00ab
26ba25
+...... 05.00ff
26ba25
+...... 06.0004
26ba25
+...... 06.0047
26ba25
+...... 06.00ab
26ba25
+...... 06.00ff
26ba25
+...... card05
26ba25
+...... card06
26ba25
+
26ba25
+A set of default device drivers are also created to control each type of AP
26ba25
+device that can be assigned to the LPAR on which a linux host is running:
26ba25
+
26ba25
+/sys/bus/ap
26ba25
+... [drivers]
26ba25
+...... [cex2acard]        for Crypto Express 2/3 accelerator cards
26ba25
+...... [cex2aqueue]       for AP queues served by Crypto Express 2/3
26ba25
+                          accelerator cards
26ba25
+...... [cex4card]         for Crypto Express 4/5/6 accelerator and coprocessor
26ba25
+                          cards
26ba25
+...... [cex4queue]        for AP queues served by Crypto Express 4/5/6
26ba25
+                          accelerator and coprocessor cards
26ba25
+...... [pcixcccard]       for Crypto Express 2/3 coprocessor cards
26ba25
+...... [pcixccqueue]      for AP queues served by Crypto Express 2/3
26ba25
+                          coprocessor cards
26ba25
+
26ba25
+Binding AP devices to device drivers
26ba25
+------------------------------------
26ba25
+There are two sysfs files that specify bitmasks marking a subset of the APQN
26ba25
+range as 'usable by the default AP queue device drivers' or 'not usable by the
26ba25
+default device drivers' and thus available for use by the alternate device
26ba25
+driver(s). The sysfs locations of the masks are:
26ba25
+
26ba25
+   /sys/bus/ap/apmask
26ba25
+   /sys/bus/ap/aqmask
26ba25
+
26ba25
+   The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
26ba25
+   (APID). Each bit in the mask, from left to right (i.e., from most significant
26ba25
+   to least significant bit in big endian order), corresponds to an APID from
26ba25
+   0-255. If a bit is set, the APID is marked as usable only by the default AP
26ba25
+   queue device drivers; otherwise, the APID is usable by the vfio_ap
26ba25
+   device driver.
26ba25
+
26ba25
+   The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
26ba25
+   (APQI). Each bit in the mask, from left to right (i.e., from most significant
26ba25
+   to least significant bit in big endian order), corresponds to an APQI from
26ba25
+   0-255. If a bit is set, the APQI is marked as usable only by the default AP
26ba25
+   queue device drivers; otherwise, the APQI is usable by the vfio_ap device
26ba25
+   driver.
26ba25
+
26ba25
+   Take, for example, the following mask:
26ba25
+
26ba25
+      0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
26ba25
+
26ba25
+    It indicates:
26ba25
+
26ba25
+      1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6
26ba25
+      belong to the vfio_ap device driver's pool.
26ba25
+
26ba25
+   The APQN of each AP queue device assigned to the linux host is checked by the
26ba25
+   AP bus against the set of APQNs derived from the cross product of APIDs
26ba25
+   and APQIs marked as usable only by the default AP queue device drivers. If a
26ba25
+   match is detected,  only the default AP queue device drivers will be probed;
26ba25
+   otherwise, the vfio_ap device driver will be probed.
26ba25
+
26ba25
+   By default, the two masks are set to reserve all APQNs for use by the default
26ba25
+   AP queue device drivers. There are two ways the default masks can be changed:
26ba25
+
26ba25
+   1. The sysfs mask files can be edited by echoing a string into the
26ba25
+      respective sysfs mask file in one of two formats:
26ba25
+
26ba25
+      * An absolute hex string starting with 0x - like "0x12345678" - sets
26ba25
+        the mask. If the given string is shorter than the mask, it is padded
26ba25
+        with 0s on the right; for example, specifying a mask value of 0x41 is
26ba25
+        the same as specifying:
26ba25
+
26ba25
+           0x4100000000000000000000000000000000000000000000000000000000000000
26ba25
+
26ba25
+        Keep in mind that the mask reads from left to right (i.e., most
26ba25
+        significant to least significant bit in big endian order), so the mask
26ba25
+        above identifies device numbers 1 and 7 (01000001).
26ba25
+
26ba25
+        If the string is longer than the mask, the operation is terminated with
26ba25
+        an error (EINVAL).
26ba25
+
26ba25
+      * Individual bits in the mask can be switched on and off by specifying
26ba25
+        each bit number to be switched in a comma separated list. Each bit
26ba25
+        number string must be prepended with a ('+') or minus ('-') to indicate
26ba25
+        the corresponding bit is to be switched on ('+') or off ('-'). Some
26ba25
+        valid values are:
26ba25
+
26ba25
+           "+0"    switches bit 0 on
26ba25
+           "-13"   switches bit 13 off
26ba25
+           "+0x41" switches bit 65 on
26ba25
+           "-0xff" switches bit 255 off
26ba25
+
26ba25
+           The following example:
26ba25
+              +0,-6,+0x47,-0xf0
26ba25
+
26ba25
+              Switches bits 0 and 71 (0x47) on
26ba25
+              Switches bits 6 and 240 (0xf0) off
26ba25
+
26ba25
+        Note that the bits not specified in the list remain as they were before
26ba25
+        the operation.
26ba25
+
26ba25
+   2. The masks can also be changed at boot time via parameters on the kernel
26ba25
+      command line like this:
26ba25
+
26ba25
+         ap.apmask=0xffff ap.aqmask=0x40
26ba25
+
26ba25
+         This would create the following masks:
26ba25
+
26ba25
+            apmask:
26ba25
+            0xffff000000000000000000000000000000000000000000000000000000000000
26ba25
+
26ba25
+            aqmask:
26ba25
+            0x4000000000000000000000000000000000000000000000000000000000000000
26ba25
+
26ba25
+         Resulting in these two pools:
26ba25
+
26ba25
+            default drivers pool:    adapter 0-15, domain 1
26ba25
+            alternate drivers pool:  adapter 16-255, domains 0, 2-255
26ba25
+
26ba25
+Configuring an AP matrix for a linux guest.
26ba25
+------------------------------------------
26ba25
+The sysfs interfaces for configuring an AP matrix for a guest are built on the
26ba25
+VFIO mediated device framework. To configure an AP matrix for a guest, a
26ba25
+mediated matrix device must first be created for the /sys/devices/vfio_ap/matrix
26ba25
+device. When the vfio_ap device driver is loaded, it registers with the VFIO
26ba25
+mediated device framework. When the driver registers, the sysfs interfaces for
26ba25
+creating mediated matrix devices is created:
26ba25
+
26ba25
+/sys/devices
26ba25
+... [vfio_ap]
26ba25
+......[matrix]
26ba25
+......... [mdev_supported_types]
26ba25
+............ [vfio_ap-passthrough]
26ba25
+............... create
26ba25
+............... [devices]
26ba25
+
26ba25
+A mediated AP matrix device is created by writing a UUID to the attribute file
26ba25
+named 'create', for example:
26ba25
+
26ba25
+   uuidgen > create
26ba25
+
26ba25
+   or
26ba25
+
26ba25
+   echo $uuid > create
26ba25
+
26ba25
+When a mediated AP matrix device is created, a sysfs directory named after
26ba25
+the UUID is created in the 'devices' subdirectory:
26ba25
+
26ba25
+/sys/devices
26ba25
+... [vfio_ap]
26ba25
+......[matrix]
26ba25
+......... [mdev_supported_types]
26ba25
+............ [vfio_ap-passthrough]
26ba25
+............... create
26ba25
+............... [devices]
26ba25
+.................. [$uuid]
26ba25
+
26ba25
+There will also be three sets of attribute files created in the mediated
26ba25
+matrix device's sysfs directory to configure an AP matrix for the
26ba25
+KVM guest:
26ba25
+
26ba25
+/sys/devices
26ba25
+... [vfio_ap]
26ba25
+......[matrix]
26ba25
+......... [mdev_supported_types]
26ba25
+............ [vfio_ap-passthrough]
26ba25
+............... create
26ba25
+............... [devices]
26ba25
+.................. [$uuid]
26ba25
+..................... assign_adapter
26ba25
+..................... assign_control_domain
26ba25
+..................... assign_domain
26ba25
+..................... matrix
26ba25
+..................... unassign_adapter
26ba25
+..................... unassign_control_domain
26ba25
+..................... unassign_domain
26ba25
+
26ba25
+assign_adapter
26ba25
+   To assign an AP adapter to the mediated matrix device, its APID is written
26ba25
+   to the 'assign_adapter' file. This may be done multiple times to assign more
26ba25
+   than one adapter. The APID may be specified using conventional semantics
26ba25
+   as a decimal, hexadecimal, or octal number. For example, to assign adapters
26ba25
+   4, 5 and 16 to a mediated matrix device in decimal, hexadecimal and octal
26ba25
+   respectively:
26ba25
+
26ba25
+       echo 4 > assign_adapter
26ba25
+       echo 0x5 > assign_adapter
26ba25
+       echo 020 > assign_adapter
26ba25
+
26ba25
+   In order to successfully assign an adapter:
26ba25
+
26ba25
+   * The adapter number specified must represent a value from 0 up to the
26ba25
+     maximum adapter number allowed by the machine model. If an adapter number
26ba25
+     higher than the maximum is specified, the operation will terminate with
26ba25
+     an error (ENODEV).
26ba25
+
26ba25
+   * All APQNs that can be derived from the adapter ID being assigned and the
26ba25
+     IDs of the previously assigned domains must be bound to the vfio_ap device
26ba25
+     driver. If no domains have yet been assigned, then there must be at least
26ba25
+     one APQN with the specified APID bound to the vfio_ap driver. If no such
26ba25
+     APQNs are bound to the driver, the operation will terminate with an
26ba25
+     error (EADDRNOTAVAIL).
26ba25
+
26ba25
+     No APQN that can be derived from the adapter ID and the IDs of the
26ba25
+     previously assigned domains can be assigned to another mediated matrix
26ba25
+     device. If an APQN is assigned to another mediated matrix device, the
26ba25
+     operation will terminate with an error (EADDRINUSE).
26ba25
+
26ba25
+unassign_adapter
26ba25
+   To unassign an AP adapter, its APID is written to the 'unassign_adapter'
26ba25
+   file. This may also be done multiple times to unassign more than one adapter.
26ba25
+
26ba25
+assign_domain
26ba25
+   To assign a usage domain, the domain number is written into the
26ba25
+   'assign_domain' file. This may be done multiple times to assign more than one
26ba25
+   usage domain. The domain number is specified using conventional semantics as
26ba25
+   a decimal, hexadecimal, or octal number. For example, to assign usage domains
26ba25
+   4, 8, and 71 to a mediated matrix device in decimal, hexadecimal and octal
26ba25
+   respectively:
26ba25
+
26ba25
+      echo 4 > assign_domain
26ba25
+      echo 0x8 > assign_domain
26ba25
+      echo 0107 > assign_domain
26ba25
+
26ba25
+   In order to successfully assign a domain:
26ba25
+
26ba25
+   * The domain number specified must represent a value from 0 up to the
26ba25
+     maximum domain number allowed by the machine model. If a domain number
26ba25
+     higher than the maximum is specified, the operation will terminate with
26ba25
+     an error (ENODEV).
26ba25
+
26ba25
+   * All APQNs that can be derived from the domain ID being assigned and the IDs
26ba25
+     of the previously assigned adapters must be bound to the vfio_ap device
26ba25
+     driver. If no domains have yet been assigned, then there must be at least
26ba25
+     one APQN with the specified APQI bound to the vfio_ap driver. If no such
26ba25
+     APQNs are bound to the driver, the operation will terminate with an
26ba25
+     error (EADDRNOTAVAIL).
26ba25
+
26ba25
+     No APQN that can be derived from the domain ID being assigned and the IDs
26ba25
+     of the previously assigned adapters can be assigned to another mediated
26ba25
+     matrix device. If an APQN is assigned to another mediated matrix device,
26ba25
+     the operation will terminate with an error (EADDRINUSE).
26ba25
+
26ba25
+unassign_domain
26ba25
+   To unassign a usage domain, the domain number is written into the
26ba25
+   'unassign_domain' file. This may be done multiple times to unassign more than
26ba25
+   one usage domain.
26ba25
+
26ba25
+assign_control_domain
26ba25
+   To assign a control domain, the domain number is written into the
26ba25
+   'assign_control_domain' file. This may be done multiple times to
26ba25
+   assign more than one control domain. The domain number may be specified using
26ba25
+   conventional semantics as a decimal, hexadecimal, or octal number. For
26ba25
+   example, to assign  control domains 4, 8, and 71 to  a mediated matrix device
26ba25
+   in decimal, hexadecimal and octal respectively:
26ba25
+
26ba25
+      echo 4 > assign_domain
26ba25
+      echo 0x8 > assign_domain
26ba25
+      echo 0107 > assign_domain
26ba25
+
26ba25
+   In order to successfully assign a control domain, the domain number
26ba25
+   specified must represent a value from 0 up to the maximum domain number
26ba25
+   allowed by the machine model. If a control domain number higher than the
26ba25
+   maximum is specified, the operation will terminate with an error (ENODEV).
26ba25
+
26ba25
+unassign_control_domain
26ba25
+   To unassign a control domain, the domain number is written into the
26ba25
+   'unassign_domain' file. This may be done multiple times to unassign more than
26ba25
+   one control domain.
26ba25
+
26ba25
+Notes: Hot plug/unplug is not currently supported for mediated AP matrix
26ba25
+devices, so no changes to the AP matrix will be allowed while a guest using
26ba25
+the mediated matrix device is running. Attempts to assign an adapter,
26ba25
+domain or control domain will be rejected and an error (EBUSY) returned.
26ba25
+
26ba25
+Starting a Linux Guest Configured with an AP Matrix:
26ba25
+===================================================
26ba25
+To provide a mediated matrix device for use by a guest, the following option
26ba25
+must be specified on the QEMU command line:
26ba25
+
26ba25
+   -device vfio_ap,sysfsdev=$path-to-mdev
26ba25
+
26ba25
+The sysfsdev parameter specifies the path to the mediated matrix device.
26ba25
+There are a number of ways to specify this path:
26ba25
+
26ba25
+/sys/devices/vfio_ap/matrix/$uuid
26ba25
+/sys/bus/mdev/devices/$uuid
26ba25
+/sys/bus/mdev/drivers/vfio_mdev/$uuid
26ba25
+/sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
26ba25
+
26ba25
+When the linux guest is started, the guest will open the mediated
26ba25
+matrix device's file descriptor to get information about the mediated matrix
26ba25
+device. The vfio_ap device driver will update the APM, AQM, and ADM fields in
26ba25
+the guest's CRYCB with the adapter, usage domain and control domains assigned
26ba25
+via the mediated matrix device's sysfs attribute files. Programs running on the
26ba25
+linux guest will then:
26ba25
+
26ba25
+1. Have direct access to the APQNs derived from the cross product of the AP
26ba25
+   adapter numbers (APID) and queue indexes (APQI) specified in the APM and AQM
26ba25
+   fields of the guests's CRYCB respectively. These APQNs identify the AP queues
26ba25
+   that are valid for use by the guest; meaning, AP commands can be sent by the
26ba25
+   guest to any of these queues for processing.
26ba25
+
26ba25
+2. Have authorization to process AP commands to change a control domain
26ba25
+   identified in the ADM field of the guest's CRYCB. The AP command must be sent
26ba25
+   to a valid APQN (see 1 above).
26ba25
+
26ba25
+CPU model features:
26ba25
+
26ba25
+Three CPU model features are available for controlling guest access to AP
26ba25
+facilities:
26ba25
+
26ba25
+1. AP facilities feature
26ba25
+
26ba25
+   The AP facilities feature indicates that AP facilities are installed on the
26ba25
+   guest. This feature will be exposed for use only if the AP facilities
26ba25
+   are installed on the host system. The feature is s390-specific and is
26ba25
+   represented as a parameter of the -cpu option on the QEMU command line:
26ba25
+
26ba25
+      qemu-system-s390x -cpu $model,ap=on|off
26ba25
+
26ba25
+      Where:
26ba25
+
26ba25
+         $model is the CPU model defined for the guest (defaults to the model of
26ba25
+                the host system if not specified).
26ba25
+
26ba25
+         ap=on|off indicates whether AP facilities are installed (on) or not
26ba25
+                   (off). The default for CPU models zEC12 or newer
26ba25
+                   is ap=on. AP facilities must be installed on the guest if a
26ba25
+                   vfio-ap device (-device vfio-ap,sysfsdev=$path) is configured
26ba25
+                   for the guest, or the guest will fail to start.
26ba25
+
26ba25
+2. Query Configuration Information (QCI) facility
26ba25
+
26ba25
+   The QCI facility is used by the AP bus running on the guest to query the
26ba25
+   configuration of the AP facilities. This facility will be available
26ba25
+   only if the QCI facility is installed on the host system. The feature is
26ba25
+   s390-specific and is represented as a parameter of the -cpu option on the
26ba25
+   QEMU command line:
26ba25
+
26ba25
+      qemu-system-s390x -cpu $model,apqci=on|off
26ba25
+
26ba25
+      Where:
26ba25
+
26ba25
+         $model is the CPU model defined for the guest
26ba25
+
26ba25
+         apqci=on|off indicates whether the QCI facility is installed (on) or
26ba25
+                      not (off). The default for CPU models zEC12 or newer
26ba25
+                      is apqci=on; for older models, QCI will not be installed.
26ba25
+
26ba25
+                      If QCI is installed (apqci=on) but AP facilities are not
26ba25
+                      (ap=off), an error message will be logged, but the guest
26ba25
+                      will be allowed to start. It makes no sense to have QCI
26ba25
+                      installed if the AP facilities are not; this is considered
26ba25
+                      an invalid configuration.
26ba25
+
26ba25
+                      If the QCI facility is not installed, APQNs with an APQI
26ba25
+                      greater than 15 will not be detected by the AP bus
26ba25
+                      running on the guest.
26ba25
+
26ba25
+3. Adjunct Process Facility Test (APFT) facility
26ba25
+
26ba25
+   The APFT facility is used by the AP bus running on the guest to test the
26ba25
+   AP facilities available for a given AP queue. This facility will be available
26ba25
+   only if the APFT facility is installed on the host system. The feature is
26ba25
+   s390-specific and is represented as a parameter of the -cpu option on the
26ba25
+   QEMU command line:
26ba25
+
26ba25
+      qemu-system-s390x -cpu $model,apft=on|off
26ba25
+
26ba25
+      Where:
26ba25
+
26ba25
+         $model is the CPU model defined for the guest (defaults to the model of
26ba25
+                the host system if not specified).
26ba25
+
26ba25
+         apft=on|off indicates whether the APFT facility is installed (on) or
26ba25
+                     not (off). The default for CPU models zEC12 and
26ba25
+                     newer is apft=on for older models, APFT will not be
26ba25
+                     installed.
26ba25
+
26ba25
+                     If APFT is installed (apft=on) but AP facilities are not
26ba25
+                     (ap=off), an error message will be logged, but the guest
26ba25
+                     will be allowed to start. It makes no sense to have APFT
26ba25
+                     installed if the AP facilities are not; this is considered
26ba25
+                     an invalid configuration.
26ba25
+
26ba25
+                     It also makes no sense to turn APFT off because the AP bus
26ba25
+                     running on the guest will not detect CEX4 and newer devices
26ba25
+                     without it. Since only CEX4 and newer devices are supported
26ba25
+                     for guest usage, no AP devices can be made accessible to a
26ba25
+                     guest started without APFT installed.
26ba25
+
26ba25
+Example: Configure AP Matrixes for Three Linux Guests:
26ba25
+=====================================================
26ba25
+Let's now provide an example to illustrate how KVM guests may be given
26ba25
+access to AP facilities. For this example, we will show how to configure
26ba25
+three guests such that executing the lszcrypt command on the guests would
26ba25
+look like this:
26ba25
+
26ba25
+Guest1
26ba25
+------
26ba25
+CARD.DOMAIN TYPE  MODE
26ba25
+------------------------------
26ba25
+05          CEX5C CCA-Coproc
26ba25
+05.0004     CEX5C CCA-Coproc
26ba25
+05.00ab     CEX5C CCA-Coproc
26ba25
+06          CEX5A Accelerator
26ba25
+06.0004     CEX5A Accelerator
26ba25
+06.00ab     CEX5C CCA-Coproc
26ba25
+
26ba25
+Guest2
26ba25
+------
26ba25
+CARD.DOMAIN TYPE  MODE
26ba25
+------------------------------
26ba25
+05          CEX5A Accelerator
26ba25
+05.0047     CEX5A Accelerator
26ba25
+05.00ff     CEX5A Accelerator (5,4), (5,171), (6,4), (6,171),
26ba25
+
26ba25
+Guest3
26ba25
+------
26ba25
+CARD.DOMAIN TYPE  MODE
26ba25
+------------------------------
26ba25
+06          CEX5A Accelerator
26ba25
+06.0047     CEX5A Accelerator
26ba25
+06.00ff     CEX5A Accelerator
26ba25
+
26ba25
+These are the steps:
26ba25
+
26ba25
+1. Install the vfio_ap module on the linux host. The dependency chain for the
26ba25
+   vfio_ap module is:
26ba25
+   * iommu
26ba25
+   * s390
26ba25
+   * zcrypt
26ba25
+   * vfio
26ba25
+   * vfio_mdev
26ba25
+   * vfio_mdev_device
26ba25
+   * KVM
26ba25
+
26ba25
+   To build the vfio_ap module, the kernel build must be configured with the
26ba25
+   following Kconfig elements selected:
26ba25
+   * IOMMU_SUPPORT
26ba25
+   * S390
26ba25
+   * ZCRYPT
26ba25
+   * S390_AP_IOMMU
26ba25
+   * VFIO
26ba25
+   * VFIO_MDEV
26ba25
+   * VFIO_MDEV_DEVICE
26ba25
+   * KVM
26ba25
+
26ba25
+   If using make menuconfig select the following to build the vfio_ap module:
26ba25
+   -> Device Drivers
26ba25
+      -> IOMMU Hardware Support
26ba25
+         select S390 AP IOMMU Support
26ba25
+      -> VFIO Non-Privileged userspace driver framework
26ba25
+         -> Mediated device driver frramework
26ba25
+            -> VFIO driver for Mediated devices
26ba25
+   -> I/O subsystem
26ba25
+      -> VFIO support for AP devices
26ba25
+
26ba25
+2. Secure the AP queues to be used by the three guests so that the host can not
26ba25
+   access them. To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff,
26ba25
+   06.0004, 06.0047, 06.00ab, and 06.00ff for use by the vfio_ap device driver,
26ba25
+   the corresponding APQNs must be removed from the default queue drivers pool
26ba25
+   as follows:
26ba25
+
26ba25
+      echo -5,-6 > /sys/bus/ap/apmask
26ba25
+
26ba25
+      echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
26ba25
+
26ba25
+   This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,
26ba25
+   06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The
26ba25
+   sysfs directory for the vfio_ap device driver will now contain symbolic links
26ba25
+   to the AP queue devices bound to it:
26ba25
+
26ba25
+   /sys/bus/ap
26ba25
+   ... [drivers]
26ba25
+   ...... [vfio_ap]
26ba25
+   ......... [05.0004]
26ba25
+   ......... [05.0047]
26ba25
+   ......... [05.00ab]
26ba25
+   ......... [05.00ff]
26ba25
+   ......... [06.0004]
26ba25
+   ......... [06.0047]
26ba25
+   ......... [06.00ab]
26ba25
+   ......... [06.00ff]
26ba25
+
26ba25
+   Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later)
26ba25
+   can be bound to the vfio_ap device driver. The reason for this is to
26ba25
+   simplify the implementation by not needlessly complicating the design by
26ba25
+   supporting older devices that will go out of service in the relatively near
26ba25
+   future, and for which there are few older systems on which to test.
26ba25
+
26ba25
+   The administrator, therefore, must take care to secure only AP queues that
26ba25
+   can be bound to the vfio_ap device driver. The device type for a given AP
26ba25
+   queue device can be read from the parent card's sysfs directory. For example,
26ba25
+   to see the hardware type of the queue 05.0004:
26ba25
+
26ba25
+   cat /sys/bus/ap/devices/card05/hwtype
26ba25
+
26ba25
+   The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the
26ba25
+   vfio_ap device driver.
26ba25
+
26ba25
+3. Create the mediated devices needed to configure the AP matrixes for the
26ba25
+   three guests and to provide an interface to the vfio_ap driver for
26ba25
+   use by the guests:
26ba25
+
26ba25
+   /sys/devices/vfio_ap/matrix/
26ba25
+   --- [mdev_supported_types]
26ba25
+   ------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
26ba25
+   --------- create
26ba25
+   --------- [devices]
26ba25
+
26ba25
+   To create the mediated devices for the three guests:
26ba25
+
26ba25
+       uuidgen > create
26ba25
+       uuidgen > create
26ba25
+       uuidgen > create
26ba25
+
26ba25
+        or
26ba25
+
26ba25
+        echo $uuid1 > create
26ba25
+        echo $uuid2 > create
26ba25
+        echo $uuid3 > create
26ba25
+
26ba25
+   This will create three mediated devices in the [devices] subdirectory named
26ba25
+   after the UUID used to create the mediated device. We'll call them $uuid1,
26ba25
+   $uuid2 and $uuid3 and this is the sysfs directory structure after creation:
26ba25
+
26ba25
+   /sys/devices/vfio_ap/matrix/
26ba25
+   --- [mdev_supported_types]
26ba25
+   ------ [vfio_ap-passthrough]
26ba25
+   --------- [devices]
26ba25
+   ------------ [$uuid1]
26ba25
+   --------------- assign_adapter
26ba25
+   --------------- assign_control_domain
26ba25
+   --------------- assign_domain
26ba25
+   --------------- matrix
26ba25
+   --------------- unassign_adapter
26ba25
+   --------------- unassign_control_domain
26ba25
+   --------------- unassign_domain
26ba25
+
26ba25
+   ------------ [$uuid2]
26ba25
+   --------------- assign_adapter
26ba25
+   --------------- assign_control_domain
26ba25
+   --------------- assign_domain
26ba25
+   --------------- matrix
26ba25
+   --------------- unassign_adapter
26ba25
+   ----------------unassign_control_domain
26ba25
+   ----------------unassign_domain
26ba25
+
26ba25
+   ------------ [$uuid3]
26ba25
+   --------------- assign_adapter
26ba25
+   --------------- assign_control_domain
26ba25
+   --------------- assign_domain
26ba25
+   --------------- matrix
26ba25
+   --------------- unassign_adapter
26ba25
+   ----------------unassign_control_domain
26ba25
+   ----------------unassign_domain
26ba25
+
26ba25
+4. The administrator now needs to configure the matrixes for the mediated
26ba25
+   devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).
26ba25
+
26ba25
+   This is how the matrix is configured for Guest1:
26ba25
+
26ba25
+      echo 5 > assign_adapter
26ba25
+      echo 6 > assign_adapter
26ba25
+      echo 4 > assign_domain
26ba25
+      echo 0xab > assign_domain
26ba25
+
26ba25
+      Control domains can similarly be assigned using the assign_control_domain
26ba25
+      sysfs file.
26ba25
+
26ba25
+      If a mistake is made configuring an adapter, domain or control domain,
26ba25
+      you can use the unassign_xxx interfaces to unassign the adapter, domain or
26ba25
+      control domain.
26ba25
+
26ba25
+      To display the matrix configuration for Guest1:
26ba25
+
26ba25
+         cat matrix
26ba25
+
26ba25
+         The output will display the APQNs in the format xx.yyyy, where xx is
26ba25
+         the adapter number and yyyy is the domain number. The output for Guest1
26ba25
+         will look like this:
26ba25
+
26ba25
+         05.0004
26ba25
+         05.00ab
26ba25
+         06.0004
26ba25
+         06.00ab
26ba25
+
26ba25
+   This is how the matrix is configured for Guest2:
26ba25
+
26ba25
+      echo 5 > assign_adapter
26ba25
+      echo 0x47 > assign_domain
26ba25
+      echo 0xff > assign_domain
26ba25
+
26ba25
+   This is how the matrix is configured for Guest3:
26ba25
+
26ba25
+      echo 6 > assign_adapter
26ba25
+      echo 0x47 > assign_domain
26ba25
+      echo 0xff > assign_domain
26ba25
+
26ba25
+5. Start Guest1:
26ba25
+
26ba25
+   /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
26ba25
+      -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
26ba25
+
26ba25
+7. Start Guest2:
26ba25
+
26ba25
+   /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
26ba25
+      -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
26ba25
+
26ba25
+7. Start Guest3:
26ba25
+
26ba25
+   /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
26ba25
+      -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...
26ba25
+
26ba25
+When the guest is shut down, the mediated matrix devices may be removed.
26ba25
+
26ba25
+Using our example again, to remove the mediated matrix device $uuid1:
26ba25
+
26ba25
+   /sys/devices/vfio_ap/matrix/
26ba25
+      --- [mdev_supported_types]
26ba25
+      ------ [vfio_ap-passthrough]
26ba25
+      --------- [devices]
26ba25
+      ------------ [$uuid1]
26ba25
+      --------------- remove
26ba25
+
26ba25
+
26ba25
+   echo 1 > remove
26ba25
+
26ba25
+   This will remove all of the mdev matrix device's sysfs structures including
26ba25
+   the mdev device itself. To recreate and reconfigure the mdev matrix device,
26ba25
+   all of the steps starting with step 3 will have to be performed again. Note
26ba25
+   that the remove will fail if a guest using the mdev is still running.
26ba25
+
26ba25
+   It is not necessary to remove an mdev matrix device, but one may want to
26ba25
+   remove it if no guest will use it during the remaining lifetime of the linux
26ba25
+   host. If the mdev matrix device is removed, one may want to also reconfigure
26ba25
+   the pool of adapters and queues reserved for use by the default drivers.
26ba25
+
26ba25
+Limitations
26ba25
+===========
26ba25
+* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN
26ba25
+  to the default drivers pool of a queue that is still assigned to a mediated
26ba25
+  device in use by a guest. It is incumbent upon the administrator to
26ba25
+  ensure there is no mediated device in use by a guest to which the APQN is
26ba25
+  assigned lest the host be given access to the private data of the AP queue
26ba25
+  device, such as a private key configured specifically for the guest.
26ba25
+
26ba25
+* Dynamically modifying the AP matrix for a running guest (which would amount to
26ba25
+  hot(un)plug of AP devices for the guest) is currently not supported
26ba25
+
26ba25
+* Live guest migration is not supported for guests using AP devices.
26ba25
-- 
26ba25
1.8.3.1
26ba25