Blob Blame History Raw
The microcode_ctl package shipped with RHEL contains provisions for issues with
microcode loading on the old kernels.  While those provisions are expected
to suit most users, several knobs are provided in order to provide ability
to override the default behaviour.


General behaviour
=================
In RHEL 8 (as well as RHEL 7 before it), there are currently two main handlers
for CPU microcode update:
 * Early microcode update. It uses GenuineIntel.bin or AuthenticAMD.bin file
   placed at the beginning of an initramfs image
   (/boot/initramfs-KERNEL_VERSION.img, where "KERNEL_VERSION" is a kernel
   version in the same format as provided by "uname -r") as a source
   of microcode data, and is performed very early during the boot process
   (if the relevant microcode file is available in the aforementioned file).
 * On-demand (late) microcode update. It can be triggered by writing "1" to
   /sys/devices/system/cpu/microcode/reload file (provided my the "microcode"
   module). It relies on request_firmware infrastructure, which searches (and
   loads, if found) microcode from a file present in one of the following
   directories (in the search order):
       /lib/firmware/updates/KERNEL_VERSION/
       /lib/firmware/updates/
       /lib/firmware/KERNEL_VERSION/
       /lib/firmware/
  (there is also an additional directory that can be configured via the
  "fw_path_para" module option of the "firmware_class" module; as this module
  is built-in in RHEL kernel, a boot parameter "firmware_class.fw_path_para"
  should be used for that purpose; this is out of the document's scope, however)

The firmware for Intel CPUs is searched in "intel-ucode" subdirectory, and for
AMD CPUs, a file under "amd-ucode" is searched.

For Intel CPUs, the name of the specific microcode file the kernel tries to load
has the format "FF-MM-SS", where "FF" is the family number, "MM" is the model
number, and "SS" is the stepping. All those numbers are zero-filled to two digits
and are written in hexadecimal (letters are in the lower case).  For AMD CPUs,
the file name has the format "microcode_amd_famFFh.bin", where "FF" is the
family number, written in hexadecimal, letters are in the lower case, not
zero-filled.

The early microcode is placed into initramfs image by the "dracut" script, which
scans the aforementioned subdirectories of the configured list of firmware
directories (by default, the list consists of two directories in RHEL 8,
"/lib/firmware/updates" and "/lib/firmware").

In RHEL 8, AMD CPU microcode is shipped as a part of the linux-firmware package,
and Intel microcode is shipped as a part of the microcode_ctl package.

The microcode_ctl package currently includes the following:
 * Intel CPU microcode files, placed in /usr/share/microcode_ctl/intel-ucode
   directory (currently there are none);
 * A dracut module, /usr/lib/dracut/modules.d/99microcode_ctl-fw_dir_override,
   that controls which additional firmware directories will be added to dracut's
   default configuration;
 * A dracut configuration file, /usr/lib/dracut/dracut.conf.d/01-microcode.conf,
   that enables inclusion of early microcode to the generated initramfs
   in dracut;
 * A dracut configuration file,
   /usr/lib/dracut/dracut.conf.d/99-microcode-override.conf, that provides a way
   to quickly disable 99microcode_ctl-fw_dir-override dracut module;
 * A systemd service file, microcode.service, that triggers microcode reload
   late during boot;
 * A set of directories in /usr/share/microcode_ctl/ucode_with_caveats, each
   of which contains configuration and related data for various caveats related
   to microcode:
   * readme - description of caveat and related information,
   * config - caveat configuration file, with syntax as described in "Caveat
     configuration" section below,
   * intel-ucode - directory containing microcode files related to the caveat;
 * A set of support scripts, placed in /usr/libexec/microcode_ctl:
   * "check_caveats" is an utility script that performs checks of the target
     kernel (and running CPU) in accordance with caveat configuration files
     in ucode_with_caveats directory and reports whether it passes them or not,
   * "reload_microcode" is a script that is called by microcode.service and
     triggers microcode reloading (by writing "1" to
     /sys/devices/system/cpu/microcode/reload) if the running kernel passes
     check_caveats checks,
   * "update_ucode" is a script that populates symlinks to microcode files
     in /lib/firmware, so it can be picked up by relevant kernels for the late
     microcode loading.

Also, microcode_ctl RPM includes triggers that run update_ucode script on every
installation or removal of a kernel RPM in order to provide microcode files
for newly installed kernels and cleanup symlinks for the uninstalled ones.


Caveat configuration
--------------------
There is a directory for each caveat under
/usr/share/microcode_ctl/ucode_with_caveats, containing the following files:
 * "config", a configuration file for the caveat;
 * "readme", that contains description of the caveat;
 * set of related associated microcode files.

"config" file is a set of lines each containing option name and its value,
separated by white space.  Currently, the following options are supported:
 * "model" option, which has format "VENDOR_ID FF-MM-SS", that specifies
   to which CPU model the caveat is applicable (check_caveats ignores caveats
   with non-matching models if "-m" option is passed to it). Can be set
   in the configuration file only once (the last provided value is used).
 * "vendor" option specifies CPUs of which vendor (as provided
   in the /proc/cpuinfo file) the caveat is applicable to (check_caveats
   ignores caveats with non-matching models when it is invoked with "-m"
   option). Can be set in the configuration file only once.
 * "path" is a glob pattern that specifies set of microcode files associated
   with the caveat as a relative path to the caveat directory. This option
   is used for populating files in /lib/firmware by update_ucode script and
   for matching microcode file when dracut is run in host-only mode
   (as in that case it uses only the first directory in firmware directory list
   to look for the microcode file applicable to the host CPU).  Can be set
   in the configuration file multiple times.
 * "kernel" is a minimal kernel version that supports proper handling
   of the related microcode files during late microcode load.  It may be
   provided in one of the following formats that affect the way it is compared
   to the running kernel version:
    * A.B.C (where A, B, and C are decimal numbers), "upstream version". In this
      case, simple version comparison against the respective part of the running
      kernel version is used, and the running kernel version should be greater
      or equal than the version provided in the configuration option in order
      for comparison to succeed (that is, the first part, major version number,
      of the running kernel version should be greater than the value provided
      in the configuration option, or those should be equal and the second part,
      minor version number, should be greater than the minor version number
      of the kernel version provided in the configuration option, or the first
      two parts should be equal and the third part, patch level, should
      be greater or equal the patch level of the version in the configuration
      option).
    * A.B.C-Y (where A, B, C, and Y are decimal numbers), "Y-stream version".
      In this case, A.B.C part should be equal, and Y part of the running kernel
      version should be greater or equal than the Y part of the configuration
      option version in order to satisfy the comparison requirement.
    * A.B.C-Y.Z1.Z2 (where A, B, C, Y, Z1, and Z2 are decimal numbers),
      "Z-stream version". In this case, A.B.C-Y part should be equal and Z1.Z2
      part of the running kernel should be greater or equal than the respective
      part of the configuration option version (when compared as a version)
      for comparison to succeed.
   Kernel version check passed if at least one comparison of the running kernel
   version against a kernel version provided in a configuration option
   succeeded.  The "kernel" configuration option can be provided
   in the configuration file multiple times.
 * "kernel_early" is a minimal kernel version that supports proper handling
   of the related microcode during early microcode load. The format of the
   option and its semantics is similar to the "kernel" configuration options.
   This option can be provided multiple times as well.
 * "mc_min_ver_late" is the minimal version of the currently loaded microcode
   on the CPU (as reported in /proc/cpuinfo) that supports late microcode
   update.  Microcode update will be attempted only if the currently loaded
   microcode version is greater or equal the microcode version provided
   in the configuration option. Can be set in the configuration file only once.
 * "disable" is a way to disable a specific caveat from inside its
   configuration. Argument for the argument is a list of stages ("early",
   "late") for which the caveat should be disable. The configuration option
   can be provided multiple times in a configuration file.
 * "blacklist" is a marker for a start of list of blacklisted model names,
   one model name per line. The model name of the running CPU (as reported
   in /proc/cpuinfo) is compared against the names in the provided list, and,
   if there is a match, caveat check fails.


check_caveats script
--------------------
"check_caveats" is an utility script (called by update_ucode, reload_microcode,
dracut module) that performs checks of the target kernel (and running CPU)
in accordance with caveat configuration files in directory
"/usr/share/microcode_ctl/ucode_with_caveats", and returns information, whether
the system passes the checks, or not.

Usage:
    check_caveats [-e] [-k TARGET_KVER] [-c CONFIG]* [-m] [-v]'

Options:
  -e - check for early microcode load possibility (instead of late microcode
       load). "kernel_early" caveat configuration options are used for checking
       instead of "kernel", and "mc_min_ver_late" is not checked.
  -k - target kernel version to check against, $(uname -r) is used otherwise.
  -c - caveat(s) to check, all caveat configurations found inside
       $MC_CAVEATS_DATA_DIR are checked otherwise.
  -m - ignore caveats that do not apply to the current CPU model.
  -v - verbose output.

Environment:
  MC_CAVEATS_DATA_DIR - directory that contains caveats configurations,
                        "/usr/share/microcode_ctl/ucode_with_caveats"
			by default.
  FW_DIR - directory containing firmware files (per-kernel configuration
           overrides are checked there), "/lib/firmware" by default.
  CFG_DIR - directory containing global caveats overrides,
            "/etc/microcode_ctl/ucode_with_caveats" by default.

Output:
  Script returns information about caveats check results. Output has a format
  of "KEY VALUE1 VALUE2 ..." with KEY defining the semantics of the VALUEs.
  Currently, the following data is issued:
   - "cfgs" - list of caveats that have been processed (and not skipped
      due to missing "config", "readme", or a disallow-* override described
      below);
   - "skip_cfgs" - list of caveats that have been skipped (due to missing
     config/readme file, or because of overrides);
   - "paths" - list of glob patterns matching files associated with caveats
     that have been processed;
   - "ok_cfgs" - list of caveat configurations that have all the checks passed
     (or have enforced by one of force-* overrides described below);
   - "ok_paths" - list of glob patterns associated with caveat files from
     the "ok_cfgs" list;
   - "fail_cfgs" - list of caveats that have one of the checks failed.
   - "fail_paths" - list of glob patterns associated with caveats from the
     "fail_cfgs" list.

Return value:
  - 0 in case caveats check has passed, 1 otherwise.
  - In "-d" mode, 0 is always returned.

Overrides:

When check_caveats perform its checks, it also checks for presence of files
in specific places, and, if they exist, check_caveats skips a caveat or ignores
its checks; that mechanism allows overriding the information provided
in configuration on local systems and affect the behaviour of the microcode
update process.

Current list of overrides (where $FW_DIR and $CFG_DIR are the environment
options described earlier; $kver - the currently processed kernel version,
$s is the requested stage ("early" or "late"), $cfg is the caveat directory
name):
    $FW_DIR/$kver/disallow-$s-$cfg - skip a caveat for the requested stage for
                                     a specific kernel version..
    $FW_DIR/$kver/force-$s-$cfg - apply a specific caveat file for a specific
                                  kernel version for the requested stage without
				  performing any checks.
    $FW_DIR/$kver/disallow-$cfg - skip a caveat for any stage for a specific
                                  kernel version.
    $FW_DIR/$kver/force-$cfg - apply a specific caveat for any stage
                               for a specific kernel version without checks.
    $FW_DIR/$kver/disallow-$s - skip all caveats for a specific stage
                                for a specific kernel version.
    $CFG_DIR/disallow-$s-$cfg - skip a caveat for a specific stage for all
                                kernel versions.
    $FW_DIR/$kver/force-$s - apply all caveats for a specific stage
                             for a specific kernel version without checks.
    $CFG_DIR/force-$s-$cfg - apply a specific caveat for a specific stage for
                             all kernel versions without checks.
    $FW_DIR/$kver/disallow - skip all caveats for all stages for a specific
                             kernel version.
    $CFG_DIR/disallow-$cfg - skip a caveat for all stages for all kernel
                             versions.
    $FW_DIR/$kver/force - apply all caveats for all stages for a specific kernel
                          version without checks.
    $CFG_DIR/force-$cfg - apply a caveat for all stages for all kernel versions
                          without checks.
    $CFG_DIR/disallow-$s - skip all caveat for all kernel versions
                           for a specific stage.
    $CFG_DIR/force-$s - apply all caveats for all kernel versions for  specific
                        stage without checks.
    $CFG_DIR/disallow - skip all caveats for all stages for all kernel versions
                        (disable everything).
    $CFG_DIR/force - force all caveats for all stages for all kernel versions
                     (enable everything).

The "apply" action above means creating symlinks in /lib/firmware by
update_ucode in case of the "late" stage and adding caveat directory to the list
of firmware directories by dracut plugin in case of the "early" stage.

The files are checked for existence until the first match, so more specific
overrides can override more broad ones.

Also, a caveat is ignored if it lacks either config or readme file.


update_ucode script
-------------------
"update_ucode" populates symlinks to microcode files in accordance with caveats
configuration.  It enables late microcode loading that is invoked by triggering
/sys/devices/system/cpu/microcode/reload file.  Since caveats depend
on the kernel version, symlinks are populated inside
"/lib/firmware/KERNEL_VERSION" directory for each installed kernel.
As a consequence, this script is triggered upon each kernel package installation
and removal.

The script has two parts: common and kernel-version-specific.

During the common part, files are populated from
/usr/share/microcode_ctl/intel-ucode in /lib/firmware/intel-ucode. There are
several possibilities to affect the process:
 * Presence of "/etc/microcode_ctl/intel-ucode-disallow" file leads to skipping
   the common part of the script.
 * The same for "/lib/firmware/intel-ucode-disallow".

During the kernel-version-specific part, each caveat is checked against every
kernel version, and those combinations, for which caveat check succeeds,
gets the symlinks to the associated microcode files populated.
 * Absence of "/lib/firmware/KERNEL_VERSION/readme-CAVEAT" prevents update_ucode
   from removing symlinks related to the caveat for specific kernel version.
 * Since the check is being done by check_caveats, all the overrides that
   described there also stay.

Usage:
    update_ucode [--action {add|remove|refresh|list}] [--kernel KERNELVER]*
                 [--verbose] [--dry-run] [--cleanup intel_ucode caveats_ucode]
                 [--skip-common] [--skip-kernel-specific]

Options:
  --action - action to perform. Currently, the following actions are supported:
              * "add" - create new symlinks.
              * "remove" - remove old symlinks that are no longer needed.
              * "refresh" - re-populate symlinks.
              * "list" - list files under control of update_ucode.
             By default, "refresh" action is executed.
  --kernel - kernel version to process. By default, list of kernel versions
             is formed based on contents of /lib/firmware and /lib/modules
             directories.
  --verbose - verbose output.
  --dry-run - do not call commands, just print the invocation lines.
  --cleanup - cleanup mode. Used by post-uninstall script during package
              upgrades. Removes excess files in accordance to the contents
              of the files provided in the arguments to the option.
  --skip-common - do not process /lib/firmware directory.
  --skip-kernel-specific - do not process /lib/firmware/KERNEL_VERSION
                           directories.

Return value:
  0 on success, 1 on error.


reload_microcode script
-----------------------
"reload_microcode" is a script that is called by microcode.service and
triggers late microcode reloading (by writing "1" to
/sys/devices/system/cpu/microcode/reload) if the following check are passed:
 * the microcode update performed not in a virtualised environment;
 * running kernel passes "check_caveats" checks that applicable to the current
   CPU model.

For a virtualised environment check, the script searches the "/proc/cpuinfo"
file for presence of the "hypervisor" flag among CPU features (it corresponds
to a CPUID feature bit set by hypervisors in order to inform that the kernel
operates inside a virtual machine).  This check can be overridden and skipped
by creation of a file "/etc/microcode_ctl/ignore-hypervisor-flag".

The script has no options and always returns 0.


99microcode_ctl-fw_dir_override dracut module
---------------------------------------------
This dracut module injects directories with microcode files for caveats
that pass "early" check_caveats check (with "-e" flag). In addition
to "check_caveats" overrides, the following abilities to control module's
behaviour are present:
 * Presence of one of the following files:
   - /etc/microcode_ctl/ucode_with_caveats/skip-host-only-check
   - /etc/microcode_ctl/ucode_with_caveats/skip-host-only-check-$cfg
   - /lib/firmware/$kver/skip-host-only-check
   - /lib/firmware/$kver/skip-host-only-check-$cfg
   (where "$kver" is the kernel version in question and "$cfg" is the caveat
   directory name) allows skipping matching of microcode file name when dracut's
   Host-Only mode is enabled.

When caveats_check succeeds, caveats directory (not its possibly populated
version for late microcode update: "/lib/firmware/KERNEL_VERSION";
it is done so in order
to have ability to configure list of caveats enabled for early and late
microcode update, independently) is added to dracut's list of firmware search
directories.

The module can be disabled by running dracut with
"-o microcode_ctl-fw_dir_override" (for one-time exclusion), or it can
be disabled permanently by uncommenting string
"omit_dracutmodules+=' microcode_ctl-fw_dir_override '" in
/usr/lib/dracut/dracut.conf.d/99-microcode-override.conf configuration file.

See dracut(8), section "Omitting dracut Modules", and dracut.conf(5), variable
"omit_dracutmodules" for additional information.


Caveats
=======

Intel Broadwell-EP/EX ("BDX-ML B/M/R0") caveat
----------------------------------------------
Microcode update process on Intel Broadwell-EP/EX CPUs (BDX-ML B/M/R0,
family 6, model 79, stepping 1) has issues that lead to system instability.
A series of changes for the Linux kernel has been developed in order to work
around those issues; however, as it turned out, some systems have issues even
when a microcode update performed on a kernel that contains those changes.
As a result, microcode update for this CPU model is disabled by default;
the microcode file, however, is still shipped as a part of microcode_ctl
package and can be used for performing a microcode update if it is enforced
via the aforementioned overridden. (See sections "check_caveats script"
and "reload_microcode script" for details).

Affected microcode: intel-ucode/06-4f-01.

Mitigation: late microcode loading is disabled for the affected CPU model.

Minimum versions of the kernel package that contain the aforementioned patch
series:
 - Upstream/RHEL 8: 4.17.0
 - RHEL 7.6 onwards:  3.10.0-894
 - RHEL 7.5:  3.10.0-862.6.1
 - RHEL 7.4:  3.10.0-693.35.1
 - RHEL 7.3:  3.10.0-514.52.1
 - RHEL 7.2:  3.10.0-327.70.1
 - RHEL 6.10: 2.6.32-754.1.1
 - RHEL 6.7:  2.6.32-573.58.1
 - RHEL 6.6:  2.6.32-504.71.1
 - RHEL 6.5:  2.6.32-431.90.1
 - RHEL 6.4:  2.6.32-358.90.1


Early microcode load inside a virtual machine
---------------------------------------------
RHEL 8 kernel supports performing microcode update during early boot stage
from a cpio archive placed at the beginning of the initramfs image.  However,
when an early microcode update is attempted inside some virtualised
environments, that may result in unexpected system behaviour.

Affected microcode: all.

Mitigation: early microcode loading is disabled for all CPU models.

Minimum versions of kernel RPM that contain the fix:
 - Upstream/RHEL 8: 4.10.0
 - RHEL 7.6 onwards: 3.10.0-930
 - RHEL 7.5: 3.10.0-862.14.1
 - RHEL 7.4: 3.10.0-693.38.1
 - RHEL 7.3: 3.10.0-514.57.1
 - RHEL 7.2; 3.10.0-327.73.1


Additional information
======================

Information regarding microcode versions required for mitigating specific
side-channel cache attacks is available in the following knowledge base
articles:
 * CVE-2017-5715 ("Spectre"):
   https://access.redhat.com/articles/3436091
 * CVE-2018-3639 ("Speculative Store Bypass"):
   https://access.redhat.com/articles/3540901
 * CVE-2018-3620, CVE-2018-3646 ("L1 Terminal Fault Attack"):
   https://access.redhat.com/articles/3562741