The microcode_ctl package shipped with RHEL contains provisions for issues with

microcode loading on the old kernels.  While those provisions are expected

to suit most users, several knobs are provided in order to provide ability

to override the default behaviour.

General behaviour

=================

In RHEL 8 (as well as RHEL 7 before it), there are currently two main handlers

for CPU microcode update:

 * Early microcode update. It uses GenuineIntel.bin or AuthenticAMD.bin file

   placed at the beginning of an initramfs image

   (/boot/initramfs-KERNEL_VERSION.img, where "KERNEL_VERSION" is a kernel

   version in the same format as provided by "uname -r") as a source

   of microcode data, and is performed very early during the boot process

   (if the relevant microcode file is available in the aforementioned file).

 * On-demand (late) microcode update. It can be triggered by writing "1" to

   /sys/devices/system/cpu/microcode/reload file (provided my the "microcode"

   module). It relies on request_firmware infrastructure, which searches (and

   loads, if found) microcode from a file present in one of the following

   directories (in the search order):

       /lib/firmware/updates/KERNEL_VERSION/

       /lib/firmware/updates/

       /lib/firmware/KERNEL_VERSION/

       /lib/firmware/

  (there is also an additional directory that can be configured via the

  "fw_path_para" module option of the "firmware_class" module; as this module

  is built-in in RHEL kernel, a boot parameter "firmware_class.fw_path_para"

  should be used for that purpose; this is out of the document's scope, however)

The firmware for Intel CPUs is searched in "intel-ucode" subdirectory, and for

AMD CPUs, a file under "amd-ucode" is searched.

For Intel CPUs, the name of the specific microcode file the kernel tries to load

has the format "FF-MM-SS", where "FF" is the family number, "MM" is the model

number, and "SS" is the stepping. All those numbers are zero-filled to two digits

and are written in hexadecimal (letters are in the lower case).  For AMD CPUs,

the file name has the format "microcode_amd_famFFh.bin", where "FF" is the

family number, written in hexadecimal, letters are in the lower case, not

zero-filled.

The early microcode is placed into initramfs image by the "dracut" script, which

scans the aforementioned subdirectories of the configured list of firmware

directories (by default, the list consists of two directories in RHEL 8,

"/lib/firmware/updates" and "/lib/firmware").

In RHEL 8, AMD CPU microcode is shipped as a part of the linux-firmware package,

and Intel microcode is shipped as a part of the microcode_ctl package.

The microcode_ctl package currently includes the following:

 * Intel CPU microcode files, placed in /usr/share/microcode_ctl/intel-ucode

   directory (currently there are none);

 * A dracut module, /usr/lib/dracut/modules.d/99microcode_ctl-fw_dir_override,

   that controls which additional firmware directories will be added to dracut's

   default configuration;

 * A dracut configuration file, /usr/lib/dracut/dracut.conf.d/01-microcode.conf,

   that enables inclusion of early microcode to the generated initramfs

   in dracut;

 * A dracut configuration file,

   /usr/lib/dracut/dracut.conf.d/99-microcode-override.conf, that provides a way

   to quickly disable 99microcode_ctl-fw_dir-override dracut module;

 * A systemd service file, microcode.service, that triggers microcode reload

   late during boot;

 * A set of directories in /usr/share/microcode_ctl/ucode_with_caveats, each

   of which contains configuration and related data for various caveats related

   to microcode:

   * readme - description of caveat and related information,

   * config - caveat configuration file, with syntax as described in "Caveat

     configuration" section below,

   * intel-ucode - directory containing microcode files related to the caveat;

 * A set of support scripts, placed in /usr/libexec/microcode_ctl:

   * "check_caveats" is an utility script that performs checks of the target

     kernel (and running CPU) in accordance with caveat configuration files

     in ucode_with_caveats directory and reports whether it passes them or not,

   * "reload_microcode" is a script that is called by microcode.service and

     triggers microcode reloading (by writing "1" to

     /sys/devices/system/cpu/microcode/reload) if the running kernel passes

     check_caveats checks,

   * "update_ucode" is a script that populates symlinks to microcode files

     in /lib/firmware, so it can be picked up by relevant kernels for the late

     microcode loading.

Also, microcode_ctl RPM includes triggers that run update_ucode script on every

installation or removal of a kernel RPM in order to provide microcode files

for newly installed kernels and cleanup symlinks for the uninstalled ones.

Caveat configuration

--------------------

There is a directory for each caveat under

/usr/share/microcode_ctl/ucode_with_caveats, containing the following files:

 * "config", a configuration file for the caveat;

 * "readme", that contains description of the caveat;

 * set of related associated microcode files.

"config" file is a set of lines each containing option name and its value,

separated by white space.  Currently, the following options are supported:

 * "model" option, which has format "VENDOR_ID FF-MM-SS", that specifies

   to which CPU model the caveat is applicable (check_caveats ignores caveats

   with non-matching models if "-m" option is passed to it). Can be set

   in the configuration file only once (the last provided value is used).

 * "vendor" option specifies CPUs of which vendor (as provided

   in the /proc/cpuinfo file) the caveat is applicable to (check_caveats

   ignores caveats with non-matching models when it is invoked with "-m"

   option). Can be set in the configuration file only once.

 * "path" is a glob pattern that specifies set of microcode files associated

   with the caveat as a relative path to the caveat directory. This option

   is used for populating files in /lib/firmware by update_ucode script and

   for matching microcode file when dracut is run in host-only mode

   (as in that case it uses only the first directory in firmware directory list

   to look for the microcode file applicable to the host CPU).  Can be set

   in the configuration file multiple times.

 * "kernel" is a minimal kernel version that supports proper handling

   of the related microcode files during late microcode load.  It may be

   provided in one of the following formats that affect the way it is compared

   to the running kernel version:

    * A.B.C (where A, B, and C are decimal numbers), "upstream version". In this

      case, simple version comparison against the respective part of the running

      kernel version is used, and the running kernel version should be greater

      or equal than the version provided in the configuration option in order

      for comparison to succeed (that is, the first part, major version number,

      of the running kernel version should be greater than the value provided

      in the configuration option, or those should be equal and the second part,

      minor version number, should be greater than the minor version number

      of the kernel version provided in the configuration option, or the first

      two parts should be equal and the third part, patch level, should

      be greater or equal the patch level of the version in the configuration

      option).

    * A.B.C-Y (where A, B, C, and Y are decimal numbers), "Y-stream version".

      In this case, A.B.C part should be equal, and Y part of the running kernel

      version should be greater or equal than the Y part of the configuration

      option version in order to satisfy the comparison requirement.

    * A.B.C-Y.Z1.Z2 (where A, B, C, Y, Z1, and Z2 are decimal numbers),

      "Z-stream version". In this case, A.B.C-Y part should be equal and Z1.Z2

      part of the running kernel should be greater or equal than the respective

      part of the configuration option version (when compared as a version)

      for comparison to succeed.

   Kernel version check passed if at least one comparison of the running kernel

   version against a kernel version provided in a configuration option

   succeeded.  The "kernel" configuration option can be provided

   in the configuration file multiple times.

 * "kernel_early" is a minimal kernel version that supports proper handling

   of the related microcode during early microcode load. The format of the

   option and its semantics is similar to the "kernel" configuration options.

   This option can be provided multiple times as well.

 * "mc_min_ver_late" is the minimal version of the currently loaded microcode

   on the CPU (as reported in /proc/cpuinfo) that supports late microcode

   update.  Microcode update will be attempted only if the currently loaded

   microcode version is greater or equal the microcode version provided

   in the configuration option. Can be set in the configuration file only once.

 * "disable" is a way to disable a specific caveat from inside its

   configuration. Argument for the argument is a list of stages ("early",

   "late") for which the caveat should be disable. The configuration option

   can be provided multiple times in a configuration file.

 * "blacklist" is a marker for a start of list of blacklisted model names,

   one model name per line. The model name of the running CPU (as reported

   in /proc/cpuinfo) is compared against the names in the provided list, and,

   if there is a match, caveat check fails.

check_caveats script

--------------------

"check_caveats" is an utility script (called by update_ucode, reload_microcode,

dracut module) that performs checks of the target kernel (and running CPU)

in accordance with caveat configuration files in directory

"/usr/share/microcode_ctl/ucode_with_caveats", and returns information, whether

the system passes the checks, or not.

Usage:

    check_caveats [-e] [-k TARGET_KVER] [-c CONFIG]* [-m] [-v]'

Options:

  -e - check for early microcode load possibility (instead of late microcode

       load). "kernel_early" caveat configuration options are used for checking

       instead of "kernel", and "mc_min_ver_late" is not checked.

  -k - target kernel version to check against, $(uname -r) is used otherwise.

  -c - caveat(s) to check, all caveat configurations found inside

       $MC_CAVEATS_DATA_DIR are checked otherwise.

  -m - ignore caveats that do not apply to the current CPU model.

  -v - verbose output.

Environment:

  MC_CAVEATS_DATA_DIR - directory that contains caveats configurations,

                        "/usr/share/microcode_ctl/ucode_with_caveats"

			by default.

  FW_DIR - directory containing firmware files (per-kernel configuration

           overrides are checked there), "/lib/firmware" by default.

  CFG_DIR - directory containing global caveats overrides,

            "/etc/microcode_ctl/ucode_with_caveats" by default.

Output:

  Script returns information about caveats check results. Output has a format

  of "KEY VALUE1 VALUE2 ..." with KEY defining the semantics of the VALUEs.

  Currently, the following data is issued:

   - "cfgs" - list of caveats that have been processed (and not skipped

      due to missing "config", "readme", or a disallow-* override described

      below);

   - "skip_cfgs" - list of caveats that have been skipped (due to missing

     config/readme file, or because of overrides);

   - "paths" - list of glob patterns matching files associated with caveats

     that have been processed;

   - "ok_cfgs" - list of caveat configurations that have all the checks passed

     (or have enforced by one of force-* overrides described below);

   - "ok_paths" - list of glob patterns associated with caveat files from

     the "ok_cfgs" list;

   - "fail_cfgs" - list of caveats that have one of the checks failed.

   - "fail_paths" - list of glob patterns associated with caveats from the

     "fail_cfgs" list.

Return value:

  - 0 in case caveats check has passed, 1 otherwise.

  - In "-d" mode, 0 is always returned.

Overrides:

When check_caveats perform its checks, it also checks for presence of files

in specific places, and, if they exist, check_caveats skips a caveat or ignores

its checks; that mechanism allows overriding the information provided

in configuration on local systems and affect the behaviour of the microcode

update process.

Current list of overrides (where $FW_DIR and $CFG_DIR are the environment

options described earlier; $kver - the currently processed kernel version,

$s is the requested stage ("early" or "late"), $cfg is the caveat directory

name):

    $FW_DIR/$kver/disallow-$s-$cfg - skip a caveat for the requested stage for

                                     a specific kernel version..

    $FW_DIR/$kver/force-$s-$cfg - apply a specific caveat file for a specific

                                  kernel version for the requested stage without

				  performing any checks.

    $FW_DIR/$kver/disallow-$cfg - skip a caveat for any stage for a specific

                                  kernel version.

    $FW_DIR/$kver/force-$cfg - apply a specific caveat for any stage

                               for a specific kernel version without checks.

    $FW_DIR/$kver/disallow-$s - skip all caveats for a specific stage

                                for a specific kernel version.

    $CFG_DIR/disallow-$s-$cfg - skip a caveat for a specific stage for all

                                kernel versions.

    $FW_DIR/$kver/force-$s - apply all caveats for a specific stage

                             for a specific kernel version without checks.

    $CFG_DIR/force-$s-$cfg - apply a specific caveat for a specific stage for

                             all kernel versions without checks.

    $FW_DIR/$kver/disallow - skip all caveats for all stages for a specific

                             kernel version.

    $CFG_DIR/disallow-$cfg - skip a caveat for all stages for all kernel

                             versions.

    $FW_DIR/$kver/force - apply all caveats for all stages for a specific kernel

                          version without checks.

    $CFG_DIR/force-$cfg - apply a caveat for all stages for all kernel versions

                          without checks.

    $CFG_DIR/disallow-$s - skip all caveat for all kernel versions

                           for a specific stage.

    $CFG_DIR/force-$s - apply all caveats for all kernel versions for  specific

                        stage without checks.

    $CFG_DIR/disallow - skip all caveats for all stages for all kernel versions

                        (disable everything).

    $CFG_DIR/force - force all caveats for all stages for all kernel versions

                     (enable everything).

The "apply" action above means creating symlinks in /lib/firmware by

update_ucode in case of the "late" stage and adding caveat directory to the list

of firmware directories by dracut plugin in case of the "early" stage.

The files are checked for existence until the first match, so more specific

overrides can override more broad ones.

Also, a caveat is ignored if it lacks either config or readme file.

update_ucode script

-------------------

"update_ucode" populates symlinks to microcode files in accordance with caveats

configuration.  It enables late microcode loading that is invoked by triggering

/sys/devices/system/cpu/microcode/reload file.  Since caveats depend

on the kernel version, symlinks are populated inside

"/lib/firmware/KERNEL_VERSION" directory for each installed kernel.

As a consequence, this script is triggered upon each kernel package installation

and removal.

The script has two parts: common and kernel-version-specific.

During the common part, files are populated from

/usr/share/microcode_ctl/intel-ucode in /lib/firmware/intel-ucode. There are

several possibilities to affect the process:

 * Presence of "/etc/microcode_ctl/intel-ucode-disallow" file leads to skipping

   the common part of the script.

 * The same for "/lib/firmware/intel-ucode-disallow".

During the kernel-version-specific part, each caveat is checked against every

kernel version, and those combinations, for which caveat check succeeds,

gets the symlinks to the associated microcode files populated.

 * Absence of "/lib/firmware/KERNEL_VERSION/readme-CAVEAT" prevents update_ucode

   from removing symlinks related to the caveat for specific kernel version.

 * Since the check is being done by check_caveats, all the overrides that

   described there also stay.

Usage:

    update_ucode [--action {add|remove|refresh|list}] [--kernel KERNELVER]*

                 [--verbose] [--dry-run] [--cleanup intel_ucode caveats_ucode]

                 [--skip-common] [--skip-kernel-specific]

Options:

  --action - action to perform. Currently, the following actions are supported:

              * "add" - create new symlinks.

              * "remove" - remove old symlinks that are no longer needed.

              * "refresh" - re-populate symlinks.

              * "list" - list files under control of update_ucode.

             By default, "refresh" action is executed.

  --kernel - kernel version to process. By default, list of kernel versions

             is formed based on contents of /lib/firmware and /lib/modules

             directories.

  --verbose - verbose output.

  --dry-run - do not call commands, just print the invocation lines.

  --cleanup - cleanup mode. Used by post-uninstall script during package

              upgrades. Removes excess files in accordance to the contents

              of the files provided in the arguments to the option.

  --skip-common - do not process /lib/firmware directory.

  --skip-kernel-specific - do not process /lib/firmware/KERNEL_VERSION

                           directories.

Return value:

  0 on success, 1 on error.

reload_microcode script

-----------------------

"reload_microcode" is a script that is called by microcode.service and

triggers late microcode reloading (by writing "1" to

/sys/devices/system/cpu/microcode/reload) if the following check are passed:

 * the microcode update performed not in a virtualised environment;

 * running kernel passes "check_caveats" checks that applicable to the current

   CPU model.

For a virtualised environment check, the script searches the "/proc/cpuinfo"

file for presence of the "hypervisor" flag among CPU features (it corresponds

to a CPUID feature bit set by hypervisors in order to inform that the kernel

operates inside a virtual machine).  This check can be overridden and skipped

by creation of a file "/etc/microcode_ctl/ignore-hypervisor-flag".

The script has no options and always returns 0.

99microcode_ctl-fw_dir_override dracut module

---------------------------------------------

This dracut module injects directories with microcode files for caveats

that pass "early" check_caveats check (with "-e" flag). In addition

to "check_caveats" overrides, the following abilities to control module's

behaviour are present:

 * Presence of one of the following files:

   - /etc/microcode_ctl/ucode_with_caveats/skip-host-only-check

   - /etc/microcode_ctl/ucode_with_caveats/skip-host-only-check-$cfg

   - /lib/firmware/$kver/skip-host-only-check

   - /lib/firmware/$kver/skip-host-only-check-$cfg

   (where "$kver" is the kernel version in question and "$cfg" is the caveat

   directory name) allows skipping matching of microcode file name when dracut's

   Host-Only mode is enabled.

When caveats_check succeeds, caveats directory (not its possibly populated

version for late microcode update: "/lib/firmware/KERNEL_VERSION";

it is done so in order

to have ability to configure list of caveats enabled for early and late

microcode update, independently) is added to dracut's list of firmware search

directories.

The module can be disabled by running dracut with

"-o microcode_ctl-fw_dir_override" (for one-time exclusion), or it can

be disabled permanently by uncommenting string

"omit_dracutmodules+=' microcode_ctl-fw_dir_override '" in

/usr/lib/dracut/dracut.conf.d/99-microcode-override.conf configuration file.

See dracut(8), section "Omitting dracut Modules", and dracut.conf(5), variable

"omit_dracutmodules" for additional information.

Caveats

=======

Intel Broadwell-EP/EX ("BDX-ML B/M/R0") caveat

----------------------------------------------

Microcode update process on Intel Broadwell-EP/EX CPUs (BDX-ML B/M/R0,

family 6, model 79, stepping 1) has issues that lead to system instability.

A series of changes for the Linux kernel has been developed in order to work

around those issues; however, as it turned out, some systems have issues even

when a microcode update performed on a kernel that contains those changes.

As a result, microcode update for this CPU model is disabled by default;

the microcode file, however, is still shipped as a part of microcode_ctl

package and can be used for performing a microcode update if it is enforced

via the aforementioned overridden. (See sections "check_caveats script"

and "reload_microcode script" for details).

Affected microcode: intel-ucode/06-4f-01.

Mitigation: late microcode loading is disabled for the affected CPU model.

Minimum versions of the kernel package that contain the aforementioned patch

series:

 - Upstream/RHEL 8: 4.17.0

 - RHEL 7.6 onwards:  3.10.0-894

 - RHEL 7.5:  3.10.0-862.6.1

 - RHEL 7.4:  3.10.0-693.35.1

 - RHEL 7.3:  3.10.0-514.52.1

 - RHEL 7.2:  3.10.0-327.70.1

 - RHEL 6.10: 2.6.32-754.1.1

 - RHEL 6.7:  2.6.32-573.58.1

 - RHEL 6.6:  2.6.32-504.71.1

 - RHEL 6.5:  2.6.32-431.90.1

 - RHEL 6.4:  2.6.32-358.90.1

Early microcode load inside a virtual machine

---------------------------------------------

RHEL 8 kernel supports performing microcode update during early boot stage

from a cpio archive placed at the beginning of the initramfs image.  However,

when an early microcode update is attempted inside some virtualised

environments, that may result in unexpected system behaviour.

Affected microcode: all.

Mitigation: early microcode loading is disabled for all CPU models.

Minimum versions of kernel RPM that contain the fix:

 - Upstream/RHEL 8: 4.10.0

 - RHEL 7.6 onwards: 3.10.0-930

 - RHEL 7.5: 3.10.0-862.14.1

 - RHEL 7.4: 3.10.0-693.38.1

 - RHEL 7.3: 3.10.0-514.57.1

 - RHEL 7.2; 3.10.0-327.73.1

Additional information

======================

Information regarding microcode versions required for mitigating specific

side-channel cache attacks is available in the following knowledge base

articles:

 * CVE-2017-5715 ("Spectre"):

   https://access.redhat.com/articles/3436091

 * CVE-2018-3639 ("Speculative Store Bypass"):

   https://access.redhat.com/articles/3540901

 * CVE-2018-3620, CVE-2018-3646 ("L1 Terminal Fault Attack"):

   https://access.redhat.com/articles/3562741