The microcode_ctl package shipped with RHEL contains provisions for issues with
microcode loading on the old kernels. While those provisions are expected
to suit most users, several knobs are provided in order to provide ability
to override the default behaviour.
General behaviour
=================
In RHEL 8 (as well as RHEL 7 before it), there are currently two main handlers
for CPU microcode update:
* Early microcode update. It uses GenuineIntel.bin or AuthenticAMD.bin file
placed at the beginning of an initramfs image
(/boot/initramfs-KERNEL_VERSION.img, where "KERNEL_VERSION" is a kernel
version in the same format as provided by "uname -r") as a source
of microcode data, and is performed very early during the boot process
(if the relevant microcode file is available in the aforementioned file).
* On-demand (late) microcode update. It can be triggered by writing "1" to
/sys/devices/system/cpu/microcode/reload file (provided my the "microcode"
module). It relies on request_firmware infrastructure, which searches (and
loads, if found) microcode from a file present in one of the following
directories (in the search order):
/lib/firmware/updates/KERNEL_VERSION/
/lib/firmware/updates/
/lib/firmware/KERNEL_VERSION/
/lib/firmware/
(there is also an additional directory that can be configured via the
"fw_path_para" module option of the "firmware_class" module; as this module
is built-in in RHEL kernel, a boot parameter "firmware_class.fw_path_para"
should be used for that purpose; this is out of the document's scope, however)
The firmware for Intel CPUs is searched in "intel-ucode" subdirectory, and for
AMD CPUs, a file under "amd-ucode" is searched.
For Intel CPUs, the name of the specific microcode file the kernel tries to load
has the format "FF-MM-SS", where "FF" is the family number, "MM" is the model
number, and "SS" is the stepping. All those numbers are zero-filled to two digits
and are written in hexadecimal (letters are in the lower case). For AMD CPUs,
the file name has the format "microcode_amd_famFFh.bin", where "FF" is the
family number, written in hexadecimal, letters are in the lower case, not
zero-filled.
The early microcode is placed into initramfs image by the "dracut" script, which
scans the aforementioned subdirectories of the configured list of firmware
directories (by default, the list consists of two directories in RHEL 8,
"/lib/firmware/updates" and "/lib/firmware").
In RHEL 8, AMD CPU microcode is shipped as a part of the linux-firmware package,
and Intel microcode is shipped as a part of the microcode_ctl package.
The microcode_ctl package currently includes the following:
* Intel CPU microcode files, placed in /usr/share/microcode_ctl/intel-ucode
directory (currently there are none);
* A dracut module, /usr/lib/dracut/modules.d/99microcode_ctl-fw_dir_override,
that controls which additional firmware directories will be added to dracut's
default configuration;
* A dracut configuration file, /usr/lib/dracut/dracut.conf.d/01-microcode.conf,
that enables inclusion of early microcode to the generated initramfs
in dracut;
* A dracut configuration file,
/usr/lib/dracut/dracut.conf.d/99-microcode-override.conf, that provides a way
to quickly disable 99microcode_ctl-fw_dir-override dracut module;
* A systemd service file, microcode.service, that triggers microcode reload
late during boot;
* A set of directories in /usr/share/microcode_ctl/ucode_with_caveats, each
of which contains configuration and related data for various caveats related
to microcode:
* readme - description of caveat and related information,
* config - caveat configuration file, with syntax as described in "Caveat
configuration" section below,
* intel-ucode - directory containing microcode files related to the caveat;
* A set of support scripts, placed in /usr/libexec/microcode_ctl:
* "check_caveats" is an utility script that performs checks of the target
kernel (and running CPU) in accordance with caveat configuration files
in ucode_with_caveats directory and reports whether it passes them or not,
* "reload_microcode" is a script that is called by microcode.service and
triggers microcode reloading (by writing "1" to
/sys/devices/system/cpu/microcode/reload) if the running kernel passes
check_caveats checks,
* "update_ucode" is a script that populates symlinks to microcode files
in /lib/firmware, so it can be picked up by relevant kernels for the late
microcode loading.
Also, microcode_ctl RPM includes triggers that run update_ucode script on every
installation or removal of a kernel RPM in order to provide microcode files
for newly installed kernels and cleanup symlinks for the uninstalled ones.
Caveat configuration
--------------------
There is a directory for each caveat under
/usr/share/microcode_ctl/ucode_with_caveats, containing the following files:
* "config", a configuration file for the caveat;
* "readme", that contains description of the caveat;
* set of related associated microcode files.
"config" file is a set of lines each containing option name and its value,
separated by white space. Currently, the following options are supported:
* "model" option, which has format "VENDOR_ID FF-MM-SS", that specifies
to which CPU model the caveat is applicable (check_caveats ignores caveats
with non-matching models if "-m" option is passed to it). Can be set
in the configuration file only once (the last provided value is used).
* "vendor" option specifies CPUs of which vendor (as provided
in the /proc/cpuinfo file) the caveat is applicable to (check_caveats
ignores caveats with non-matching models when it is invoked with "-m"
option). Can be set in the configuration file only once.
* "path" is a glob pattern that specifies set of microcode files associated
with the caveat as a relative path to the caveat directory. This option
is used for populating files in /lib/firmware by update_ucode script and
for matching microcode file when dracut is run in host-only mode
(as in that case it uses only the first directory in firmware directory list
to look for the microcode file applicable to the host CPU). Can be set
in the configuration file multiple times.
* "kernel" is a minimal kernel version that supports proper handling
of the related microcode files during late microcode load. It may be
provided in one of the following formats that affect the way it is compared
to the running kernel version:
* A.B.C (where A, B, and C are decimal numbers), "upstream version". In this
case, simple version comparison against the respective part of the running
kernel version is used, and the running kernel version should be greater
or equal than the version provided in the configuration option in order
for comparison to succeed (that is, the first part, major version number,
of the running kernel version should be greater than the value provided
in the configuration option, or those should be equal and the second part,
minor version number, should be greater than the minor version number
of the kernel version provided in the configuration option, or the first
two parts should be equal and the third part, patch level, should
be greater or equal the patch level of the version in the configuration
option).
* A.B.C-Y (where A, B, C, and Y are decimal numbers), "Y-stream version".
In this case, A.B.C part should be equal, and Y part of the running kernel
version should be greater or equal than the Y part of the configuration
option version in order to satisfy the comparison requirement.
* A.B.C-Y.Z1.Z2 (where A, B, C, Y, Z1, and Z2 are decimal numbers),
"Z-stream version". In this case, A.B.C-Y part should be equal and Z1.Z2
part of the running kernel should be greater or equal than the respective
part of the configuration option version (when compared as a version)
for comparison to succeed.
Kernel version check passed if at least one comparison of the running kernel
version against a kernel version provided in a configuration option
succeeded. The "kernel" configuration option can be provided
in the configuration file multiple times.
* "kernel_early" is a minimal kernel version that supports proper handling
of the related microcode during early microcode load. The format of the
option and its semantics is similar to the "kernel" configuration options.
This option can be provided multiple times as well.
* "mc_min_ver_late" is the minimal version of the currently loaded microcode
on the CPU (as reported in /proc/cpuinfo) that supports late microcode
update. Microcode update will be attempted only if the currently loaded
microcode version is greater or equal the microcode version provided
in the configuration option. Can be set in the configuration file only once.
* "disable" is a way to disable a specific caveat from inside its
configuration. Argument for the argument is a list of stages ("early",
"late") for which the caveat should be disable. The configuration option
can be provided multiple times in a configuration file.
* "blacklist" is a marker for a start of list of blacklisted model names,
one model name per line. The model name of the running CPU (as reported
in /proc/cpuinfo) is compared against the names in the provided list, and,
if there is a match, caveat check fails.
check_caveats script
--------------------
"check_caveats" is an utility script (called by update_ucode, reload_microcode,
dracut module) that performs checks of the target kernel (and running CPU)
in accordance with caveat configuration files in directory
"/usr/share/microcode_ctl/ucode_with_caveats", and returns information, whether
the system passes the checks, or not.
Usage:
check_caveats [-e] [-k TARGET_KVER] [-c CONFIG]* [-m] [-v]'
Options:
-e - check for early microcode load possibility (instead of late microcode
load). "kernel_early" caveat configuration options are used for checking
instead of "kernel", and "mc_min_ver_late" is not checked.
-k - target kernel version to check against, $(uname -r) is used otherwise.
-c - caveat(s) to check, all caveat configurations found inside
$MC_CAVEATS_DATA_DIR are checked otherwise.
-m - ignore caveats that do not apply to the current CPU model.
-v - verbose output.
Environment:
MC_CAVEATS_DATA_DIR - directory that contains caveats configurations,
"/usr/share/microcode_ctl/ucode_with_caveats"
by default.
FW_DIR - directory containing firmware files (per-kernel configuration
overrides are checked there), "/lib/firmware" by default.
CFG_DIR - directory containing global caveats overrides,
"/etc/microcode_ctl/ucode_with_caveats" by default.
Output:
Script returns information about caveats check results. Output has a format
of "KEY VALUE1 VALUE2 ..." with KEY defining the semantics of the VALUEs.
Currently, the following data is issued:
- "cfgs" - list of caveats that have been processed (and not skipped
due to missing "config", "readme", or a disallow-* override described
below);
- "skip_cfgs" - list of caveats that have been skipped (due to missing
config/readme file, or because of overrides);
- "paths" - list of glob patterns matching files associated with caveats
that have been processed;
- "ok_cfgs" - list of caveat configurations that have all the checks passed
(or have enforced by one of force-* overrides described below);
- "ok_paths" - list of glob patterns associated with caveat files from
the "ok_cfgs" list;
- "fail_cfgs" - list of caveats that have one of the checks failed.
- "fail_paths" - list of glob patterns associated with caveats from the
"fail_cfgs" list.
Return value:
- 0 in case caveats check has passed, 1 otherwise.
- In "-d" mode, 0 is always returned.
Overrides:
When check_caveats perform its checks, it also checks for presence of files
in specific places, and, if they exist, check_caveats skips a caveat or ignores
its checks; that mechanism allows overriding the information provided
in configuration on local systems and affect the behaviour of the microcode
update process.
Current list of overrides (where $FW_DIR and $CFG_DIR are the environment
options described earlier; $kver - the currently processed kernel version,
$s is the requested stage ("early" or "late"), $cfg is the caveat directory
name):
$FW_DIR/$kver/disallow-$s-$cfg - skip a caveat for the requested stage for
a specific kernel version..
$FW_DIR/$kver/force-$s-$cfg - apply a specific caveat file for a specific
kernel version for the requested stage without
performing any checks.
$FW_DIR/$kver/disallow-$cfg - skip a caveat for any stage for a specific
kernel version.
$FW_DIR/$kver/force-$cfg - apply a specific caveat for any stage
for a specific kernel version without checks.
$FW_DIR/$kver/disallow-$s - skip all caveats for a specific stage
for a specific kernel version.
$CFG_DIR/disallow-$s-$cfg - skip a caveat for a specific stage for all
kernel versions.
$FW_DIR/$kver/force-$s - apply all caveats for a specific stage
for a specific kernel version without checks.
$CFG_DIR/force-$s-$cfg - apply a specific caveat for a specific stage for
all kernel versions without checks.
$FW_DIR/$kver/disallow - skip all caveats for all stages for a specific
kernel version.
$CFG_DIR/disallow-$cfg - skip a caveat for all stages for all kernel
versions.
$FW_DIR/$kver/force - apply all caveats for all stages for a specific kernel
version without checks.
$CFG_DIR/force-$cfg - apply a caveat for all stages for all kernel versions
without checks.
$CFG_DIR/disallow-$s - skip all caveat for all kernel versions
for a specific stage.
$CFG_DIR/force-$s - apply all caveats for all kernel versions for specific
stage without checks.
$CFG_DIR/disallow - skip all caveats for all stages for all kernel versions
(disable everything).
$CFG_DIR/force - force all caveats for all stages for all kernel versions
(enable everything).
The "apply" action above means creating symlinks in /lib/firmware by
update_ucode in case of the "late" stage and adding caveat directory to the list
of firmware directories by dracut plugin in case of the "early" stage.
The files are checked for existence until the first match, so more specific
overrides can override more broad ones.
Also, a caveat is ignored if it lacks either config or readme file.
update_ucode script
-------------------
"update_ucode" populates symlinks to microcode files in accordance with caveats
configuration. It enables late microcode loading that is invoked by triggering
/sys/devices/system/cpu/microcode/reload file. Since caveats depend
on the kernel version, symlinks are populated inside
"/lib/firmware/KERNEL_VERSION" directory for each installed kernel.
As a consequence, this script is triggered upon each kernel package installation
and removal.
The script has two parts: common and kernel-version-specific.
During the common part, files are populated from
/usr/share/microcode_ctl/intel-ucode in /lib/firmware/intel-ucode. There are
several possibilities to affect the process:
* Presence of "/etc/microcode_ctl/intel-ucode-disallow" file leads to skipping
the common part of the script.
* The same for "/lib/firmware/intel-ucode-disallow".
During the kernel-version-specific part, each caveat is checked against every
kernel version, and those combinations, for which caveat check succeeds,
gets the symlinks to the associated microcode files populated.
* Absence of "/lib/firmware/KERNEL_VERSION/readme-CAVEAT" prevents update_ucode
from removing symlinks related to the caveat for specific kernel version.
* Since the check is being done by check_caveats, all the overrides that
described there also stay.
Usage:
update_ucode [--action {add|remove|refresh|list}] [--kernel KERNELVER]*
[--verbose] [--dry-run] [--cleanup intel_ucode caveats_ucode]
[--skip-common] [--skip-kernel-specific]
Options:
--action - action to perform. Currently, the following actions are supported:
* "add" - create new symlinks.
* "remove" - remove old symlinks that are no longer needed.
* "refresh" - re-populate symlinks.
* "list" - list files under control of update_ucode.
By default, "refresh" action is executed.
--kernel - kernel version to process. By default, list of kernel versions
is formed based on contents of /lib/firmware and /lib/modules
directories.
--verbose - verbose output.
--dry-run - do not call commands, just print the invocation lines.
--cleanup - cleanup mode. Used by post-uninstall script during package
upgrades. Removes excess files in accordance to the contents
of the files provided in the arguments to the option.
--skip-common - do not process /lib/firmware directory.
--skip-kernel-specific - do not process /lib/firmware/KERNEL_VERSION
directories.
Return value:
0 on success, 1 on error.
reload_microcode script
-----------------------
"reload_microcode" is a script that is called by microcode.service and
triggers late microcode reloading (by writing "1" to
/sys/devices/system/cpu/microcode/reload) if the following check are passed:
* the microcode update performed not in a virtualised environment;
* running kernel passes "check_caveats" checks that applicable to the current
CPU model.
For a virtualised environment check, the script searches the "/proc/cpuinfo"
file for presence of the "hypervisor" flag among CPU features (it corresponds
to a CPUID feature bit set by hypervisors in order to inform that the kernel
operates inside a virtual machine). This check can be overridden and skipped
by creation of a file "/etc/microcode_ctl/ignore-hypervisor-flag".
The script has no options and always returns 0.
99microcode_ctl-fw_dir_override dracut module
---------------------------------------------
This dracut module injects directories with microcode files for caveats
that pass "early" check_caveats check (with "-e" flag). In addition
to "check_caveats" overrides, the following abilities to control module's
behaviour are present:
* Presence of one of the following files:
- /etc/microcode_ctl/ucode_with_caveats/skip-host-only-check
- /etc/microcode_ctl/ucode_with_caveats/skip-host-only-check-$cfg
- /lib/firmware/$kver/skip-host-only-check
- /lib/firmware/$kver/skip-host-only-check-$cfg
(where "$kver" is the kernel version in question and "$cfg" is the caveat
directory name) allows skipping matching of microcode file name when dracut's
Host-Only mode is enabled.
When caveats_check succeeds, caveats directory (not its possibly populated
version for late microcode update: "/lib/firmware/KERNEL_VERSION";
it is done so in order
to have ability to configure list of caveats enabled for early and late
microcode update, independently) is added to dracut's list of firmware search
directories.
The module can be disabled by running dracut with
"-o microcode_ctl-fw_dir_override" (for one-time exclusion), or it can
be disabled permanently by uncommenting string
"omit_dracutmodules+=' microcode_ctl-fw_dir_override '" in
/usr/lib/dracut/dracut.conf.d/99-microcode-override.conf configuration file.
See dracut(8), section "Omitting dracut Modules", and dracut.conf(5), variable
"omit_dracutmodules" for additional information.
Caveats
=======
Intel Broadwell-EP/EX ("BDX-ML B/M/R0") caveat
----------------------------------------------
Microcode update process on Intel Broadwell-EP/EX CPUs (BDX-ML B/M/R0,
family 6, model 79, stepping 1) has issues that lead to system instability.
A series of changes for the Linux kernel has been developed in order to work
around those issues; however, as it turned out, some systems have issues even
when a microcode update performed on a kernel that contains those changes.
As a result, microcode update for this CPU model is disabled by default;
the microcode file, however, is still shipped as a part of microcode_ctl
package and can be used for performing a microcode update if it is enforced
via the aforementioned overridden. (See sections "check_caveats script"
and "reload_microcode script" for details).
Affected microcode: intel-ucode/06-4f-01.
Mitigation: late microcode loading is disabled for the affected CPU model.
Minimum versions of the kernel package that contain the aforementioned patch
series:
- Upstream/RHEL 8: 4.17.0
- RHEL 7.6 onwards: 3.10.0-894
- RHEL 7.5: 3.10.0-862.6.1
- RHEL 7.4: 3.10.0-693.35.1
- RHEL 7.3: 3.10.0-514.52.1
- RHEL 7.2: 3.10.0-327.70.1
- RHEL 6.10: 2.6.32-754.1.1
- RHEL 6.7: 2.6.32-573.58.1
- RHEL 6.6: 2.6.32-504.71.1
- RHEL 6.5: 2.6.32-431.90.1
- RHEL 6.4: 2.6.32-358.90.1
Early microcode load inside a virtual machine
---------------------------------------------
RHEL 8 kernel supports performing microcode update during early boot stage
from a cpio archive placed at the beginning of the initramfs image. However,
when an early microcode update is attempted inside some virtualised
environments, that may result in unexpected system behaviour.
Affected microcode: all.
Mitigation: early microcode loading is disabled for all CPU models.
Minimum versions of kernel RPM that contain the fix:
- Upstream/RHEL 8: 4.10.0
- RHEL 7.6 onwards: 3.10.0-930
- RHEL 7.5: 3.10.0-862.14.1
- RHEL 7.4: 3.10.0-693.38.1
- RHEL 7.3: 3.10.0-514.57.1
- RHEL 7.2; 3.10.0-327.73.1
Additional information
======================
Information regarding microcode versions required for mitigating specific
side-channel cache attacks is available in the following knowledge base
articles:
* CVE-2017-5715 ("Spectre"):
https://access.redhat.com/articles/3436091
* CVE-2018-3639 ("Speculative Store Bypass"):
https://access.redhat.com/articles/3540901
* CVE-2018-3620, CVE-2018-3646 ("L1 Terminal Fault Attack"):
https://access.redhat.com/articles/3562741