Petr Šabata f5bf49
Firmware assisted dump (fadump) HOWTO
Petr Šabata f5bf49
Petr Šabata f5bf49
Introduction
Petr Šabata f5bf49
Petr Šabata f5bf49
Firmware assisted dump is a new feature in the 3.4 mainline kernel supported
Petr Šabata f5bf49
only on powerpc architecture. The goal of firmware-assisted dump is to enable
Petr Šabata f5bf49
the dump of a crashed system, and to do so from a fully-reset system, and to
Petr Šabata f5bf49
minimize the total elapsed time until the system is back in production use. A
Petr Šabata f5bf49
complete documentation on implementation can be found at
Petr Šabata f5bf49
Documentation/powerpc/firmware-assisted-dump.txt in upstream linux kernel tree
Petr Šabata f5bf49
from 3.4 version and above.
Petr Šabata f5bf49
Petr Šabata f5bf49
Please note that the firmware-assisted dump feature is only available on Power6
Petr Šabata f5bf49
and above systems with recent firmware versions.
Petr Šabata f5bf49
Petr Šabata f5bf49
Overview
Petr Šabata f5bf49
Petr Šabata f5bf49
Fadump
Petr Šabata f5bf49
Petr Šabata f5bf49
Fadump is a robust kernel crash dumping mechanism to get reliable kernel crash
Petr Šabata f5bf49
dump with assistance from firmware. This approach does not use kexec, instead
Petr Šabata f5bf49
firmware assists in booting the kdump kernel while preserving memory contents.
Petr Šabata f5bf49
Unlike kdump, the system is fully reset, and loaded with a fresh copy of the
Petr Šabata f5bf49
kernel. In particular, PCI and I/O devices are reinitialized and are in a
Petr Šabata f5bf49
clean, consistent state.  This second kernel, often called a capture kernel,
Petr Šabata f5bf49
boots with very little memory and captures the dump image.
Petr Šabata f5bf49
Petr Šabata f5bf49
The first kernel registers the sections of memory with the Power firmware for
Petr Šabata f5bf49
dump preservation during OS initialization. These registered sections of memory
Petr Šabata f5bf49
are reserved by the first kernel during early boot. When a system crashes, the
Petr Šabata f5bf49
Power firmware fully resets the system, preserves all the system memory
Petr Šabata f5bf49
contents, save the low memory (boot memory of size larger of 5% of system
Petr Šabata f5bf49
RAM or 256MB) of RAM to the previous registered region. It will also save
Petr Šabata f5bf49
system registers, and hardware PTE's.
Petr Šabata f5bf49
Petr Šabata f5bf49
Fadump is supported only on ppc64 platform. The standard kernel and capture
Petr Šabata f5bf49
kernel are one and the same on ppc64.
Petr Šabata f5bf49
Petr Šabata f5bf49
If you're reading this document, you should already have kexec-tools
Petr Šabata f5bf49
installed. If not, you install it via the following command:
Petr Šabata f5bf49
Petr Šabata f5bf49
    # yum install kexec-tools
Petr Šabata f5bf49
Petr Šabata f5bf49
Fadump Operational Flow:
Petr Šabata f5bf49
Petr Šabata f5bf49
Like kdump, fadump also exports the ELF formatted kernel crash dump through
Petr Šabata f5bf49
/proc/vmcore. Hence existing kdump infrastructure can be used to capture fadump
Petr Šabata f5bf49
vmcore. The idea is to keep the functionality transparent to end user. From
Petr Šabata f5bf49
user perspective there is no change in the way kdump init script works.
Petr Šabata f5bf49
Petr Šabata f5bf49
However, unlike kdump, fadump does not pre-load kdump kernel and initrd into
Petr Šabata f5bf49
reserved memory, instead it always uses default OS initrd during second boot
Petr Šabata f5bf49
after crash. Hence, for fadump, we rebuild the new kdump initrd and replace it
Petr Šabata f5bf49
with default initrd. Before replacing existing default initrd we take a backup
Petr Šabata f5bf49
of original default initrd for user's reference. The dracut package has been
Petr Šabata f5bf49
enhanced to rebuild the default initrd with vmcore capture steps. The initrd
Petr Šabata f5bf49
image is rebuilt as per the configuration in /etc/kdump.conf file.
Petr Šabata f5bf49
Petr Šabata f5bf49
The control flow of fadump works as follows:
Petr Šabata f5bf49
01. System panics.
Petr Šabata f5bf49
02. At the crash, kernel informs power firmware that kernel has crashed.
Petr Šabata f5bf49
03. Firmware takes the control and reboots the entire system preserving
Petr Šabata f5bf49
    only the memory (resets all other devices).
Petr Šabata f5bf49
04. The reboot follows the normal booting process (non-kexec).
Petr Šabata f5bf49
05. The boot loader loads the default kernel and initrd from /boot
Petr Šabata f5bf49
06. The default initrd loads and runs /init
Petr Šabata f5bf49
07. dracut-kdump.sh script present in fadump aware default initrd checks if
Petr Šabata f5bf49
    '/proc/device-tree/rtas/ibm,kernel-dump'  file exists  before executing
Petr Šabata f5bf49
    steps to capture vmcore.
Petr Šabata f5bf49
    (This check will help to bypass the vmcore capture steps during normal boot
Petr Šabata f5bf49
     process.)
Petr Šabata f5bf49
09. Captures dump according to /etc/kdump.conf
Petr Šabata f5bf49
10. Is dump capture successful (yes goto 12, no goto 11)
Petr Šabata f5bf49
11. Perform the failure action specified in /etc/kdump.conf
Petr Šabata f5bf49
    (The default failure action is reboot, if unspecified)
Petr Šabata f5bf49
12. Perform the final action specified in /etc/kdump.conf
Petr Šabata f5bf49
    (The default final action is reboot, if unspecified)
Petr Šabata f5bf49
Petr Šabata f5bf49
Petr Šabata f5bf49
How to configure fadump:
Petr Šabata f5bf49
Petr Šabata f5bf49
Again, we assume if you're reading this document, you should already have
Petr Šabata f5bf49
kexec-tools installed. If not, you install it via the following command:
Petr Šabata f5bf49
Petr Šabata f5bf49
    # yum install kexec-tools
Petr Šabata f5bf49
Petr Šabata f5bf49
Make the kernel to be configured with FADump as the default boot entry, if
Petr Šabata f5bf49
it isn't already:
Petr Šabata f5bf49
Petr Šabata f5bf49
   # grubby --set-default=/boot/vmlinuz-<kver>
Petr Šabata f5bf49
Petr Šabata f5bf49
Boot into the kernel to be configured for FADump. To be able to do much of
Petr Šabata f5bf49
anything interesting in the way of debug analysis, you'll also need to install
Petr Šabata f5bf49
the kernel-debuginfo package, of the same arch as your running kernel, and the
Petr Šabata f5bf49
crash utility:
Petr Šabata f5bf49
Petr Šabata f5bf49
    # yum --enablerepo=\*debuginfo install kernel-debuginfo.$(uname -m) crash
Petr Šabata f5bf49
Petr Šabata f5bf49
Next up, we need to modify some boot parameters to enable firmware assisted
Petr Šabata f5bf49
dump. With the help of grubby, it's very easy to append "fadump=on" to the end
Petr Šabata f5bf49
of your kernel boot parameters. To reserve the appropriate amount of memory
Petr Šabata f5bf49
for boot memory preservation, pass 'crashkernel=X' kernel cmdline parameter.
Petr Šabata f5bf49
For the recommended value of X, see 'FADump Memory Requirements' section.
Petr Šabata f5bf49
Petr Šabata f5bf49
   # grubby --args="fadump=on crashkernel=6G" --update-kernel=/boot/vmlinuz-`uname -r`
Petr Šabata f5bf49
DistroBaker a10140
By default, FADump reserved memory will be initialized as CMA area to make the
DistroBaker a10140
memory available through CMA allocator on the production kernel. We can opt out
DistroBaker a10140
of this, making reserved memory unavailable to production kernel, by booting the
DistroBaker a10140
linux kernel with 'fadump=nocma' instead of 'fadump=on'.
DistroBaker a10140
Petr Šabata f5bf49
The term 'boot memory' means size of the low memory chunk that is required for
Petr Šabata f5bf49
a kernel to boot successfully when booted with restricted memory.  By default,
Petr Šabata f5bf49
the boot memory size will be the larger of 5% of system RAM or 256MB.
Petr Šabata f5bf49
Alternatively, user can also specify boot memory size through boot parameter
Petr Šabata f5bf49
'fadump_reserve_mem=' which will override the default calculated size. Use this
Petr Šabata f5bf49
option if default boot memory size is not sufficient for second kernel to boot
Petr Šabata f5bf49
successfully.
Petr Šabata f5bf49
Petr Šabata f5bf49
After making said changes, reboot your system, so that the specified memory is
Petr Šabata f5bf49
reserved and left untouched by the normal system. Take note that the output of
Petr Šabata f5bf49
'free -m' will show X MB less memory than without this parameter, which is
Petr Šabata f5bf49
expected. If you see OOM (Out Of Memory) error messages while loading capture
Petr Šabata f5bf49
kernel, then you should bump up the memory reservation size.
Petr Šabata f5bf49
Petr Šabata f5bf49
Now that you've got that reserved memory region set up, you want to turn on
Petr Šabata f5bf49
the kdump init script:
Petr Šabata f5bf49
Petr Šabata f5bf49
    # systemctl enable kdump.service
Petr Šabata f5bf49
Petr Šabata f5bf49
Then, start up kdump as well:
Petr Šabata f5bf49
Petr Šabata f5bf49
    # systemctl start kdump.service
Petr Šabata f5bf49
Petr Šabata f5bf49
This should turn on the firmware assisted functionality in kernel by
Petr Šabata f5bf49
echo'ing 1 to /sys/kernel/fadump_registered, leaving the system ready
Petr Šabata f5bf49
to capture a vmcore upon crashing. For journaling filesystems like XFS an
Petr Šabata f5bf49
additional step is required to ensure bootloader does not pick the
Petr Šabata f5bf49
older initrd (without vmcore capture scripts):
Petr Šabata f5bf49
Petr Šabata f5bf49
  * If /boot is a separate partition, run the below commands as the root user,
Petr Šabata f5bf49
    or as a user with CAP_SYS_ADMIN rights:
Petr Šabata f5bf49
Petr Šabata f5bf49
        # fsfreeze -f
Petr Šabata f5bf49
        # fsfreeze -u
Petr Šabata f5bf49
Petr Šabata f5bf49
  * If /boot is not a separate partition, reboot the system.
Petr Šabata f5bf49
Petr Šabata f5bf49
After reboot check if the kdump service is up and running with:
Petr Šabata f5bf49
Petr Šabata f5bf49
  # systemctl status kdump.service
Petr Šabata f5bf49
Petr Šabata f5bf49
To test out whether FADump is configured properly, you can force-crash your
Petr Šabata f5bf49
system by echo'ing a 'c' into /proc/sysrq-trigger:
Petr Šabata f5bf49
Petr Šabata f5bf49
    # echo c > /proc/sysrq-trigger
Petr Šabata f5bf49
Petr Šabata f5bf49
You should see some panic output, followed by the system reset and booting into
Petr Šabata f5bf49
fresh copy of kernel. When default initrd loads and runs /init, vmcore should
Petr Šabata f5bf49
be copied out to disk (by default, in /var/crash/<YYYY.MM.DD-HH:MM:SS>/vmcore),
Petr Šabata f5bf49
then the system rebooted back into your normal kernel.
Petr Šabata f5bf49
Petr Šabata f5bf49
Once back to your normal kernel, you can use the previously installed crash
Petr Šabata f5bf49
kernel in conjunction with the previously installed kernel-debuginfo to
Petr Šabata f5bf49
perform postmortem analysis:
Petr Šabata f5bf49
Petr Šabata f5bf49
    # crash /usr/lib/debug/lib/modules/2.6.17-1.2621.el5/vmlinux
Petr Šabata f5bf49
    /var/crash/2006-08-23-15:34/vmcore
Petr Šabata f5bf49
Petr Šabata f5bf49
    crash> bt
Petr Šabata f5bf49
Petr Šabata f5bf49
and so on...
Petr Šabata f5bf49
Petr Šabata f5bf49
Saving vmcore-dmesg.txt
Petr Šabata f5bf49
-----------------------
Petr Šabata f5bf49
Kernel log bufferes are one of the most important information available
Petr Šabata f5bf49
in vmcore. Now before saving vmcore, kernel log bufferes are extracted
Petr Šabata f5bf49
from /proc/vmcore and saved into a file vmcore-dmesg.txt. After
Petr Šabata f5bf49
vmcore-dmesg.txt, vmcore is saved. Destination disk and directory for
Petr Šabata f5bf49
vmcore-dmesg.txt is same as vmcore. Note that kernel log buffers will
Petr Šabata f5bf49
not be available if dump target is raw device.
Petr Šabata f5bf49
Petr Šabata f5bf49
FADump Memory Requirements:
Petr Šabata f5bf49
Petr Šabata f5bf49
  System Memory          Recommended memory
Petr Šabata f5bf49
--------------------- ----------------------
Petr Šabata f5bf49
    4 GB - 16 GB     :        768 MB
Petr Šabata f5bf49
   16 GB - 64 GB     :       1024 MB
Petr Šabata f5bf49
   64 GB - 128 GB    :          2 GB
Petr Šabata f5bf49
  128 GB - 1 TB      :          4 GB
Petr Šabata f5bf49
    1 TB - 2 TB      :          6 GB
Petr Šabata f5bf49
    2 TB - 4 TB      :         12 GB
Petr Šabata f5bf49
    4 TB - 8 TB      :         20 GB
Petr Šabata f5bf49
    8 TB - 16 TB     :         36 GB
Petr Šabata f5bf49
   16 TB - 32 TB     :         64 GB
Petr Šabata f5bf49
   32 TB - 64 TB     :        128 GB
Petr Šabata f5bf49
   64 TB & above     :        180 GB
Petr Šabata f5bf49
Petr Šabata f5bf49
Things to remember:
Petr Šabata f5bf49
Petr Šabata f5bf49
1) The memory required to boot capture Kernel is a moving target that depends
Petr Šabata f5bf49
   on many factors like hardware attached to the system, kernel and modules in
Petr Šabata f5bf49
   use, packages installed and services enabled, there is no one-size-fits-all.
Petr Šabata f5bf49
   But the above recommendations are based on system memory. So, the above
Petr Šabata f5bf49
   recommendations for FADump come with a few assumptions, based on available
Petr Šabata f5bf49
   system memory, about the resources the system could have. So, please take
Petr Šabata f5bf49
   the recommendations with a pinch of salt and remember to try capturing dump
Petr Šabata f5bf49
   a few times to confirm that the system is configured successfully with dump
Petr Šabata f5bf49
   capturing support.
Petr Šabata f5bf49
Petr Šabata f5bf49
2) Though the memory requirements for FADump seem high, this memory is not
Petr Šabata f5bf49
   completely set aside but made available for userspace applications to use,
Petr Šabata f5bf49
   through the CMA allocator.
Petr Šabata f5bf49
Petr Šabata f5bf49
3) As the same initrd is used for booting production kernel as well as capture
Petr Šabata f5bf49
   kernel and with dump being captured in a restricted memory environment, few
Petr Šabata f5bf49
   optimizations (like not inclding network dracut module, disabling multipath
Petr Šabata f5bf49
   and such) are applied while building the initrd. In case, the production
Petr Šabata f5bf49
   environment needs these optimizations to be avoided, dracut_args option in
Petr Šabata f5bf49
   /etc/kdump.conf file could be leveraged. For example, if a user wishes for
Petr Šabata f5bf49
   network module to be included in the initrd, adding the below entry in
Petr Šabata f5bf49
   /etc/kdump.conf file and restarting kdump service would take care of it.
Petr Šabata f5bf49
Petr Šabata f5bf49
   dracut_args --add "network"
Petr Šabata f5bf49
Petr Šabata f5bf49
4) If FADump is configured to capture vmcore to a remote dump target using SSH
Petr Šabata f5bf49
   or NFS protocol, the network interface is renamed to kdump-<interface-name>
Petr Šabata f5bf49
   if <interface-name> is generic, for example, *eth#, or net#. This problem
Petr Šabata f5bf49
   occurs because the vmcore capture scripts in the initial RAM disk (initrd)
Petr Šabata f5bf49
   add the kdump- prefix to the network interface name to secure persistent
Petr Šabata f5bf49
   naming. As the same initrd is used for production kernel boot, the interface
Petr Šabata f5bf49
   name is changed for the production kernel too.
Petr Šabata f5bf49
Petr Šabata f5bf49
Dump Triggering methods:
Petr Šabata f5bf49
Petr Šabata f5bf49
This section talks about the various ways, other than a Kernel Panic, in which
Petr Šabata f5bf49
fadump can be triggered. The following methods assume that fadump is configured
Petr Šabata f5bf49
on your system, with the scripts enabled as described in the section above.
Petr Šabata f5bf49
Petr Šabata f5bf49
1) AltSysRq C
Petr Šabata f5bf49
Petr Šabata f5bf49
FAdump can be triggered with the combination of the 'Alt','SysRq' and 'C'
Petr Šabata f5bf49
keyboard keys. Please refer to the following link for more details:
Petr Šabata f5bf49
Petr Šabata f5bf49
https://fedoraproject.org/wiki/QA/Sysrq
Petr Šabata f5bf49
Petr Šabata f5bf49
In addition, on PowerPC boxes, fadump can also be triggered via Hardware
Petr Šabata f5bf49
Management Console(HMC) using 'Ctrl', 'O' and 'C' keyboard keys.
Petr Šabata f5bf49
Petr Šabata f5bf49
2) Kernel OOPs
Petr Šabata f5bf49
Petr Šabata f5bf49
If we want to generate a dump everytime the Kernel OOPses, we can achieve this
Petr Šabata f5bf49
by setting the 'Panic On OOPs' option as follows:
Petr Šabata f5bf49
Petr Šabata f5bf49
    # echo 1 > /proc/sys/kernel/panic_on_oops
Petr Šabata f5bf49
Petr Šabata f5bf49
3) PowerPC specific methods:
Petr Šabata f5bf49
Petr Šabata f5bf49
On IBM PowerPC machines, issuing a soft reset invokes the XMON debugger(if
Petr Šabata f5bf49
XMON is configured). To configure XMON one needs to compile the kernel with
Petr Šabata f5bf49
the CONFIG_XMON and CONFIG_XMON_DEFAULT options, or by compiling with
Petr Šabata f5bf49
CONFIG_XMON and booting the kernel with xmon=on option.
Petr Šabata f5bf49
Petr Šabata f5bf49
Following are the ways to remotely issue a soft reset on PowerPC boxes, which
Petr Šabata f5bf49
would drop you to XMON. Pressing a 'X' (capital alphabet X) followed by an
Petr Šabata f5bf49
'Enter' here will trigger the dump.
Petr Šabata f5bf49
Petr Šabata f5bf49
3.1) HMC
Petr Šabata f5bf49
Petr Šabata f5bf49
Hardware Management Console(HMC) available on Power4 and Power5 machines allow
Petr Šabata f5bf49
partitions to be reset remotely. This is specially useful in hang situations
Petr Šabata f5bf49
where the system is not accepting any keyboard inputs.
Petr Šabata f5bf49
Petr Šabata f5bf49
Once you have HMC configured, the following steps will enable you to trigger
Petr Šabata f5bf49
fadump via a soft reset:
Petr Šabata f5bf49
Petr Šabata f5bf49
On Power4
Petr Šabata f5bf49
  Using GUI
Petr Šabata f5bf49
Petr Šabata f5bf49
    * In the right pane, right click on the partition you wish to dump.
Petr Šabata f5bf49
    * Select "Operating System->Reset".
Petr Šabata f5bf49
    * Select "Soft Reset".
Petr Šabata f5bf49
    * Select "Yes".
Petr Šabata f5bf49
Petr Šabata f5bf49
  Using HMC Commandline
Petr Šabata f5bf49
Petr Šabata f5bf49
    # reset_partition -m <machine> -p <partition> -t soft
Petr Šabata f5bf49
Petr Šabata f5bf49
On Power5
Petr Šabata f5bf49
  Using GUI
Petr Šabata f5bf49
Petr Šabata f5bf49
    * In the right pane, right click on the partition you wish to dump.
Petr Šabata f5bf49
    * Select "Restart Partition".
Petr Šabata f5bf49
    * Select "Dump".
Petr Šabata f5bf49
    * Select "OK".
Petr Šabata f5bf49
Petr Šabata f5bf49
  Using HMC Commandline
Petr Šabata f5bf49
Petr Šabata f5bf49
    # chsysstate -m <managed system name> -n <lpar name> -o dumprestart -r lpar
Petr Šabata f5bf49
Petr Šabata f5bf49
3.2) Blade Management Console for Blade Center
Petr Šabata f5bf49
Petr Šabata f5bf49
To initiate a dump operation, go to Power/Restart option under "Blade Tasks" in
Petr Šabata f5bf49
the Blade Management Console. Select the corresponding blade for which you want
Petr Šabata f5bf49
to initate the dump and then click "Restart blade with NMI". This issues a
Petr Šabata f5bf49
system reset and invokes xmon debugger.
Petr Šabata f5bf49
Petr Šabata f5bf49
Petr Šabata f5bf49
Advanced Setups & Failure action:
Petr Šabata f5bf49
Petr Šabata f5bf49
Kdump and fadump exhibit similar behavior in terms of setup & failure action.
Petr Šabata f5bf49
For fadump advanced setup related information see section "Advanced Setups" in
Petr Šabata f5bf49
"kexec-kdump-howto.txt" document. Refer to "Failure action" section in "kexec-
Petr Šabata f5bf49
kdump-howto.txt" document for fadump failure action related information.
Petr Šabata f5bf49
Petr Šabata f5bf49
Compression and filtering
Petr Šabata f5bf49
Petr Šabata f5bf49
Refer "Compression and filtering" section in "kexec-kdump-howto.txt" document.
Petr Šabata f5bf49
Compression and filtering are same for kdump & fadump.
Petr Šabata f5bf49
Petr Šabata f5bf49
Petr Šabata f5bf49
Notes on rootfs mount:
Petr Šabata f5bf49
Dracut is designed to mount rootfs by default. If rootfs mounting fails it
Petr Šabata f5bf49
will refuse to go on. So fadump leaves rootfs mounting to dracut currently.
Petr Šabata f5bf49
We make the assumtion that proper root= cmdline is being passed to dracut
Petr Šabata f5bf49
initramfs for the time being. If you need modify "KDUMP_COMMANDLINE=" in
Petr Šabata f5bf49
/etc/sysconfig/kdump, you will need to make sure that appropriate root=
Petr Šabata f5bf49
options are copied from /proc/cmdline. In general it is best to append
Petr Šabata f5bf49
command line options using "KDUMP_COMMANDLINE_APPEND=" instead of replacing
Petr Šabata f5bf49
the original command line completely.
Petr Šabata f5bf49
Petr Šabata f5bf49
How to disable FADump:
Petr Šabata f5bf49
DistroBaker a10140
Remove "fadump=on"/"fadump=nocma" from kernel cmdline parameters OR replace
DistroBaker a10140
it with "fadump=off" kernel cmdline parameter:
Petr Šabata f5bf49
Petr Šabata f5bf49
   # grubby --update-kernel=/boot/vmlinuz-`uname -r` --remove-args="fadump=on"
DistroBaker a10140
or
DistroBaker a10140
   # grubby --update-kernel=/boot/vmlinuz-`uname -r` --remove-args="fadump=nocma"
DistroBaker a10140
OR
DistroBaker a10140
   # grubby --update-kernel=/boot/vmlinuz-`uname -r` --args="fadump=off"
Petr Šabata f5bf49
Petr Šabata f5bf49
If KDump is to be used as the dump capturing mechanism, update the crashkernel
Petr Šabata f5bf49
parameter (Else, remove "crashkernel=" parameter too, using grubby):
Petr Šabata f5bf49
Petr Šabata f5bf49
   # grubby --update-kernel=/boot/vmlinuz-$kver --args="crashkernl=auto"
Petr Šabata f5bf49
Petr Šabata f5bf49
Reboot the system for the settings to take effect.