893e0b
Kexec/Kdump HOWTO
893e0b
893e0b
Introduction
893e0b
893e0b
Kexec and kdump are new features in the 2.6 mainstream kernel. These features
893e0b
are included in Red Hat Enterprise Linux 5. The purpose of these features
893e0b
is to ensure faster boot up and creation of reliable kernel vmcores for
893e0b
diagnostic purposes.
893e0b
893e0b
Overview
893e0b
893e0b
Kexec
893e0b
893e0b
Kexec is a fastboot mechanism which allows booting a Linux kernel from the
893e0b
context of already running kernel without going through BIOS. BIOS can be very
893e0b
time consuming especially on the big servers with lots of peripherals. This can
893e0b
save a lot of time for developers who end up booting a machine numerous times.
893e0b
893e0b
Kdump
893e0b
893e0b
Kdump is a new kernel crash dumping mechanism and is very reliable because
893e0b
the crash dump is captured from the context of a freshly booted kernel and
893e0b
not from the context of the crashed kernel. Kdump uses kexec to boot into
893e0b
a second kernel whenever system crashes. This second kernel, often called
893e0b
a capture kernel, boots with very little memory and captures the dump image.
893e0b
893e0b
The first kernel reserves a section of memory that the second kernel uses
893e0b
to boot. Kexec enables booting the capture kernel without going through BIOS
893e0b
hence contents of first kernel's memory are preserved, which is essentially
893e0b
the kernel crash dump.
893e0b
893e0b
Kdump is supported on the i686, x86_64, ia64 and ppc64 platforms. The
893e0b
standard kernel and capture kernel are one in the same on i686, x86_64,
893e0b
ia64 and ppc64.
893e0b
893e0b
If you're reading this document, you should already have kexec-tools
893e0b
installed. If not, you install it via the following command:
893e0b
893e0b
    # yum install kexec-tools
893e0b
893e0b
Now load a kernel with kexec:
893e0b
893e0b
    # kver=`uname -r` # kexec -l /boot/vmlinuz-$kver
893e0b
    --initrd=/boot/initrd-$kver.img \
893e0b
        --command-line="`cat /proc/cmdline`"
893e0b
893e0b
NOTE: The above will boot you back into the kernel you're currently running,
893e0b
if you want to load a different kernel, substitute it in place of `uname -r`.
893e0b
893e0b
Now reboot your system, taking note that it should bypass the BIOS:
893e0b
893e0b
    # reboot
893e0b
893e0b
893e0b
How to configure kdump:
893e0b
893e0b
Again, we assume if you're reading this document, you should already have
893e0b
kexec-tools installed. If not, you install it via the following command:
893e0b
893e0b
    # yum install kexec-tools
893e0b
893e0b
To be able to do much of anything interesting in the way of debug analysis,
893e0b
you'll also need to install the kernel-debuginfo package, of the same arch
893e0b
as your running kernel, and the crash utility:
893e0b
893e0b
    # yum --enablerepo=\*debuginfo install kernel-debuginfo.$(uname -m) crash
893e0b
893e0b
Next up, we need to modify some boot parameters to reserve a chunk of memory for
893e0b
the capture kernel. With the help of grubby, it's very easy to append
893e0b
"crashkernel=128M" to the end of your kernel boot parameters. Note that the X
893e0b
values are such that X = the amount of memory to reserve for the capture kernel.
893e0b
And based on arch and system configuration, one might require more than 128M to
893e0b
be reserved for kdump. One need to experiment and test kdump, if 128M is not
893e0b
sufficient, try reserving more memory.
893e0b
893e0b
   # grubby --args="crashkernel=128M" --update-kernel=/boot/vmlinuz-`uname -r`
893e0b
893e0b
Note that there is an alternative form in which to specify a crashkernel
893e0b
memory reservation, in the event that more control is needed over the size and
893e0b
placement of the reserved memory.  The format is:
893e0b
893e0b
crashkernel=range1:size1[,range2:size2,...][@offset]
893e0b
893e0b
Where range<n> specifies a range of values that are matched against the amount
893e0b
of physical RAM present in the system, and the corresponding size<n> value
893e0b
specifies the amount of kexec memory to reserve.  For example:
893e0b
893e0b
crashkernel=512M-2G:64M,2G-:128M
893e0b
893e0b
This line tells kexec to reserve 64M of ram if the system contains between
893e0b
512M and 2G of physical memory.  If the system contains 2G or more of physical
893e0b
memory, 128M should be reserved.
893e0b
603de6
Besides, since kdump needs to access /proc/kallsyms during a kernel
603de6
loading if KASLR is enabled, check /proc/sys/kernel/kptr_restrict to
603de6
make sure that the content of /proc/kallsyms is exposed correctly.
603de6
We recommend to set the value of kptr_restrict to '1'. Otherwise
603de6
capture kernel loading could fail.
603de6
893e0b
After making said changes, reboot your system, so that the X MB of memory is
893e0b
left untouched by the normal system, reserved for the capture kernel. Take note
893e0b
that the output of 'free -m' will show X MB less memory than without this
893e0b
parameter, which is expected. You may be able to get by with less than 128M, but
893e0b
testing with only 64M has proven unreliable of late. On ia64, as much as 512M
893e0b
may be required.
893e0b
893e0b
Now that you've got that reserved memory region set up, you want to turn on
893e0b
the kdump init script:
893e0b
893e0b
    # chkconfig kdump on
893e0b
893e0b
Then, start up kdump as well:
893e0b
893e0b
    # systemctl start kdump.service
893e0b
893e0b
This should load your kernel-kdump image via kexec, leaving the system ready
893e0b
to capture a vmcore upon crashing. To test this out, you can force-crash
893e0b
your system by echo'ing a c into /proc/sysrq-trigger:
893e0b
893e0b
    # echo c > /proc/sysrq-trigger
893e0b
893e0b
You should see some panic output, followed by the system restarting into
893e0b
the kdump kernel. When the boot process gets to the point where it starts
893e0b
the kdump service, your vmcore should be copied out to disk (by default,
893e0b
in /var/crash/<YYYY-MM-DD-HH:MM>/vmcore), then the system rebooted back into
893e0b
your normal kernel.
893e0b
893e0b
Once back to your normal kernel, you can use the previously installed crash
893e0b
kernel in conjunction with the previously installed kernel-debuginfo to
893e0b
perform postmortem analysis:
893e0b
893e0b
    # crash /usr/lib/debug/lib/modules/2.6.17-1.2621.el5/vmlinux
893e0b
    /var/crash/2006-08-23-15:34/vmcore
893e0b
893e0b
    crash> bt
893e0b
893e0b
and so on...
893e0b
603de6
Notes:
603de6
603de6
When kdump starts, the kdump kernel is loaded together with the kdump
603de6
initramfs. To save memory usage and disk space, the kdump initramfs is
603de6
generated strictly against the system it will run on, and contains the
603de6
minimum set of kernel modules and utilities to boot the machine to a stage
603de6
where the dump target could be mounted.
603de6
603de6
With kdump service enabled, kdumpctl will try to detect possible system
603de6
change and rebuild the kdump initramfs if needed. But it can not guarantee
603de6
to cover every possible case. So after a hardware change, disk migration,
603de6
storage setup update or any similar system level changes, it's highly
603de6
recommended to rebuild the initramfs manually with following command:
603de6
603de6
    # kdumpctl rebuild
603de6
893e0b
Saving vmcore-dmesg.txt
893e0b
----------------------
893e0b
Kernel log bufferes are one of the most important information available
893e0b
in vmcore. Now before saving vmcore, kernel log bufferes are extracted
893e0b
from /proc/vmcore and saved into a file vmcore-dmesg.txt. After
893e0b
vmcore-dmesg.txt, vmcore is saved. Destination disk and directory for
893e0b
vmcore-dmesg.txt is same as vmcore. Note that kernel log buffers will
893e0b
not be available if dump target is raw device.
893e0b
893e0b
Dump Triggering methods:
893e0b
893e0b
This section talks about the various ways, other than a Kernel Panic, in which
893e0b
Kdump can be triggered. The following methods assume that Kdump is configured
893e0b
on your system, with the scripts enabled as described in the section above.
893e0b
893e0b
1) AltSysRq C
893e0b
893e0b
Kdump can be triggered with the combination of the 'Alt','SysRq' and 'C'
893e0b
keyboard keys. Please refer to the following link for more details:
893e0b
603de6
https://access.redhat.com/solutions/2023
893e0b
893e0b
In addition, on PowerPC boxes, Kdump can also be triggered via Hardware
893e0b
Management Console(HMC) using 'Ctrl', 'O' and 'C' keyboard keys.
893e0b
893e0b
2) NMI_WATCHDOG
893e0b
893e0b
In case a machine has a hard hang, it is quite possible that it does not
893e0b
respond to keyboard interrupts. As a result 'Alt-SysRq' keys will not help
893e0b
trigger a dump. In such scenarios Nmi Watchdog feature can prove to be useful.
893e0b
The following link has more details on configuring Nmi watchdog option.
893e0b
603de6
https://access.redhat.com/solutions/125103
893e0b
893e0b
Once this feature has been enabled in the kernel, any lockups will result in an
893e0b
OOPs message to be generated, followed by Kdump being triggered.
893e0b
893e0b
3) Kernel OOPs
893e0b
893e0b
If we want to generate a dump everytime the Kernel OOPses, we can achieve this
893e0b
by setting the 'Panic On OOPs' option as follows:
893e0b
893e0b
    # echo 1 > /proc/sys/kernel/panic_on_oops
893e0b
893e0b
This is enabled by default on RHEL5.
893e0b
893e0b
4) NMI(Non maskable interrupt) button
893e0b
893e0b
In cases where the system is in a hung state, and is not accepting keyboard
893e0b
interrupts, using NMI button for triggering Kdump can be very useful. NMI
893e0b
button is present on most of the newer x86 and x86_64 machines. Please refer
893e0b
to the User guides/manuals to locate the button, though in most occasions it
893e0b
is not very well documented. In most cases it is hidden behind a small hole
893e0b
on the front or back panel of the machine. You could use a toothpick or some
893e0b
other non-conducting probe to press the button.
893e0b
893e0b
For example, on the IBM X series 366 machine, the NMI button is located behind
893e0b
a small hole on the bottom center of the rear panel.
893e0b
893e0b
To enable this method of dump triggering using NMI button, you will need to set
893e0b
the 'unknown_nmi_panic' option as follows:
893e0b
893e0b
   # echo 1 > /proc/sys/kernel/unknown_nmi_panic
893e0b
893e0b
5) PowerPC specific methods:
893e0b
893e0b
On IBM PowerPC machines, issuing a soft reset invokes the XMON debugger(if
893e0b
XMON is configured). To configure XMON one needs to compile the kernel with
893e0b
the CONFIG_XMON and CONFIG_XMON_DEFAULT options, or by compiling with
893e0b
CONFIG_XMON and booting the kernel with xmon=on option.
893e0b
893e0b
Following are the ways to remotely issue a soft reset on PowerPC boxes, which
893e0b
would drop you to XMON. Pressing a 'X' (capital alphabet X) followed by an
893e0b
'Enter' here will trigger the dump.
893e0b
893e0b
5.1) HMC
893e0b
893e0b
Hardware Management Console(HMC) available on Power4 and Power5 machines allow
893e0b
partitions to be reset remotely. This is specially useful in hang situations
893e0b
where the system is not accepting any keyboard inputs.
893e0b
893e0b
Once you have HMC configured, the following steps will enable you to trigger
893e0b
Kdump via a soft reset:
893e0b
893e0b
On Power4
893e0b
  Using GUI
893e0b
893e0b
    * In the right pane, right click on the partition you wish to dump.
893e0b
    * Select "Operating System->Reset".
893e0b
    * Select "Soft Reset".
893e0b
    * Select "Yes".
893e0b
893e0b
  Using HMC Commandline
893e0b
893e0b
    # reset_partition -m <machine> -p <partition> -t soft
893e0b
893e0b
On Power5
893e0b
  Using GUI
893e0b
893e0b
    * In the right pane, right click on the partition you wish to dump.
893e0b
    * Select "Restart Partition".
893e0b
    * Select "Dump".
893e0b
    * Select "OK".
893e0b
893e0b
  Using HMC Commandline
893e0b
893e0b
    # chsysstate -m <managed system name> -n <lpar name> -o dumprestart -r lpar
893e0b
893e0b
5.2) Blade Management Console for Blade Center
893e0b
893e0b
To initiate a dump operation, go to Power/Restart option under "Blade Tasks" in
893e0b
the Blade Management Console. Select the corresponding blade for which you want
893e0b
to initate the dump and then click "Restart blade with NMI". This issues a
893e0b
system reset and invokes xmon debugger.
893e0b
893e0b
893e0b
Advanced Setups:
893e0b
893e0b
In addition to being able to capture a vmcore to your system's local file
893e0b
system, kdump can be configured to capture a vmcore to a number of other
893e0b
locations, including a raw disk partition, a dedicated file system, an NFS
893e0b
mounted file system, or a remote system via ssh/scp. Additional options
893e0b
exist for specifying the relative path under which the dump is captured,
893e0b
what to do if the capture fails, and for compressing and filtering the dump
893e0b
(so as to produce smaller, more manageable, vmcore files).
893e0b
893e0b
In theory, dumping to a location other than the local file system should be
893e0b
safer than kdump's default setup, as its possible the default setup will try
893e0b
dumping to a file system that has become corrupted. The raw disk partition and
893e0b
dedicated file system options allow you to still dump to the local system,
893e0b
but without having to remount your possibly corrupted file system(s),
893e0b
thereby decreasing the chance a vmcore won't be captured. Dumping to an
893e0b
NFS server or remote system via ssh/scp also has this advantage, as well
893e0b
as allowing for the centralization of vmcore files, should you have several
893e0b
systems from which you'd like to obtain vmcore files. Of course, note that
893e0b
these configurations could present problems if your network is unreliable.
893e0b
893e0b
Advanced setups are configured via modifications to /etc/kdump.conf,
893e0b
which out of the box, is fairly well documented itself. Any alterations to
893e0b
/etc/kdump.conf should be followed by a restart of the kdump service, so
893e0b
the changes can be incorporated in the kdump initrd. Restarting the kdump
893e0b
service is as simple as '/sbin/systemctl restart kdump.service'.
893e0b
893e0b
893e0b
Note that kdump.conf is used as a configuration mechanism for capturing dump
893e0b
files from the initramfs (in the interests of safety), the root file system is
893e0b
mounted, and the init process is started, only as a last resort if the
893e0b
initramfs fails to capture the vmcore.  As such, configuration made in
893e0b
/etc/kdump.conf is only applicable to capture recorded in the initramfs.  If
893e0b
for any reason the init process is started on the root file system, only a
893e0b
simple copying of the vmcore from /proc/vmcore to /var/crash/$DATE/vmcore will
893e0b
be preformed.
893e0b
893e0b
For both local filesystem and nfs dump the dump target must be mounted before
893e0b
building kdump initramfs. That means one needs to put an entry for the dump
893e0b
file system in /etc/fstab so that after reboot when kdump service starts,
893e0b
it can find the dump target and build initramfs instead of failing.
893e0b
Usually the dump target should be used only for kdump. If you worry about
893e0b
someone uses the filesystem for something else other than dumping vmcore
893e0b
you can mount it as read-only. Mkdumprd will still remount it as read-write
893e0b
for creating dump directory and will move it back to read-only afterwards.
893e0b
893e0b
Raw partition
893e0b
893e0b
Raw partition dumping requires that a disk partition in the system, at least
893e0b
as large as the amount of memory in the system, be left unformatted. Assuming
893e0b
/dev/vg/lv_kdump is left unformatted, kdump.conf can be configured with
893e0b
'raw /dev/vg/lv_kdump', and the vmcore file will be copied via dd directly
893e0b
onto partition /dev/vg/lv_kdump. Restart the kdump service via
893e0b
'/sbin/systemctl restart kdump.service' to commit this change to your kdump
893e0b
initrd. Dump target should be persistent device name, such as lvm or device
893e0b
mapper canonical name.
893e0b
893e0b
Dedicated file system
893e0b
893e0b
Similar to raw partition dumping, you can format a partition with the file
893e0b
system of your choice, Again, it should be at least as large as the amount
893e0b
of memory in the system. Assuming it should be at least as large as the
893e0b
amount of memory in the system. Assuming /dev/vg/lv_kdump has been
893e0b
formatted ext4, specify 'ext4 /dev/vg/lv_kdump' in kdump.conf, and a
893e0b
vmcore file will be copied onto the file system after it has been mounted.
893e0b
Dumping to a dedicated partition has the advantage that you can dump multiple
893e0b
vmcores to the file system, space permitting, without overwriting previous ones,
893e0b
as would be the case in a raw partition setup. Restart the kdump service via
893e0b
'/sbin/systemctl restart kdump.service' to commit this change to
893e0b
your kdump initrd.  Note that for local file systems ext4 and ext2 are
893e0b
supported as dumpable targets.  Kdump will not prevent you from specifying
893e0b
other filesystems, and they will most likely work, but their operation
893e0b
cannot be guaranteed.  for instance specifying a vfat filesystem or msdos
893e0b
filesystem will result in a successful load of the kdump service, but during
893e0b
crash recovery, the dump will fail if the system has more than 2GB of memory
893e0b
(since vfat and msdos filesystems do not support more than 2GB files).
893e0b
Be careful of your filesystem selection when using this target.
893e0b
893e0b
It is recommended to use persistent device names or UUID/LABEL for file system
893e0b
dumps. One example of persistent device is /dev/vg/<devname>.
893e0b
893e0b
NFS mount
893e0b
893e0b
Dumping over NFS requires an NFS server configured to export a file system
893e0b
with full read/write access for the root user. All operations done within
893e0b
the kdump initial ramdisk are done as root, and to write out a vmcore file,
893e0b
we obviously must be able to write to the NFS mount. Configuring an NFS
893e0b
server is outside the scope of this document, but either the no_root_squash
893e0b
or anonuid options on the NFS server side are likely of interest to permit
893e0b
the kdump initrd operations write to the NFS mount as root.
893e0b
893e0b
Assuming your're exporting /dump on the machine nfs-server.example.com,
893e0b
once the mount is properly configured, specify it in kdump.conf, via
893e0b
'nfs nfs-server.example.com:/dump'. The server portion can be specified either
893e0b
by host name or IP address. Following a system crash, the kdump initrd will
893e0b
mount the NFS mount and copy out the vmcore to your NFS server. Restart the
893e0b
kdump service via '/sbin/systemctl restart kdump.service' to commit this change
893e0b
to your kdump initrd.
893e0b
893e0b
Special mount via "dracut_args"
893e0b
893e0b
You can utilize "dracut_args" to pass "--mount" to kdump, see dracut manpage
893e0b
about the format of "--mount" for details. If there is any "--mount" specified
893e0b
via "dracut_args", kdump will build it as the mount target without doing any
893e0b
validation (mounting or checking like mount options, fs size, save path, etc),
893e0b
so you must test it to ensure all the correctness. You cannot use other targets
893e0b
in /etc/kdump.conf if you use "--mount" in "dracut_args". You also cannot specify
893e0b
mutliple "--mount" targets via "dracut_args".
893e0b
893e0b
One use case of "--mount" in "dracut_args" is you do not want to mount dump target
893e0b
before kdump service startup, for example, to reduce the burden of the shared nfs
893e0b
server. Such as the example below:
893e0b
dracut_args --mount "192.168.1.1:/share /mnt/test nfs4 defaults"
893e0b
893e0b
NOTE:
893e0b
- <mountpoint> must be specified as an absolute path.
893e0b
893e0b
Remote system via ssh/scp
893e0b
893e0b
Dumping over ssh/scp requires setting up passwordless ssh keys for every
893e0b
machine you wish to have dump via this method. First up, configure kdump.conf
893e0b
for ssh/scp dumping, adding a config line of 'ssh user@server', where 'user'
893e0b
can be any user on the target system you choose, and 'server' is the host
893e0b
name or IP address of the target system. Using a dedicated, restricted user
893e0b
account on the target system is recommended, as there will be keyless ssh
893e0b
access to this account.
893e0b
893e0b
Once kdump.conf is appropriately configured, issue the command
893e0b
'kdumpctl propagate' to automatically set up the ssh host keys and transmit
893e0b
the necessary bits to the target server. You'll have to type in 'yes'
893e0b
to accept the host key for your targer server if this is the first time
893e0b
you've connected to it, and then input the target system user's password
893e0b
to send over the necessary ssh key file. Restart the kdump service via
893e0b
'/sbin/systemctl restart kdump.service' to commit this change to your kdump initrd.
893e0b
893e0b
Path
893e0b
====
893e0b
"path" represents the file system path in which vmcore will be saved. In
893e0b
fact kdump creates a directory $hostip-$date with-in "path" and saves
893e0b
vmcore there. So practically dump is saved in $path/$hostip-$date/. To
893e0b
simplify discussion further, if we say dump will be saved in $path, it
893e0b
is implied that kdump will create another directory inside path and
893e0b
save vmcore there.
893e0b
893e0b
If a dump target is specified in kdump.conf, then "path" is relative to the
893e0b
specified dump target. For example, if dump target is "ext4 /dev/sda", then
893e0b
dump will be saved in "$path" directory on /dev/sda.
893e0b
893e0b
Same is the case for nfs dump. If user specified "nfs foo.com:/export/tmp/"
893e0b
as dump target, then dump will effectively be saved in
893e0b
"foo.com:/export/tmp/var/crash/" directory.
893e0b
893e0b
Interpretation of path changes a bit if user has not specified a dump
893e0b
target explicitly in kdump.conf. In this case, "path" represents the
893e0b
absolute path from root. And dump target and adjusted path are arrived
893e0b
at automatically depending on what's mounted in the current system.
893e0b
893e0b
Following are few examples.
893e0b
893e0b
path /var/crash/
893e0b
----------------
893e0b
Assuming there is no disk mounted on /var/ or on /var/crash, dump will
893e0b
be saved on disk backing rootfs in directory /var/crash.
893e0b
893e0b
path /var/crash/ (A separate disk mounted on /var)
893e0b
--------------------------------------------------
893e0b
Say a disk /dev/sdb is mouted on /var. In this case dump target will
893e0b
become /dev/sdb and path will become "/crash" and dump will be saved
893e0b
on "sdb:/crash/" directory.
893e0b
893e0b
path /var/crash/ (NFS mounted on /var)
893e0b
-------------------------------------
893e0b
Say foo.com:/export/tmp is mounted on /var. In this case dump target is
893e0b
nfs server and path will be adjusted to "/crash" and dump will be saved to
893e0b
foo.com:/export/tmp/crash/ directory.
893e0b
893e0b
Kdump boot directory
893e0b
====================
893e0b
Usually kdump kernel is the same as 1st kernel. So kdump will try to find
893e0b
kdump kernel under /boot according to /proc/cmdline. E.g we execute below
893e0b
command and get an output:
893e0b
	cat /proc/cmdline
893e0b
	BOOT_IMAGE=/xxx/vmlinuz-3.yyy.zzz  root=xxxx .....
893e0b
Then kdump kernel will be /boot/xxx/vmlinuz-3.yyy.zzz.
893e0b
However a variable KDUMP_BOOTDIR in /etc/sysconfig/kdump is provided to
893e0b
user if kdump kernel is put in a different directory.
893e0b
893e0b
Kdump Post-Capture Executable
893e0b
893e0b
It is possible to specify a custom script or binary you wish to run following
893e0b
an attempt to capture a vmcore. The executable is passed an exit code from
893e0b
the capture process, which can be used to trigger different actions from
893e0b
within your post-capture executable.
893e0b
893e0b
Kdump Pre-Capture Executable
893e0b
893e0b
It is possible to specify a custom script or binary you wish to run before
893e0b
capturing a vmcore. Exit status of this binary is interpreted:
893e0b
0 - continue with dump process as usual
893e0b
non 0 - reboot the system
893e0b
893e0b
Extra Binaries
893e0b
893e0b
If you have specific binaries or scripts you want to have made available
893e0b
within your kdump initrd, you can specify them by their full path, and they
893e0b
will be included in your kdump initrd, along with all dependent libraries.
893e0b
This may be particularly useful for those running post-capture scripts that
893e0b
rely on other binaries.
893e0b
893e0b
Extra Modules
893e0b
893e0b
By default, only the bare minimum of kernel modules will be included in your
893e0b
kdump initrd. Should you wish to capture your vmcore files to a non-boot-path
893e0b
storage device, such as an iscsi target disk or clustered file system, you may
893e0b
need to manually specify additional kernel modules to load into your kdump
893e0b
initrd.
893e0b
603de6
Failure action
893e0b
==============
603de6
Failure action specifies what to do when dump to configured dump target
603de6
fails. By default, failure action is "reboot" and that is system reboots
893e0b
if attempt to save dump to dump target fails.
893e0b
603de6
There are other failure actions available though.
893e0b
893e0b
- dump_to_rootfs
893e0b
	This option tries to mount root and save dump on root filesystem
893e0b
	in a path specified by "path". This option will generally make
893e0b
	sense when dump target is not root filesystem. For example, if
893e0b
	dump is being saved over network using "ssh" then one can specify
603de6
	failure action to "dump_to_rootfs" to try saving dump to root
603de6
	filesystem if dump over network fails.
893e0b
893e0b
- shell
893e0b
	Drop into a shell session inside initramfs.
893e0b
- halt
893e0b
	Halt system after failure
893e0b
- poweroff
893e0b
	Poweroff system after failure.
893e0b
893e0b
Compression and filtering
893e0b
893e0b
The 'core_collector' parameter in kdump.conf allows you to specify a custom
893e0b
dump capture method. The most common alternate method is makedumpfile, which
893e0b
is a dump filtering and compression utility provided with kexec-tools. On
893e0b
some architectures, it can drastically reduce the size of your vmcore files,
893e0b
which becomes very useful on systems with large amounts of memory.
893e0b
893e0b
A typical setup is 'core_collector makedumpfile -F -l --message-level 1 -d 31',
893e0b
but check the output of '/sbin/makedumpfile --help' for a list of all available
893e0b
options (-i and -g don't need to be specified, they're automatically taken care
893e0b
of). Note that use of makedumpfile requires that the kernel-debuginfo package
893e0b
corresponding with your running kernel be installed.
893e0b
893e0b
Core collector command format depends on dump target type. Typically for
893e0b
filesystem (local/remote), core_collector should accept two arguments.
893e0b
First one is source file and second one is target file. For ex.
893e0b
893e0b
ex1.
893e0b
---
893e0b
core_collector "cp --sparse=always"
893e0b
893e0b
Above will effectively be translated to:
893e0b
893e0b
cp --sparse=always /proc/vmcore <dest-path>/vmcore
893e0b
893e0b
ex2.
893e0b
---
893e0b
core_collector "makedumpfile -l --message-level 1 -d 31"
893e0b
893e0b
Above will effectively be translated to:
893e0b
893e0b
makedumpfile -l --message-level 1 -d 31 /proc/vmcore <dest-path>/vmcore
893e0b
893e0b
893e0b
For dump targets like raw and ssh, in general, core collector should expect
893e0b
one argument (source file) and should output the processed core on standard
893e0b
output (There is one exception of "scp", discussed later). This standard
893e0b
output will be saved to destination using appropriate commands.
893e0b
893e0b
raw dumps core_collector examples:
893e0b
---------
893e0b
ex3.
893e0b
---
893e0b
core_collector "cat"
893e0b
893e0b
Above will effectively be translated to.
893e0b
893e0b
cat /proc/vmcore | dd of=<target-device>
893e0b
893e0b
ex4.
893e0b
---
893e0b
core_collector "makedumpfile -F -l --message-level 1 -d 31"
893e0b
893e0b
Above will effectively be translated to.
893e0b
893e0b
makedumpfile -F -l --message-level 1 -d 31 | dd of=<target-device>
893e0b
893e0b
ssh dumps core_collector examples:
893e0b
---------
893e0b
ex5.
893e0b
---
893e0b
core_collector "cat"
893e0b
893e0b
Above will effectively be translated to.
893e0b
893e0b
cat /proc/vmcore | ssh <options> <remote-location> "dd of=path/vmcore"
893e0b
893e0b
ex6.
893e0b
---
893e0b
core_collector "makedumpfile -F -l --message-level 1 -d 31"
893e0b
893e0b
Above will effectively be translated to.
893e0b
893e0b
makedumpfile -F -l --message-level 1 -d 31 | ssh <options> <remote-location> "dd of=path/vmcore"
893e0b
893e0b
There is one exception to standard output rule for ssh dumps. And that is
893e0b
scp. As scp can handle ssh destinations for file transfers, one can
893e0b
specify "scp" as core collector for ssh targets (no output on stdout).
893e0b
893e0b
ex7.
893e0b
----
893e0b
core_collector "scp"
893e0b
893e0b
Above will effectively be translated to.
893e0b
893e0b
scp /proc/vmcore <user@host>:path/vmcore
893e0b
893e0b
About default core collector
893e0b
----------------------------
893e0b
Default core_collector for ssh/raw dump is:
893e0b
"makedumpfile -F -l --message-level 1 -d 31".
893e0b
Default core_collector for other targets is:
893e0b
"makedumpfile -l --message-level 1 -d 31".
893e0b
893e0b
Even if core_collector option is commented out in kdump.conf, makedumpfile
893e0b
is default core collector and kdump uses it internally.
893e0b
893e0b
If one does not want makedumpfile as default core_collector, then they
893e0b
need to specify one using core_collector option to change the behavior.
893e0b
893e0b
Note: If "makedumpfile -F" is used then you will get a flattened format
893e0b
vmcore.flat, you will need to use "makedumpfile -R" to rearrange the
893e0b
dump data from stdard input to a normal dumpfile (readable with analysis
893e0b
tools).
893e0b
For example: "makedumpfile -R vmcore < vmcore.flat"
893e0b
893e0b
Caveats:
893e0b
893e0b
Console frame-buffers and X are not properly supported. If you typically run
893e0b
with something along the lines of "vga=791" in your kernel config line or
893e0b
have X running, console video will be garbled when a kernel is booted via
893e0b
kexec. Note that the kdump kernel should still be able to create a dump,
893e0b
and when the system reboots, video should be restored to normal.
893e0b
893e0b
893e0b
Notes on resetting video:
893e0b
893e0b
Video is a notoriously difficult issue with kexec.  Video cards contain ROM code
893e0b
that controls their initial configuration and setup.  This code is nominally
893e0b
accessed and executed from the Bios, and otherwise not safely executable. Since
893e0b
the purpose of kexec is to reboot the system without re-executing the Bios, it
893e0b
is rather difficult if not impossible to reset video cards with kexec.  The
893e0b
result is, that if a system crashes while running in a graphical mode (i.e.
893e0b
running X), the screen may appear to become 'frozen' while the dump capture is
893e0b
taking place.  A serial console will of course reveal that the system is
893e0b
operating and capturing a vmcore image, but a casual observer will see the
893e0b
system as hung until the dump completes and a true reboot is executed.
893e0b
893e0b
There are two possiblilties to work around this issue.  One is by adding
893e0b
--reset-vga to the kexec command line options in /etc/sysconfig/kdump.  This
893e0b
tells kdump to write some reasonable default values to the video card register
893e0b
file, in the hopes of returning it to a text mode such that boot messages are
893e0b
visible on the screen.  It does not work with all video cards however.
893e0b
Secondly, it may be worth trying to add vga15fb.ko to the extra_modules list in
893e0b
/etc/kdump.conf.  This will attempt to use the video card in framebuffer mode,
893e0b
which can blank the screen prior to the start of a dump capture.
893e0b
893e0b
Notes on rootfs mount:
893e0b
Dracut is designed to mount rootfs by default. If rootfs mounting fails it
893e0b
will refuse to go on. So kdump leaves rootfs mounting to dracut currently.
893e0b
We make the assumtion that proper root= cmdline is being passed to dracut
893e0b
initramfs for the time being. If you need modify "KDUMP_COMMANDLINE=" in
893e0b
/etc/sysconfig/kdump, you will need to make sure that appropriate root=
893e0b
options are copied from /proc/cmdline. In general it is best to append
893e0b
command line options using "KDUMP_COMMANDLINE_APPEND=" instead of replacing
893e0b
the original command line completely.
893e0b
893e0b
Notes on watchdog module handling:
893e0b
893e0b
If a watchdog is active in first kernel then, we must have it's module
893e0b
loaded in crash kernel, so that either watchdog is deactivated or started
893e0b
being kicked in second kernel. Otherwise, we might face watchdog reboot
893e0b
when vmcore is being saved. When dracut watchdog module is enabled, it
893e0b
installs kernel watchdog module of active watchdog device in initrd.
893e0b
kexec-tools always add "-a watchdog" to the dracut_args if there exists at
893e0b
least one active watchdog and user has not added specifically "-o watchdog"
893e0b
in dracut_args of kdump.conf. If a watchdog module (such as hp_wdt) has
893e0b
not been written in watchdog-core framework then this option will not have
893e0b
any effect and module will not be added. Please note that only systemd
893e0b
watchdog daemon is supported as watchdog kick application.
893e0b
603de6
Notes for disk images:
603de6
603de6
Kdump initramfs is a critical component for capturing the crash dump.
603de6
But it's strictly generated for the machine it will run on, and have
603de6
no generality. If you install a new machine with a previous disk image
603de6
(eg. VMs created with disk image or snapshot), kdump could be broken
603de6
easily due to hardware changes or disk ID changes. So it's strongly
603de6
recommended to not include the kdump initramfs in the disk image in the
603de6
first place, this helps to save space, and kdumpctl will build the
603de6
initramfs automatically if it's missing. If you have already installed
603de6
a machine with a disk image which have kdump initramfs embedded, you
603de6
should rebuild the initramfs using "kdumpctl rebuild" command manually,
603de6
or else kdump may not work as expeceted.
603de6
603de6
Notes on encrypted dump target:
603de6
603de6
Currently, kdump is not working well with encrypted dump target.
603de6
First, user have to give the password manually in capture kernel,
603de6
so a working interactive terminal is required in the capture kernel.
603de6
And another major issue is that an OOM problem will occur with certain
603de6
encryption setup. For example, the default setup for LUKS2 will use a
603de6
memory hard key derivation function to mitigate brute force attach,
603de6
it's impossible to reduce the memory usage for mounting the encrypted
603de6
target. In such case, you have to either reserved enough memory for
603de6
crash kernel according, or update your encryption setup.
603de6
It's recommanded to use a non-encrypted target (eg. remote target)
603de6
instead.
603de6
603de6
Notes on device dump:
603de6
603de6
Device dump allows drivers to append dump data to vmcore, so you can
603de6
collect driver specified debug info. The drivers could append the
603de6
data without any limit, and the data is stored in memory, this may
603de6
bring a significant memory stress. So device dump is disabled by default
603de6
by passing "novmcoredd" command line option to the kdump capture kernel.
603de6
If you want to collect debug data with device dump, you need to modify
603de6
"KDUMP_COMMANDLINE_APPEND=" value in /etc/sysconfig/kdump and remove the
603de6
"novmcoredd" option. You also need to increase the "crashkernel=" value
603de6
accordingly in case of OOM issue.
603de6
Besides, kdump initramfs won't automatically include the device drivers
603de6
which support device dump, only device drivers that are required for
603de6
the dump target setup will be included. To ensure the device dump data
603de6
will be included in the vmcore, you need to force include related
603de6
device drivers by using "extra_modules" option in /etc/kdump.conf
603de6
893e0b
Parallel Dumping Operation
893e0b
==========================
893e0b
Kexec allows kdump using multiple cpus. So parallel feature can accelerate
893e0b
dumping substantially, especially in executing compression and filter.
893e0b
For example:
893e0b
893e0b
	1."makedumpfile -c --num-threads [THREAD_NUM] /proc/vmcore dumpfile"
893e0b
	2."makedumpfile -c /proc/vmcore dumpfile",
893e0b
893e0b
	1 has better performance than 2, if THREAD_NUM is larger than two
893e0b
	and the usable cpus number is larger than THREAD_NUM.
893e0b
893e0b
Notes on how to use multiple cpus on a capture kernel on x86 system:
893e0b
893e0b
Make sure that you are using a kernel that supports disable_cpu_apicid
893e0b
kernel option as a capture kernel, which is needed to avoid x86 specific
893e0b
hardware issue (*). The disable_cpu_apicid kernel option is automatically
893e0b
appended by kdumpctl script and is ignored if the kernel doesn't support it.
893e0b
893e0b
You need to specify how many cpus to be used in a capture kernel by specifying
893e0b
the number of cpus in nr_cpus kernel option in /etc/sysconfig/kdump. nr_cpus
893e0b
is 1 at default.
893e0b
893e0b
You should use necessary and sufficient number of cpus on a capture kernel.
893e0b
Warning: Don't use too many cpus on a capture kernel, or the capture kernel
893e0b
may lead to panic due to Out Of Memory.
893e0b
893e0b
(*) Without disable_cpu_apicid kernel option, capture kernel may lead to
893e0b
hang, system reset or power-off at boot, depending on your system and runtime
893e0b
situation at the time of crash.
893e0b
893e0b
Debugging Tips
893e0b
--------------
893e0b
- One can drop into a shell before/after saving vmcore with the help of
893e0b
  using kdump_pre/kdump_post hooks. Use following in one of the pre/post
893e0b
  scripts to drop into a shell.
893e0b
893e0b
  #!/bin/bash
893e0b
  _ctty=/dev/ttyS0
893e0b
  setsid /bin/sh -i -l 0<>$_ctty 1<>$_ctty 2<>$_ctty
893e0b
893e0b
  One might have to change the terminal depending on what they are using.
893e0b
893e0b
- Serial console logging for virtual machines
893e0b
893e0b
  I generally use "virsh console <domain-name>" to get to serial console.
893e0b
  I noticed after dump saving system reboots and when grub menu shows up
893e0b
  some of the previously logged messages are no more there. That means
893e0b
  any important debugging info at the end will be lost.
893e0b
893e0b
  One can log serial console as follows to make sure messages are not lost.
893e0b
893e0b
  virsh ttyconsole <domain-name>
893e0b
  ln -s <name-of-tty> /dev/modem
893e0b
  minicom -C /tmp/console-logs
893e0b
893e0b
  Now minicom should be logging serial console in file console-logs.
893e0b
893e0b