|
 |
a60cd7 |
From 95e9590bfee2df447c8f4c0fd799e8c514beca80 Mon Sep 17 00:00:00 2001
|
|
|
a60cd7 |
From: Denys Vlasenko <dvlasenk@redhat.com>
|
|
|
a60cd7 |
Date: Tue, 10 Dec 2013 13:07:35 +0100
|
|
|
a60cd7 |
Subject: [ABRT PATCH 24/27] doc/MCE_readme.txt: new file - documentation about
|
|
|
a60cd7 |
MCE handling
|
|
|
a60cd7 |
|
|
|
a60cd7 |
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
|
|
|
a60cd7 |
|
|
|
a60cd7 |
Related to rhbz#1032077
|
|
|
a60cd7 |
|
|
|
a60cd7 |
Signed-off-by: Jakub Filak <jfilak@redhat.com>
|
|
|
a60cd7 |
---
|
|
|
a60cd7 |
doc/MCE_readme.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
|
|
a60cd7 |
1 file changed, 86 insertions(+)
|
|
|
a60cd7 |
create mode 100644 doc/MCE_readme.txt
|
|
|
a60cd7 |
|
|
|
a60cd7 |
diff --git a/doc/MCE_readme.txt b/doc/MCE_readme.txt
|
|
|
a60cd7 |
new file mode 100644
|
|
|
a60cd7 |
index 0000000..ed5b627
|
|
|
a60cd7 |
--- /dev/null
|
|
|
a60cd7 |
+++ b/doc/MCE_readme.txt
|
|
|
a60cd7 |
@@ -0,0 +1,86 @@
|
|
|
a60cd7 |
+ Background
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+MCEs can be fatal (they panic kernel) or not.
|
|
|
a60cd7 |
+Fatal MCE are delivered as exception#18.
|
|
|
a60cd7 |
+Non-fatal ones sometimes are delivered as exception#18; other times
|
|
|
a60cd7 |
+they are silently recorded in magic MSRs, CPU is not alerted.
|
|
|
a60cd7 |
+Linux kernel periodically (up to 5 mins interval) reads those MSRs
|
|
|
a60cd7 |
+and if MCE is seen there, it is piped in binary form through
|
|
|
a60cd7 |
+/dev/mcelog to whoever listens on it. (Such as mcelog tool in
|
|
|
a60cd7 |
+--daemon mode; but cat
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+"Machine Check Exception:" message is printed *only* by fatal MCEs.
|
|
|
a60cd7 |
+It will be caught as vmcore if kdump is configured.
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+Non-fatal MCEs have "[Hardware Error]: Machine check events logged"
|
|
|
a60cd7 |
+message in kernel log.
|
|
|
a60cd7 |
+When /dev/mcelog is read, *no additional kernel log messages appear*.
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+> Are those magic MSR registers cleared when read via /dev/mcelog?
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+Yes.
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+> Without mcelog utility, we can directly read only binary form, right?
|
|
|
a60cd7 |
+> Not nice, but still useful, right?
|
|
|
a60cd7 |
+> (could be transferred to nice text form on other machine).
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+No, raw /dev/mcelog data is not easy to interpret on other machine.
|
|
|
a60cd7 |
+In fact, it can't be used by mcelog tool even on the same machine.
|
|
|
a60cd7 |
+Technical reason is that mcelog uses an obscure ioctl on /dev/mcelog
|
|
|
a60cd7 |
+in order to know the size of binary blob with MCE information.
|
|
|
a60cd7 |
+When run on a file, ioctl fails, and mcelog bombs out.
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+Looks like without mcelog running and processing /dev/mcelog data,
|
|
|
a60cd7 |
+non-fatal MCE's can't be easily decoded with currently existing tools.
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+mcelog tool can be configured to write log to /var/log/mcelog
|
|
|
a60cd7 |
+(RHEL6 does that) or to syslog (RHEL7 does that).
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+ How ABRT catches MCEs
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+Fatal MCEs are caught as any fatal kernel panic is caught - as a vmcore.
|
|
|
a60cd7 |
+The oops text, which goes to "backtrace" element, will be the decoded
|
|
|
a60cd7 |
+MCE message from kernel log buffer.
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+Non-fatal MCEs are caught as kernel oopses.
|
|
|
a60cd7 |
+If "Machine check events logged" message is seen in "dmesg" element,
|
|
|
a60cd7 |
+we assume it's a MCE, and create "not-reportable" element with suitable
|
|
|
a60cd7 |
+explanation.
|
|
|
a60cd7 |
+Then we check whether /var/log/mcelog exists,
|
|
|
a60cd7 |
+or whether system log contains "mcelog: Hardware event",
|
|
|
a60cd7 |
+and create a "comment" element with explanatory text, followed by
|
|
|
a60cd7 |
+last 20 lines from either of those files.
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+ How to test MCEs
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+There is an MCE injection tool and a kernel module, both named mce-inject.
|
|
|
a60cd7 |
+(The tool comes from mce-test project, may be found in ras-utils RHEL7 package).
|
|
|
a60cd7 |
+The script I used is:
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+modprobe mce-inject
|
|
|
a60cd7 |
+sync &
|
|
|
a60cd7 |
+sleep 1
|
|
|
a60cd7 |
+sync
|
|
|
a60cd7 |
+# This can crash the machine:
|
|
|
a60cd7 |
+echo "Injecting MCE from file $1"
|
|
|
a60cd7 |
+mce-inject "$1"
|
|
|
a60cd7 |
+echo "Exitcode:$?"
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+It requires files which describe MCE to simulate. I grabbed a few examples
|
|
|
a60cd7 |
+from mce-test.tar.gz (source tarball of mce-test project).
|
|
|
a60cd7 |
+I used this this file to cause a non-fatal MCE:
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+CPU 0 BANK 2
|
|
|
a60cd7 |
+STATUS VAL OVER EN
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+And this one to cause a fatal one:
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+CPU 0 BANK 4
|
|
|
a60cd7 |
+MCGSTATUS MCIP
|
|
|
a60cd7 |
+STATUS FATAL S
|
|
|
a60cd7 |
+RIP 12343434
|
|
|
a60cd7 |
+MISC 11
|
|
|
a60cd7 |
+
|
|
|
a60cd7 |
+(Not sure what failures exactly they imitate, maybe there are better examples).
|
|
|
a60cd7 |
--
|
|
|
a60cd7 |
1.8.3.1
|
|
|
a60cd7 |
|