Blame SOURCES/0024-doc-MCE_readme.txt-new-file-documentation-about-MCE-.patch

baab13
From 95e9590bfee2df447c8f4c0fd799e8c514beca80 Mon Sep 17 00:00:00 2001
baab13
From: Denys Vlasenko <dvlasenk@redhat.com>
baab13
Date: Tue, 10 Dec 2013 13:07:35 +0100
baab13
Subject: [ABRT PATCH 24/27] doc/MCE_readme.txt: new file - documentation about
baab13
 MCE handling
baab13
baab13
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
baab13
baab13
Related to rhbz#1032077
baab13
baab13
Signed-off-by: Jakub Filak <jfilak@redhat.com>
baab13
---
baab13
 doc/MCE_readme.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
baab13
 1 file changed, 86 insertions(+)
baab13
 create mode 100644 doc/MCE_readme.txt
baab13
baab13
diff --git a/doc/MCE_readme.txt b/doc/MCE_readme.txt
baab13
new file mode 100644
baab13
index 0000000..ed5b627
baab13
--- /dev/null
baab13
+++ b/doc/MCE_readme.txt
baab13
@@ -0,0 +1,86 @@
baab13
+	Background
baab13
+
baab13
+MCEs can be fatal (they panic kernel) or not.
baab13
+Fatal MCE are delivered as exception#18.
baab13
+Non-fatal ones sometimes are delivered as exception#18; other times
baab13
+they are silently recorded in magic MSRs, CPU is not alerted.
baab13
+Linux kernel periodically (up to 5 mins interval) reads those MSRs
baab13
+and if MCE is seen there, it is piped in binary form through
baab13
+/dev/mcelog to whoever listens on it. (Such as mcelog tool in
baab13
+--daemon mode; but cat 
baab13
+
baab13
+"Machine Check Exception:" message is printed *only* by fatal MCEs.
baab13
+It will be caught as vmcore if kdump is configured.
baab13
+
baab13
+Non-fatal MCEs have "[Hardware Error]: Machine check events logged"
baab13
+message in kernel log.
baab13
+When /dev/mcelog is read, *no additional kernel log messages appear*.
baab13
+
baab13
+> Are those magic MSR registers cleared when read via /dev/mcelog?
baab13
+
baab13
+Yes.
baab13
+
baab13
+> Without mcelog utility, we can directly read only binary form, right?
baab13
+> Not nice, but still useful, right?
baab13
+> (could be transferred to nice text form on other machine).
baab13
+
baab13
+No, raw /dev/mcelog data is not easy to interpret on other machine.
baab13
+In fact, it can't be used by mcelog tool even on the same machine.
baab13
+Technical reason is that mcelog uses an obscure ioctl on /dev/mcelog
baab13
+in order to know the size of binary blob with MCE information.
baab13
+When run on a file, ioctl fails, and mcelog bombs out.
baab13
+
baab13
+Looks like without mcelog running and processing /dev/mcelog data,
baab13
+non-fatal MCE's can't be easily decoded with currently existing tools.
baab13
+
baab13
+mcelog tool can be configured to write log to /var/log/mcelog
baab13
+(RHEL6 does that) or to syslog (RHEL7 does that).
baab13
+
baab13
+
baab13
+	How ABRT catches MCEs
baab13
+
baab13
+Fatal MCEs are caught as any fatal kernel panic is caught - as a vmcore.
baab13
+The oops text, which goes to "backtrace" element, will be the decoded
baab13
+MCE message from kernel log buffer.
baab13
+
baab13
+Non-fatal MCEs are caught as kernel oopses.
baab13
+If "Machine check events logged" message is seen in "dmesg" element,
baab13
+we assume it's a MCE, and create "not-reportable" element with suitable
baab13
+explanation.
baab13
+Then we check whether /var/log/mcelog exists,
baab13
+or whether system log contains "mcelog: Hardware event",
baab13
+and create a "comment" element with explanatory text, followed by
baab13
+last 20 lines from either of those files.
baab13
+
baab13
+
baab13
+	How to test MCEs
baab13
+
baab13
+There is an MCE injection tool and a kernel module, both named mce-inject.
baab13
+(The tool comes from mce-test project, may be found in ras-utils RHEL7 package).
baab13
+The script I used is:
baab13
+
baab13
+modprobe mce-inject
baab13
+sync &
baab13
+sleep 1
baab13
+sync
baab13
+# This can crash the machine:
baab13
+echo "Injecting MCE from file $1"
baab13
+mce-inject "$1"
baab13
+echo "Exitcode:$?"
baab13
+
baab13
+It requires files which describe MCE to simulate. I grabbed a few examples
baab13
+from mce-test.tar.gz (source tarball of mce-test project).
baab13
+I used this this file to cause a non-fatal MCE:
baab13
+
baab13
+CPU 0 BANK 2
baab13
+STATUS VAL OVER EN
baab13
+
baab13
+And this one to cause a fatal one:
baab13
+
baab13
+CPU 0 BANK 4
baab13
+MCGSTATUS MCIP
baab13
+STATUS FATAL S
baab13
+RIP 12343434
baab13
+MISC 11
baab13
+
baab13
+(Not sure what failures exactly they imitate, maybe there are better examples).
baab13
-- 
baab13
1.8.3.1
baab13