Blame SOURCES/0019-Fix-handling-of-Machine-Check-Exceptions.patch

baab13
From 744e4c6a6cbbb9ba0569bf8e3ab50171e974b2e3 Mon Sep 17 00:00:00 2001
baab13
From: Denys Vlasenko <dvlasenk@redhat.com>
baab13
Date: Mon, 6 Jan 2014 17:18:31 +0100
baab13
Subject: [ABRT PATCH 19/27] Fix handling of Machine Check Exceptions.
baab13
baab13
Closes #764.
baab13
baab13
If non-fatal MCE is seen, abrt will detect it as an oops
baab13
and alert user in a usual manner. When user opens this
baab13
abrt problem for reporting, he will see that "comment"
baab13
field is pre-filled with a text.
baab13
What it says depends on whether mcelog tool is installed.
baab13
If mcelog is installed, the text will say that hardware errors
baab13
were detected, and will show the tail of either /var/log/mcelog
baab13
or syslog.
baab13
Otherwise the text will say that hardware errors
baab13
were detected, but they can't be usefully diagnosed,
baab13
and user is strongly advised to install mcelog tool.
baab13
baab13
If fatal MCE is encountered, kernel always panics,
baab13
(abrt has no chance of catching the oops),
baab13
kdump kicks in, and then after reboot abrt says that new vmcore
baab13
is found. When user generates backtrace, he will see oops text
baab13
which starts with
baab13
"Machine Check Exception: BANK nnn ..." and (hopefully)
baab13
is already explanatory enough.
baab13
baab13
(Yes, it's weird that kernel shows human-readable error messages
baab13
on fatal MCEs but doesn't do that for non-fatal ones.
baab13
This makes fetching MCE info significantly different...
baab13
I wish kernel would show human-readable MCEs in both cases,
baab13
we wouldn't need mcelog then... oh well.)
baab13
baab13
In order to generate meaningful hash for MCE's,
baab13
oops hashing was extended for oopses without backtraces.
baab13
baab13
Since MCEs, unlike regular oopses, don't contain kernel version,
baab13
additional magic is added to extract kernel version
baab13
in vmcore event handling.
baab13
baab13
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
baab13
baab13
Related to rhbz#1032077
baab13
baab13
Signed-off-by: Jakub Filak <jfilak@redhat.com>
baab13
---
baab13
 src/lib/kernel.c              | 31 +++++++++++++++++++++++++
baab13
 src/plugins/koops_event.conf  | 54 +++++++++++++++++++++++++++++++++++++++++++
baab13
 src/plugins/vmcore_event.conf | 18 ++++++++++++++-
baab13
 3 files changed, 102 insertions(+), 1 deletion(-)
baab13
baab13
diff --git a/src/lib/kernel.c b/src/lib/kernel.c
baab13
index ce8815b..340ec39 100644
baab13
--- a/src/lib/kernel.c
baab13
+++ b/src/lib/kernel.c
baab13
@@ -115,8 +115,29 @@ static const char *const s_koops_suspicious_strings[] = {
baab13
      * arch/x86/kernel/cpu/mcheck/p5.c:		"CPU#%d: Machine Check Exception:  0x%8X (type 0x%8X).\n",
baab13
      * arch/x86/kernel/cpu/mcheck/mce.c:	pr_emerg(HW_ERR "CPU %d: Machine Check Exception: %Lx Bank %d: %016Lx\n",
baab13
      * drivers/edac/sb_edac.c:			printk("CPU %d: Machine Check Exception: %Lx Bank %d: %016Lx\n",
baab13
+     *
baab13
+     * MCEs can be fatal (they panic kernel) or not.
baab13
+     * Fatal MCE are delivered as exception#18 to the CPU.
baab13
+     * Non-fatal ones sometimes are delivered as exception#18;
baab13
+     * other times they are silently recorded in magic MSRs, CPU is not alerted.
baab13
+     * Linux kernel periodically (up to 5 mins interval) reads those MSRs
baab13
+     * and if MCE is seen there, it is piped in binary form through
baab13
+     * /dev/mcelog to whoever listens on it. (Such as mcelog tool in --daemon
baab13
+     * mode; but cat 
baab13
+     *
baab13
+     * "Machine Check Exception:" message is printed *only*
baab13
+     * by fatal MCEs (so far, future kernels may be different).
baab13
+     * It will be caught as vmcore if kdump is configured.
baab13
+     *
baab13
+     * Non-fatal MCEs have "[Hardware Error]: Machine check events logged"
baab13
+     * message in kernel log.
baab13
+     * When /dev/mcelog is read, *no additional kernel log messages appear*:
baab13
+     * if we want more readable data, we must rely on other tools
baab13
+     * (such as mcelog daemon consuming binary /dev/mcelog and writing
baab13
+     * human-readable /var/log/mcelog).
baab13
      */
baab13
     "Machine Check Exception:",
baab13
+    "Machine check events logged",
baab13
 
baab13
     /* X86 TRAPs */
baab13
     "divide error:",
baab13
@@ -299,6 +320,16 @@ next_line:
baab13
             if (strcasestr(curline, "Call Trace:")) /* yes, it must be case-insensitive */
baab13
                 inbacktrace = 1;
baab13
             else
baab13
+            /* Fatal MCE's have a few lines of useful information between
baab13
+             * first "Machine check exception:" line and the final "Kernel panic"
baab13
+             * line. Such oops, of course, is only detectable in kdumps (tested)
baab13
+             * or possibly pstore-saved logs (I did not try this yet).
baab13
+             * In order to capture all these lines, we treat final line
baab13
+             * as "backtrace" (which is admittedly a hack):
baab13
+             */
baab13
+            if (strstr(curline, "Kernel panic - not syncing"))
baab13
+                inbacktrace = 1;
baab13
+            else
baab13
             if (strnlen(curline, 9) > 8
baab13
              && (  (curline[0] == '(' && curline[1] == '[' && curline[2] == '<')
baab13
                 || (curline[0] == '[' && curline[1] == '<'))
baab13
diff --git a/src/plugins/koops_event.conf b/src/plugins/koops_event.conf
baab13
index c0277c8..7dfbe36 100644
baab13
--- a/src/plugins/koops_event.conf
baab13
+++ b/src/plugins/koops_event.conf
baab13
@@ -4,6 +4,60 @@ EVENT=post-create analyzer=Kerneloops
baab13
         abrt-action-analyze-oops &&
baab13
         dmesg >>dmesg &&
baab13
         abrt-action-save-kernel-data
baab13
+        abrt-action-save-kernel-data || exit $?
baab13
+        #
baab13
+        # If it exists, we can save a copy of MCE log here:
baab13
+        #test -f /var/log/mcelog && cp /var/log/mcelog .
baab13
+        # but in current config, sosreport already does that.
baab13
+        #
baab13
+        # See if MCEs were seen but mcelog isn't installed or running
baab13
+        grep -qFi 'Machine check events logged' dmesg || exit 0
baab13
+        #
baab13
+        # There was an MCE. IOW: it's not a bug, it's a HW error.
baab13
+        # Did mcelog logged it to /var/log/mcelog
baab13
+        # (RHEL6 by default does this)?
baab13
+        test -f /var/log/mcelog &&
baab13
+        {
baab13
+                # (Ab)use user comment field to inform user about it.
baab13
+                echo "The kernel log indicates that hardware errors were detected."
baab13
+                echo "/var/log/mcelog file may have more information."
baab13
+                echo "The last 20 lines of /var/log/mcelog are:"
baab13
+                echo "========================================="
baab13
+                # Redirecting sterr in case selinux makes it unreadable
baab13
+                # (annoying anyway, but at least user knows what's going on):
baab13
+                tail -n20 /var/log/mcelog 2>&1
baab13
+                exit 0
baab13
+        } >comment
baab13
+        #
baab13
+        # On RHEL7, mcelog is run so that its output ends up in syslog.
baab13
+        # Do we see that?
baab13
+        grep -qFi 'mcelog: Hardware event' /var/log/messages &&
baab13
+        {
baab13
+                echo "The kernel log indicates that hardware errors were detected."
baab13
+                echo "System log may have more information."
baab13
+                echo "The last 20 mcelog lines of system log are:"
baab13
+                echo "========================================="
baab13
+                # Redirecting sterr in case selinux makes it unreadable
baab13
+                # (annoying anyway, but at least user knows what's going on):
baab13
+                grep -Fi 'mcelog:' /var/log/messages | tail -n20 2>&1
baab13
+                exit 0
baab13
+        } >comment
baab13
+        #
baab13
+        # Apparently, there is no running mcelog daemon!
baab13
+        # Let user know that he needs one.
baab13
+        {
baab13
+        echo "The kernel log indicates that hardware errors were detected."
baab13
+        echo "The data was saved by kernel for processing by the mcelog tool."
baab13
+        echo "However, neither /var/log/mcelog nor system log contain mcelog messages."
baab13
+        echo "Most likely reason is that mcelog is not installed or not configured"
baab13
+        echo "to be started during boot."
baab13
+        echo "Without this tool running, the binary data saved by kernel"
baab13
+        echo "is of limited usefulness."
baab13
+        echo "(You can save this data anyway by running 'cat </dev/mcelog >FILE')."
baab13
+        echo "The recommended course of action is to install mcelog."
baab13
+        echo "If another hardware error would occur, a user-readable description"
baab13
+        echo "of it will be saved in system log or /var/log/mcelog."
baab13
+        } >comment
baab13
 
baab13
 # If you want behavior similar to one provided by kerneloops daemon
baab13
 # distributed by kerneloops.org - that is, if you want
baab13
diff --git a/src/plugins/vmcore_event.conf b/src/plugins/vmcore_event.conf
baab13
index f8de3c5..655d842 100644
baab13
--- a/src/plugins/vmcore_event.conf
baab13
+++ b/src/plugins/vmcore_event.conf
baab13
@@ -1,6 +1,22 @@
baab13
 # analyze
baab13
 EVENT=analyze_VMcore analyzer=vmcore
baab13
-        abrt-action-analyze-vmcore &&
baab13
+        # If kdump machinery already extracted dmesg...
baab13
+        if test -f vmcore-dmesg.txt; then
baab13
+            # ...use that
baab13
+            abrt-dump-oops -o vmcore-dmesg.txt >backtrace || exit $?
baab13
+            #
baab13
+            # Does "kernel" element exist?
baab13
+            test -f kernel && exit 0
baab13
+            #
baab13
+            # Try creating it from vmcore-dmesg.txt:
baab13
+            # MCE oopses don't have kernel version in them,
baab13
+            # but it should be specified earlier in the log.
baab13
+            k=`sed -n '/Linux version/ s/.*Linux version \([^ ]*\) .*/\1/p' vmcore-dmesg.txt | tail -n1`
baab13
+            test "$k" != "" && printf "%s" "$k" >kernel
baab13
+        else
baab13
+            # No vmcore-dmesg.txt, do it the hard way:
baab13
+            abrt-action-analyze-vmcore
baab13
+        fi &&
baab13
         abrt-action-analyze-oops &&
baab13
         abrt-action-save-kernel-data
baab13
 
baab13
-- 
baab13
1.8.3.1
baab13