Instructions for how to set up the watchdog daemon to work with IPMI's hardware watchdog

----------------------------------------------------------------------------------------

First, verify that the ipmitool utility is present on the system to allow

the watchdog timer to be turned off via the command line (which ipmitool).

This will allow the hardware watchdog timer to be turned off gracefully

should it ever become necessary.   If ipmitool is not present, install

it or download the latest version from http://ipmitool.sourceforge.net and

build and install it on your system.

Next, prior to starting up the watchdog daemon, the BMC BIOS should be set 

to enable the IPMI/BMC hardware watchdog timer, the OpenIPMI watchdog driver 

module should be inserted with the desired configuration/startup settings, 

and the watchdog daemon's configuration file should be modified to use /dev/watchdog:

     1. To setup the IPMI/BMC BIOS to enable the hardware watchdog

     timer, see BMC documentation. The main settings in the BMC BIOS 

     requiring modification to turn on the IPMI watchdog timer are: 

      - Set the BMC POST Watchdog to "ENABLED".

      - Set the BMC POST Watchdog Timeout to "5 Minutes".

     2. To insert the OpenIPMI watchdog driver module with the

     desired configuration settings, two steps are necessary: 

        i.) Configure the OpenIPMI watchdog driver by editing the 

            /etc/sysconfig/ipmi configuration file:

          - Set "IPMI_WATCHDOG=yes".

          - Set desired options via the IPMI_WATCHDOG_OPTIONS

            config entry.

           EXAMPLE: 'IPMI_WATCHDOG_OPTIONS="timeout=60 start_now=1 \

                   preop=preop_give_data action=power_cycle pretimeout=1" '

            Execute "modinfo ipmi_watchdog" for more detailed information

            on the available ipmi watchdog timer options.

          - Execute "service ipmi start" (the watchdog driver starts 

            automatically along with the other ipmi drivers).

          IMPORTANT: If "start_now=1" has been set as one of the 

               configuration options, be sure to start up the watchdog 

               daemon before the BMC timer expires!

       ii.) Set the OpenIPMI daemon and watchdog to start during bootup:

          - chkconfig ipmi on

          - chkconfig watchdog on

     3. Configure the watchdog daemon by editing the 

     /etc/watchdog.conf configuration file: 

      - Uncomment the "watchdog-device = /dev/watchdog" line.

      - Ensure that "realtime = yes" and "priority = 1" are set and not

        commented-out.

      - Uncomment the "interval" line, and set the interval to be less 

        than what you set the timeout option to be in the /etc/sysconfig/ipmi

        file (ex "timeout=60" so you might set interval to 50).

     So in the example described herein, the BMC BIOS setting is in 

     minutes (5), and the "interval" and ipmi_watchdog "timeout" settings 

     are both in seconds (50 and 60 respectively).  Therefore, the BMC 

     hardware watchdog timer is set to expire and trigger a system power 

     cycle unless reset by the watchdog daemon within 5 minutes, and the 

     watchdog daemon will reset the timer every 60 seconds.

     4. Start the Watchdog daemon:

      - execute "service watchdog start" 

IMPORTANT:  To gracefully stop/kill the watchdog daemon, be sure

to use "service watchdog stop" (which executes "kill -s SIGTERM <pid>")

and do *not* use "kill -9 <pid>".  Using "kill -9 <pid>" will cause the 

daemon to be shut off without stopping the BMC's watchdog timer, thus 

a system reboot will be triggered when the BMC's watchdog timer expires.

Alternately, or in case the watchdog daemon is killed "ungracefully", 

you can stop the BMC timer by executing the following ipmitool utility 

command before the watchdog timer expires:

 # ipmitool -v raw 0x06 0x24 0x04 0x01 0x00 0x10 0x00 0x0a

----------------------------------------------------------------------

To test the watchdog after system configuration and setup:

.  Use kill -9 on the watchdog daemon so it doesn't shut down the watchdog daemon 

   gracefully.  Verify that the system gets reset after the BMC timer expires.

.  Use "service watchdog stop" and verify that the watchdog daemon shuts off

   the BMC watchdog timer gracefully (the system doesn't get reset).

.  Set the timer on the watchdog daemon to be greater than the time set in

   the BMC BIOS for system reset and verify that the system is reset.

.  Set the timer on the daemon to be less than the time set in the

   BMC timer and verify that the BMC watchdog is poked regularly and the 

   system is not reset.

.  Test some of the other actions the BMC can take when the watchdog timer 

   goes off (see modinfo ipmi_watchdog for some other settings to try).