# Using SystemTap

This chapter documents how to install SystemTap in the system and explains how to use the `stap` utility to run SystemTap scripts.

## Running SystemTap Scripts

SystemTap is distributed with a number of command line tools that allow you to monitor the activities of the system. The `stap` command reads probing instructions from a SystemTap script, translates these instructions into C code, builds a kernel module, and loads it into the running Linux kernel. The `staprun` command runs SystemTap instrumentation, that is, a kernel module built from SystemTap scripts during a cross-instrumentation.

Running `stap` and `staprun` requires elevated privileges to the system. Because not all users can be granted root access just to run SystemTap, you can allow a non-privileged user to run SystemTap instrumentation on their machine by adding them to one of the following user groups:

stapdev

- Members of this group can use the `stap` command to run SystemTap scripts, or `staprun` to run SystemTap instrumentation modules.

  Running the `stap` command involves compiling SystemTap scripts into kernel modules and loading them into the kernel. This operation requires elevated privileges to the system, which are granted to `stapdev` members. Unfortunately, such privileges also grant effective root access to `stapdev` members. As a consequence, only grant `stapdev` group membership to users whom you can trust with root access.

stapusr

- Members of this group can only use the `staprun` command to run SystemTap instrumentation modules. In addition, they can only run modules from the `/lib/modules/`kernel_version`/systemtap/` directory. Note that this directory must be owned only by the root user, and must only be writable by the root user.

The `stap` command reads a SystemTap script either from a file, or from standard input. To tell `stap` to read a SystemTap script from a file, specify the file name on the command line:

    stap file_name

To instruct `stap` to read a SystemTap script from standard input, use the `-` switch instead of the file name. Note that any command-line options you wish to use must be inserted before the `-` switch. For example, to make the output of the `stap` command more verbose, type:

    echo "probe timer.s(1) {exit()}" | stap -v -

Below is a list of commonly used `stap` options:

-v

- Makes the output of the SystemTap session more verbose. You can repeat this option multiple times to provide more details on the script's execution, for example:

      stap -vvv script.stp

  This option is particularly useful if you encounter any errors in running the script. For more information about common SystemTap script errors, refer to errors.

-o `file_name`

- Sends the standard output to a file named `file_name`.

-S `size`,`count`

- Limits the maximum size of output files to `size` megabytes and the maximum number of stored files to `count`. This option implements logrotate operations for SystemTap and the resulting file names have a sequence number suffix.

-x `process_id`

- Sets the SystemTap handler function `target()` to the specified process ID. For more information about `target()`, refer to systemtapscript-functions.

-c '`command`'

- Sets the SystemTap handler function `target()` to the specified command and runs the SystemTap instrumentation for the duration of this command. For more information about `target()`, refer to systemtapscript-functions.

-e '`script`'

- Uses `script` rather than a file as input for the SystemTap translator.

-F

- Uses SystemTap's flight recorder mode and makes the script a background process. For more information about flight recorder mode, refer to flight-recorder.

For more information about the `stap` command, refer to the

stap

1 man page. For more information about the `staprun` command and cross-instrumentation, refer to cross-compiling or the

staprun

8 man page.

### SystemTap Flight Recorder Mode

SystemTap's flight recorder mode allows you to run a SystemTap script for long periods of time and just focus on recent output. The flight recorder mode limits the amount of output generated.

There are two variations of the flight recorder mode: in-memory and file mode. In both cases, the SystemTap script runs as a background process.

#### In-memory Flight Recorder

When flight recorder mode is used without a file name, SystemTap uses a buffer in kernel memory to store the output of the script. Once the SystemTap instrumentation module is loaded and the probes start running, the instrumentation detaches and is put in the background. When the interesting event occurs, you can reattach to the instrumentation to see the recent output in the memory buffer and any continuing output.

To run a SystemTap script by using the flight recorder in-memory mode, run the `stap` command with the `-F` command line option:

    stap -F iotime.stp

Once the script starts, `stap` prints a message similar to the following to provide you with the command to reconnect to the running script:

    Disconnecting from systemtap module.
    To reconnect, type "staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556"

When the interesting event occurs, run the following command to connect to the currently running script, output the recent data in the memory buffer, and get continuing output:

    staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556

By default, the kernel buffer is 1MB in size. You can increase this value by using the `-s` option with the size in megabytes (rounded up to the next power over 2) for the buffer. For example, `-s2` on the SystemTap command line would specify 2MB for the buffer.

#### File Flight Recorder

The flight recorder mode can also store data to files. You can control the number and size of the files kept by using the `-S` option followed by two numerical arguments separated by a comma: the first argument is the maximum size in megabytes for the each output file, the second argument is the number of recent files to keep. To specify the file name, use the `-o` option followed by the name. SystemTap automatically adds a number suffix to the file name to indicate the order of the files.

The following command starts SystemTap in file flight recorder mode with the output going to files named `/tmp/iotime.log.`[0-9]+``, each file 1MB or smaller, and keeping latest two files:

    stap -F -o /tmp/pfaults.log -S 1,2  pfaults.stp

The command prints the process ID to standard output. Sending a SIGTERM to the process terminates the SystemTap script and stops the data collection. For example, if the previous command listed 7590 as the process ID, the following command would stop the SystemTap script:

    kill -s SIGTERM 7590

In this example, only the most recent two files generated by the script are kept: SystemTap automatically removes older files. As a result, the `ls -sh /tmp/pfaults.log.*` command lists two files:

    1020K /tmp/pfaults.log.5    44K /tmp/pfaults.log.6

To examine the latest data, read the file with the highest number, in this case `/tmp/pfaults.log.6`.