# SystemTap Scripts

For the most part, SystemTap scripts are the foundation of each SystemTap session. SystemTap scripts instruct SystemTap on what type of information to collect, and what to do once that information is collected.

As stated in , SystemTap scripts are made up of two components: events and handlers. Once a SystemTap session is underway, SystemTap monitors the operating system for the specified events and executes the handlers as they occur.

Note

An event and its corresponding handler is collectively called a probe. A SystemTap script can have multiple probes.

A probe's handler is commonly referred to as a probe body.

scripts introduction probes
SystemTap scripts introduction probes
probes SystemTap scripts introduction

In terms of application development, using events and handlers is similar to instrumenting the code by inserting diagnostic print statements in a program's sequence of commands. These diagnostic print statements allow you to view a history of commands executed once the program is run.

SystemTap scripts allow insertion of the instrumentation code without recompilation of the code and allows more flexibility with regard to handlers. Events serve as the triggers for handlers to run; handlers can be specified to record specified data and print it in a certain manner.

Format

SystemTap scripts use the file extension `.stp`, and contains probes written in the following format:

scripts introduction format and syntax
SystemTap scripts introduction format and syntax
format and syntax SystemTap scripts introduction
syntax and format SystemTap scripts introduction

    probe	event {statements}

SystemTap supports multiple events per probe; multiple events are delimited by a comma (`,`). If multiple events are specified in a single probe, SystemTap will execute the handler when any of the specified events occur.

Each probe has a corresponding statement block. This statement block is enclosed in braces (`{ }`) and contains the statements to be executed per event. SystemTap executes these statements in sequence; special separators or terminators are generally not necessary between multiple statements.

Note

Statement blocks in SystemTap scripts follow the same syntax and semantics as the C programming language. A statement block can be nested within another statement block.

Systemtap allows you to write functions to factor out code to be used by a number of probes. Thus, rather than repeatedly writing the same series of statements in multiple probes, you can just place the instructions in a function, as in:

    function function_name(arguments) {statements}
    probe event {function_name(arguments)}

The `statements` in function_name are executed when the probe for event executes. The arguments are optional values passed into the function.

Important

is designed to introduce readers to the basics of SystemTap scripts. To understand SystemTap scripts better, it is advisable that you refer to ; each section therein provides a detailed explanation of the script, its events, handlers, and expected output.

## Event

SystemTap events can be broadly classified into two types: synchronous and asynchronous.

Synchronous Events

A synchronous event occurs when any process executes an instruction at a particular location in kernel code. This gives other events a reference point from which more contextual data may be available.

Events synchronous events
synchronous events Events

Examples of synchronous events include:

syscall.system_call

  The entry to the system call system_call. If the exit from a syscall is desired, appending a `.return` to the event monitor the exit of the system call instead. For example, to specify the entry and exit of the system call `close`, use `syscall.close` and `syscall.close.return` respectively.

Events syscall.system_call
syscall.system_call Events
vfs.file_operation

  The entry to the file_operation event for Virtual File System (VFS). Similar to `syscall` event, appending a `.return` to the event monitors the exit of the file_operation operation.

Events vfs.file_operation
vfs.file_operation Events
kernel.function("function")

  The entry to the kernel function function. For example, `kernel.function("sys_open")` refers to the "event" that occurs when the kernel function `sys_open` is called by any thread in the system. To specify the return of the kernel function `sys_open`, append the `return` string to the event statement; i.e. `kernel.function("sys_open").return`.

  When defining probe events, you can use asterisk (`*`) for wildcards. You can also trace the entry or exit of a function in a kernel source file. Consider the following example:

  wildcards.stp

      probe kernel.function("*@net/socket.c") { }
      probe kernel.function("*@net/socket.c").return { }

  Wild cards also work for other types of events, e.g. syscall.*

  In the previous example, the first probe's event specifies the entry of ALL functions in the kernel source file `net/socket.c`. The second probe specifies the exit of all those functions. Note that in this example, there are no statements in the handler; as such, no information will be collected or displayed.

Events kernel.function("function")
kernel.function("function") Events
Events wildcards
wildcards in events
events wildcards
kernel.trace("tracepoint")

  The static probe for tracepoint. Recent kernels (2.6.30 and newer) include instrumentation for specific events in the kernel. These events are statically marked with tracepoints. One example of a tracepoint available in systemtap is `kernel.trace("kfree_skb")` which indicates each time a network buffer is freed in the kernel.

tracepoint
Events kernel.trace("tracepoint")
kernel.trace("tracepoint") Events
module("module").function("function")

  Allows you to probe functions within modules. For example:

  moduleprobe.stp

      probe module("ext3").function("*") { }
      probe module("ext3").function("*").return { }

  The first probe in points to the entry of all functions for the `ext3` module. The second probe points to the exits of all functions for that same module; the use of the `.return` suffix is similar to `kernel.function()`. Note that the probes in do not contain statements in the probe handlers, and as such will not print any useful data (as in ).

  A system's kernel modules are typically located in `/lib/modules/kernel_version`, where kernel_version refers to the currently loaded kernel version. Modules use the file name extension `.ko`.

Events module("module")
module("module") Events

Asynchronous Events

Asynchronous events are not tied to a particular instruction or location in code. This family of probe points consists mainly of counters, timers, and similar constructs.

Events asynchronous events
asynchronous events Events

Examples of asynchronous events include:

begin

  The startup of a SystemTap session; i.e. as soon as the SystemTap script is run.

Events begin
begin Events
end

  The end of a SystemTap session.

Events end
end Events
timer events

  An event that specifies a handler to be executed periodically. For example:

  timer-s.stp

      probe timer.s(4)
      {
        printf("hello world\n")
      }

  is an example of a probe that prints `hello world` every 4 seconds. Note that you can also use the following timer events:

  - `timer.ms(milliseconds)`

  - `timer.us(microseconds)`

  - `timer.ns(nanoseconds)`

  - `timer.hz(hertz)`

  - `timer.jiffies(jiffies)`

  When used in conjunction with other probes that collect information, timer events allows you to print out get periodic updates and see how that information changes over time.

Events timer events
timer events Events

Important

SystemTap supports the use of a large collection of probe events. For more information about supported events, refer to `man stapprobes`. The SEE ALSO section of `man stapprobes` also contains links to other `man` pages that discuss supported events for specific subsystems and components.

is reference appropriate? too advanced for readers (it seems so to me)? please advise.
Events introduction
Events examples of synchronous and asynchronous events
examples of synchronous and asynchronous events Events

## Systemtap Handler/Body

Consider the following sample script:

helloworld.stp

    probe begin
    {
      printf ("hello world\n")
      exit ()
    }

In , the event `begin` (i.e. the start of the session) triggers the handler enclosed in `{ }`, which simply prints `hello world` followed by a new-line, then exits.

Note

SystemTap scripts continue to run until the `exit()` function executes. If the users wants to stop the execution of the script, it can interrupted manually with Ctrl C.

functions (used in handlers) exit()
exit() functions

printf ( ) Statements

The `printf ()` statement is one of the simplest functions for printing data. `printf ()` can also be used to display data using a wide variety of SystemTap functions in the following format:

printf() format strings

    		printf ("format string\n", arguments)

The format string specifies how arguments should be printed. The format string of simply instructs SystemTap to print `hello world`, and contains no format specifiers.

You can use the format specifiers `%s` (for strings) and `%d` (for numbers) in format strings, depending on your list of arguments. Format strings can have multiple format specifiers, each matching a corresponding argument; multiple arguments are delimited by a comma (`,`).

Note

Semantically, the SystemTap `printf` function is very similar to its C language counterpart. The aforementioned syntax and format for SystemTap's `printf` function is identical to that of the C-style `printf`.

printf() syntax and format
syntax and format printf()
format and syntax printf()

To illustrate this, consider the following probe example:

variables-in-printf-statements.stp

    probe syscall.open
    {
      printf ("%s(%d) open\n", execname(), pid())
    }

instructs SystemTap to probe all entries to the system call `open`; for each event, it prints the current `execname()` (a string with the executable name) and `pid()` (the current process ID number), followed by the word `open`. A snippet of this probe's output would look like:

editorial review: does a clarification that "format specifier1" is to "argument1", "format specifier2" is to "argument2", or is this clear enough?

    vmware-guestd(2206) open
    hald(2360) open
    hald(2360) open
    hald(2360) open
    df(3433) open
    df(3433) open
    df(3433) open
    hald(2360) open

SystemTap Functions

SystemTap supports a wide variety of functions that can be used as `printf ()` arguments. uses the SystemTap functions `execname()` (name of the process that called a kernel function/performed a system call) and `pid()` (current process ID).

functions
SystemTap script functions
handler functions
is "handler function" an appropriate term? wcohen: use "SystemTap functions" to match up language in man pages

The following is a list of commonly-used SystemTap functions:

tid()

  The ID of the current thread.

functions tid()
functions tid()
tid() functions
uid()

  The ID of the current user.

functions uid()
uid() functions
cpu()

  The current CPU number.

functions cpu()
cpu() functions
gettimeofday_s()

  The number of seconds since UNIX epoch (January 1, 1970).

functions gettimeofday_s()
gettimeofday_s() functions
ctime()

  Convert number of seconds since UNIX epoch to date.

functions ctime()
ctime() functions
pp()

  A string describing the probe point currently being handled.

functions pp()
pp() functions
thread_indent()

  This particular function is quite useful, providing you with a way to better organize your print results. The function takes one argument, an indentation delta, which indicates how many spaces to add or remove from a thread's "indentation counter". It then returns a string with some generic trace data along with an appropriate number of indentation spaces.

  The generic data included in the returned string includes a timestamp (number of microseconds since the first call to `thread_indent()` by the thread), a process name, and the thread ID. This allows you to identify what functions were called, who called them, and the duration of each function call.

  If call entries and exits immediately precede each other, it is easy to match them. However, in most cases, after a first function call entry is made several other call entries and exits may be made before the first call exits. The indentation counter helps you match an entry with its corresponding exit by indenting the next function call if it is not the exit of the previous one.

  Consider the following example on the use of `thread_indent()`:

  thread_indent.stp

      probe kernel.function("*@net/socket.c").call
      {
        printf ("%s -> %s\n", thread_indent(1), probefunc())
      }
      probe kernel.function("*@net/socket.c").return
      {
        printf ("%s <- %s\n", thread_indent(-1), probefunc())
      }

  prints out the `thread_indent()` and probe functions at each event in the following format:

      0 ftp(7223): -> sys_socketcall
      1159 ftp(7223):  -> sys_socket
      2173 ftp(7223):   -> __sock_create
      2286 ftp(7223):    -> sock_alloc_inode
      2737 ftp(7223):    <- sock_alloc_inode
      3349 ftp(7223):    -> sock_alloc
      3389 ftp(7223):    <- sock_alloc
      3417 ftp(7223):   <- __sock_create
      4117 ftp(7223):   -> sock_create
      4160 ftp(7223):   <- sock_create
      4301 ftp(7223):   -> sock_map_fd
      4644 ftp(7223):    -> sock_map_file
      4699 ftp(7223):    <- sock_map_file
      4715 ftp(7223):   <- sock_map_fd
      4732 ftp(7223):  <- sys_socket
      4775 ftp(7223): <- sys_socketcall

  This sample output contains the following information:

  - The time (in microseconds) since the initial `thread_indent()` call for the thread (included in the string from `thread_indent()`).

  - The process name (and its corresponding ID) that made the function call (included in the string from `thread_indent()`).

  - An arrow signifying whether the call was an entry (`<-`) or an exit (`->`); the indentations help you match specific function call entries with their corresponding exits.

  - The name of the function called by the process.

  remember to add a reference later to "tapsets" from here, to clarify that thread_indent is defined in tapsets as a special function of sorts
functions thread_indent()
thread_indent() functions
name

  Identifies the name of a specific system call. This variable can only be used in probes that use the event `syscall.system_call`.

local variables name
variables (local) name
name local variables
target()

  Used in conjunction with `stap script -x process ID` or `stap script -c command`. If you want to specify a script to take an argument of a process ID or command, use `target()` as the variable in the script to refer to it. For example:

  targetexample.stp

      probe syscall.* {
        if (pid() == target())
          printf("%s\n", name)
      }

  When is run with the argument `-x process ID`, it watches all system calls (as specified by the event `syscall.*`) and prints out the name of all system calls made by the specified process.

  This has the same effect as specifying `if (pid() == process ID)` each time you wish to target a specific process. However, using `target()` makes it easier for you to re-use the script, giving you the ability to simply pass a process ID as an argument each time you wish to run the script (e.g. `stap targetexample.stp -x process ID`).

functions target()
target() functions

For more information about supported SystemTap functions, refer to `man stapfuncs`.

will need a complete listing of supported handler functions? also, SystemTap function descriptions seem ambiguous, please advise.
handlers introduction
printf() format strings
format strings printf()
printf() format specifiers
format specifiers printf()
scripts introduction
SystemTap scripts introduction
scripts introduction components
SystemTap scripts introduction components
components SystemTap scripts introduction
scripts introduction events and handlers
SystemTap scripts introduction events and handlers
handlers and events SystemTap scripts introduction
scripts introduction statement blocks
SystemTap scripts introduction statement blocks
statement blocks SystemTap scripts introduction
scripts introduction functions
SystemTap scripts introduction functions
functions SystemTap scripts introduction