# User-Space Probing

SystemTap initially focused on kernel-space probing. However, there are many instances where user-space probing can help diagnose a problem. SystemTap 0.6 added support to allow probing user-space processes. SystemTap includes support for probing the entry into and return from a function in user-space processes, probing predefined markers in user-space code, and monitoring user-process events.

SystemTap requires the uprobes module to perform user-space probing. If your Linux kernel is version 3.5 or higher, it already includes uprobes. The following command can verify whether the current kernel supports uprobes natively:

    grep CONFIG_UPROBES /boot/config-`uname -r`

If uprobes is integrated, the output should be:

    CONFIG_UPROBES=y

If you are running a kernel prior to version 3.5, SystemTap will automatically build the uprobes module. However, you will also need the utrace kernel extensions, which is required by the SystemTap user-space probing to track various user-space events. More details about the utrace infrastructure are available at . The following command determines whether the currently running Linux kernel provides the needed utrace support:

    grep CONFIG_UTRACE /boot/config-`uname -r`

If the Linux kernel supports user-space probing, the following output is printed:

    CONFIG_UTRACE=y

## User-Space Events

All user-space event probes begin with process. The process events can be limited to a specific running process by specifying the process ID. The process events can also be limited to monitoring a particular executable by specifying the path to executable (PATH). SystemTap makes use of the `PATH` environment variable, so both the name used on the command-line to start the executable and the absolute path to the executable can be used. Several of the user-space probe events limit their scope to a particular executable name (PATH) because SystemTap must use debug information to statically analyze where to place the probes. But for many user-space probe events, the process ID and executable name are optional. Any `process` event in the list below that include process ID or the path to the executable must include those arguments. The process ID and path to the executable are optional for the `process` events that do not list them:

process("PATH").function("function")

  The entry to the user-space function function for the executable PATH. This event is the user-space analogue of the `kernel.function("function")` event. It allows wildcards for the function function and `.return` suffix.

process("PATH").statement("statement")

  The earliest instruction in the code for statement. This is the user-space analogue of `kernel.statement("statement")`.

process("PATH").mark("marker")

  The static probe point marker defined in PATH. Wildcards can be used for marker to specify mutiple marks with a single probe. The static probe points may also have numbered arguments ($1, $2, etc.) available to the probe. A variety of user-space packages such as Java include these static probe points. Most packages that provide static probe points also provide aliases for the raw user-space mark events. Below is one such alias for the x86_64 Java hotspot JVM:

      probe hotspot.gc_begin =
        process("/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server/libjvm.so").mark("gc__begin")

process.begin

  User-space process is created. This can be limited to a particular process ID or a full path to the executable.

process.thread.begin

  User-space thread is created. This can be limited to a particular process ID or a full path to the executable.

process.end

  User-space process died. This can be limited to a particular process ID or a full path to the executable.

process.thread.end

  User-space thread is destroyed. This can be limited to a particular process ID or a full path to the executable.

process.syscall

  User-space process makes a system call. The system call number is available via $syscall context variable, and the fist six arguments are available via $arg1 through $arg6. The ".return" suffix will place the probe at the return from the system call. For the "syscall.return" the return value is available through the $return context variable. This can be limited to a particular process ID or a full path to the executable.

Events user-space

## Accessing User-Space Target Variables

User-space target variables can be accessed in the same manner as described in . However, in Linux there are separate address spaces for the user and kernel code. When using "->" operator SystemTap will access the appropriate address space. For pointers to base types such as integers and strings there are a number of functions listed below to access user-space data. The first argument for each functions is the pointer to the data item.

user_char(address)

  Obtain the character at address for the current user process.

user_short(address)

  Obtain the short at address for the current user process.

user_int(address)

  Obtain the int at address for the current user process.

user_long(address)

  Obtain the long at address for the current user process.

user_string(address)

  Obtain the string at address for the current user process.

user_string_n(address, n)

  Obtain the string at address for the current user process and limits the string to n bytes.

target variables user-space

## User-Space Stack Backtraces

The probe point (`pp`) function indicates which particular event triggered the SystemTap event handler. A probe on the entry into a function would list the function name. However, in many cases the same probe point event may be triggered by many different modules in the program; this is particularly true for functions in shared libraries. A SystemTap backtrace of the user-space stack can provide additional context on how the probe point event is triggered.

The user-space stack backtrace generation is complicated by the compiler producing code optimized to eliminate stack frame pointers. However, the compiler also includes information in the debug information section to allow debugging tools to produce stack backtraces. SystemTap user-space stack backtrace mechanism makes use of that debug information to walk the stack to generate stack traces for 32-bit and 64-bit x86 processors; other processor architectures do not yet support the use of debug information to unwind the user-space stack. You will need to use the `-d executable` for the application executable and `-ldd` for shared libraries to ensure that the needed debug information is used to produce the user-space stack backtraces.

If you want to see how the function `xmalloc` function is being called by the command `ls`, you could use the user-space backtrack functions to provide that information. With the debuginfo for the `ls` command installed the following SystemTap command will provide a backtrace each time the `xmalloc` function is called:

    stap -d /bin/ls --ldd \
    -e 'probe process("ls").function("xmalloc") {print_usyms(ubacktrace())}' \
    -c "ls /"

When the SystemTap script runs will have output similar to the following:

    bin	dev   lib	  media  net	     proc	sbin	 sys  var
    boot	etc   lib64	  misc	 op_session  profilerc	selinux  tmp
    cgroup	home  lost+found  mnt	 opt	     root	srv	 usr
     0x4116c0 : xmalloc+0x0/0x20 [/bin/ls]
     0x4116fc : xmemdup+0x1c/0x40 [/bin/ls]
     0x40e68b : clone_quoting_options+0x3b/0x50 [/bin/ls]
     0x4087e4 : main+0x3b4/0x1900 [/bin/ls]
     0x3fa441ec5d : __libc_start_main+0xfd/0x1d0 [/lib64/libc-2.12.so]
     0x402799 : _start+0x29/0x2c [/bin/ls]
     0x4116c0 : xmalloc+0x0/0x20 [/bin/ls]
     0x4116fc : xmemdup+0x1c/0x40 [/bin/ls]
     0x40e68b : clone_quoting_options+0x3b/0x50 [/bin/ls]
     0x40884a : main+0x41a/0x1900 [/bin/ls]
     0x3fa441ec5d : __libc_start_main+0xfd/0x1d0 [/lib64/libc-2.12.so]
     ...

For more details on the functions available for user-space stack backtraces look at the `ucontext-symbols.stp` and `ucontext-unwind.stp` tapsets. The descriptions of the functions in those tapsets can also be found in the SystemTap Tapset Reference Manual.

Stack backtrace user-space