# stapprobes

## NAME



           stapprobes - systemtap probe points
    


## DESCRIPTION



           The following sections enumerate the variety of probe points
           supported by the systemtap translator, and some of the additional
           aliases defined by standard tapset scripts.  Many are individually
           documented in the _3stap_ manual section, with the _probe::_ prefix.
    


## SYNTAX



                  **probe** PROBEPOINT [**,** PROBEPOINT] **{**[STMT ...] **}**
    
           A probe declaration may list multiple comma-separated probe points
           in order to attach a handler to all of the named events.  Normal‐
           ly, the handler statements are run whenever any of events occur.
           Depending on the type of probe point, the handler statements may
           refer to context variables (denoted with a dollar-sign prefix like
           $foo) to read or write state.  This may include function parame‐
           ters for function probes, or local variables for statement probes.
    
           The syntax of a single probe point is a general dotted-symbol se‐
           quence.  This allows a breakdown of the event namespace into
           parts, somewhat like the Domain Name System does on the Internet.
           Each component identifier may be parametrized by a string or num‐
           ber literal, with a syntax like a function call.  A component may
           include a "*" character, to expand to a set of matching probe
           points.  It may also include "**" to match multiple sequential
           components at once.  Probe aliases likewise expand to other probe
           points.
    
           Probe aliases can be given on their own, or with a suffix. The
           suffix attaches to the underlying probe point that the alias is
           expanded to. For example,
    
                  syscall.read.return.maxactive(10)
    
           expands to
    
                  kernel.function("sys_read").return.maxactive(10)
    
           with the component _maxactive(10)_ being recognized as a suffix.
    
           Normally, each and every probe point resulting from wildcard- and
           alias-expansion must be resolved to some low-level system instru‐
           mentation facility (e.g., a kprobe address, marker, or a timer
           configuration), otherwise the elaboration phase will fail.
    
           However, a probe point may be followed by a "?" character, to in‐
           dicate that it is optional, and that no error should result if it
           fails to resolve.  Optionalness passes down through all levels of
           alias/wildcard expansion.  Alternately, a probe point may be fol‐
           lowed by a "!" character, to indicate that it is both optional and
           sufficient.  (Think vaguely of the Prolog cut operator.) If it
           does resolve, then no further probe points in the same comma-sepa‐
           rated list will be resolved.  Therefore, the "!"  sufficiency mark
           only makes sense in a list of probe point alternatives.
    
           Additionally, a probe point may be followed by a "if (expr)"
           statement, in order to enable/disable the probe point on-the-fly.
           With the "if" statement, if the "expr" is false when the probe
           point is hit, the whole probe body including alias's body is
           skipped. The condition is stacked up through all levels of
           alias/wildcard expansion. So the final condition becomes the logi‐
           cal-and of conditions of all expanded alias/wildcard.  The expres‐
           sions are necessarily restricted to global variables.
    
           These are all **syntactically** valid probe points.  (They are gener‐
           ally **semantically** invalid, depending on the contents of the
           tapsets, and the versions of kernel/user software installed.)
    
                  kernel.function("foo").return
                  process("/bin/vi").statement(0x2222)
                  end
                  syscall.*
                  syscall.*.return.maxactive(10)
                  syscall.{open,close}
                  sys**open
                  kernel.function("no_such_function") ?
                  module("awol").function("no_such_function") !
                  signal.*? if (switch)
                  kprobe.function("foo")
    
           Probes may be broadly classified into "synchronous" and "asynchro‐
           nous".  A "synchronous" event is deemed to occur when any proces‐
           sor executes an instruction matched by the specification.  This
           gives these probes a reference point (instruction address) from
           which more contextual data may be available.  Other families of
           probe points refer to "asynchronous" events such as timers/coun‐
           ters rolling over, where there is no fixed reference point that is
           related.  Each probe point specification may match multiple loca‐
           tions (for example, using wildcards or aliases), and all them are
           then probed.  A probe declaration may also contain several comma-
           separated specifications, all of which are probed.
    
           Brace expansion is a mechanism which allows a list of probe points
           to be generated. It is very similar to shell expansion. A compo‐
           nent may be surrounded by a pair of curly braces to indicate that
           the comma-separated sequence of one or more subcomponents will
           each constitute a new probe point. The braces may be arbitrarily
           nested. The ordering of expanded results is based on product or‐
           der.
    
           The question mark (?), exclamation mark (!) indicators and probe
           point conditions may not be placed in any expansions that are be‐
           fore the last component.
    
           The following is an example of brace expansion.
    
                  syscall.{write,read}
                  # Expands to
                  syscall.write, syscall.read
    
                  {kernel,module("nfs")}.function("nfs*")!
                  # Expands to
                  kernel.function("nfs*")!, module("nfs").function("nfs*")!
    


## DWARF DEBUGINFO



           Resolving some probe points requires DWARF debuginfo or "debug
           symbols" for the _specific program_ being instrumented.  For some
           others, DWARF is automatically synthesized on the fly from source
           code header files.  For others, it is not needed at all.  Since a
           systemtap script may use any mixture of probe points together, the
           union of their DWARF requirements has to be met on the computer
           where script compilation occurs.  (See the _--use-server_ option and
           the **stap-server(8)** man page for information about the remote com‐
           pilation facility, which allows these requirements to be met on a
           different machine.)
    
           The following point lists many of the available probe point fami‐
           lies, to classify them with respect to their need for DWARF debug‐
           info for the specific program for that probe point.
    
           **DWARF                          NON-DWARF                    SYMBOL-TABLE**
    
           kernel.function, .statement    kernel.mark                  kernel.function _*_
           module.function, .statement    process.mark, process.plt    module.function _*_
           process.function, .statement   begin, end, error, never     process.function _*_
           process.mark _*_                  timer
           .function.callee               perf
           python2, python3               procfs
           debuginfod                     kernel.statement.absolute
                                          kernel.data
           **AUTO-GENERATED-DWARF** kprobe.function
           kernel.trace                   process.statement.absolute
                                          process.begin, .end
                                          netfilter
                                          java
    
           The probe types marked with _*_ asterisks mark fallbacks, where sys‐
           temtap can sometimes infer subset or substitute information.  In
           general, the more symbolic / debugging information available, the
           higher quality probing will be available.
    


## ON-THE-FLY ARMING



           The following types of probe points may be armed/disarmed on-the-
           fly to save overheads during uninteresting times.  Arming condi‐
           tions may also be added to other types of probes, but will be
           treated as a wrapping conditional and won't benefit from overhead
           savings.
    
           **DISARMABLE                                exceptions**
           kernel.function, kernel.statement
           module.function, module.statement
           process.*.function, process.*.statement
           process.*.plt, process.*.mark
           timer.                                    timer.profile
           java
    


## PROBE POINT FAMILIES



       **BEGIN/END/ERROR**
           The probe points _begin_ and _end_ are defined by the translator to
           refer to the time of session startup and shutdown.  All "begin"
           probe handlers are run, in some sequence, during the startup of
           the session.  All global variables will have been initialized pri‐
           or to this point.  All "end" probes are run, in some sequence,
           during the _normal_ shutdown of a session, such as in the aftermath
           of an _exit ()_ function call, or an interruption from the user.  In
           the case of an error-triggered shutdown, "end" probes are not run.
           There are no target variables available in either context.
    
           If the order of execution among "begin" or "end" probes is signif‐
           icant, then an optional sequence number may be provided:
    
                  begin(N)
                  end(N)
    
           The number N may be positive or negative.  The probe handlers are
           run in increasing order, and the order between handlers with the
           same sequence number is unspecified.  When "begin" or "end" are
           given without a sequence, they are effectively sequence zero.
    
           The _error_ probe point is similar to the _end_ probe, except that
           each such probe handler run when the session ends after errors
           have occurred.  In such cases, "end" probes are skipped, but each
           "error" probe is still attempted.  This kind of probe can be used
           to clean up or emit a "final gasp".  It may also be numerically
           parametrized to set a sequence.
    
       **NEVER**
           The probe point _never_ is specially defined by the translator to
           mean "never".  Its probe handler is never run, though its state‐
           ments are analyzed for symbol / type correctness as usual.  This
           probe point may be useful in conjunction with optional probes.
    
       **SYSCALL and ND_SYSCALL**
           The _syscall.*_ and _nd_syscall.*_  aliases define several hundred
           probes, too many to detail here.  They are of the general form:
    
                  syscall.NAME
                  nd_syscall.NAME
                  syscall.NAME.return
                  nd_syscall.NAME.return
    
           Generally, a pair of probes are defined for each normal system
           call as listed in the syscalls(2) manual page, one for entry and
           one for return.  Those system calls that never return do not have
           a corresponding _.return_ probe.  The nd_* family of probes are
           about the same, except it uses **non-DWARF** based searching mecha‐
           nisms, which may result in a lower quality of symbolic context da‐
           ta (parameters), and may miss some system calls.  You may want to
           try them first, in case kernel debugging information is not imme‐
           diately available.
    
           Each probe alias provides a variety of variables. Looking at the
           tapset source code is the most reliable way.  Generally, each
           variable listed in the standard manual page is made available as a
           script-level variable, so _syscall.open_ exposes _filename_ , _flags_ ,
           and _mode_.  In addition, a standard suite of variables is available
           at most aliases:
    
           _argstr_ A pretty-printed form of the entire argument list, without
                  parentheses.
    
           _name_   The name of the system call.
    
           _retval_ For return probes, the raw numeric system-call result.
    
           _retstr_ For return probes, a pretty-printed string form of the sys‐
                  tem-call result.
    
           As usual for probe aliases, these variables are all initialized
           once from the underlying $context variables, so that later changes
           to $context variables are not automatically reflected.  Not all
           probe aliases obey all of these general guidelines.  Please report
           any bothersome ones you encounter as a bug.  Note that on some
           kernel/userspace architecture combinations (e.g., 32-bit userspace
           on 64-bit kernel), the underlying $context variables may need ex‐
           plicit sign extension / masking.  When this is an issue, consider
           using the tapset-provided variables instead of raw $context vari‐
           ables.
    
           If debuginfo availability is a problem, you may try using the non-
           DWARF syscall probe aliases instead.  Use the _nd_syscall._  prefix
           instead of _syscall._  The same context variables are available, as
           far as possible.
    
           _nd_syscall_ probes on kernels that use syscall wrappers to pass ar‐
           guments via pt_regs (currently 4.17+ on x86_64 and 4.19+ on
           aarch64) support syscall argument writing when guru mode is en‐
           abled. If a probe syscall parameter is modified in the probe body
           then immediately before the probe exits the parameter's current
           value will be written to pt_regs. This overwrites the previous
           value.  _nd_syscall_ probes also include two parameters for each of
           the syscall's string parameters.  One holds a quoted version of
           the string passed to the syscall. The other holds an unquoted ver‐
           sion of the string intended to be used when modifying the parame‐
           ter.  If the probe modifies the unquoted string variable then as
           the probe is about to exit the contents of this variable will be
           written to the user space buffer passed to the syscall. It is the
           user's responsibility to ensure that this buffer is large enough
           to hold the modified string and that it is located in a writable
           memory segment.
    
       **TIMERS**
           There are two main types of timer probes: "jiffies" timer probes
           and time interval timer probes.
    
           Intervals defined by the standard kernel "jiffies" timer may be
           used to trigger probe handlers asynchronously.  Two probe point
           variants are supported by the translator:
    
                  timer.jiffies(N)
                  timer.jiffies(N).randomize(M)
    
           The probe handler is run every N jiffies (a kernel-defined unit of
           time, typically between 1 and 60 ms).  If the "randomize" compo‐
           nent is given, a linearly distributed random value in the range
           [-M..+M] is added to N every time the handler is run.  N is re‐
           stricted to a reasonable range (1 to around a million), and M is
           restricted to be smaller than N.  There are no target variables
           provided in either context.  It is possible for such probes to be
           run concurrently on a multi-processor computer.
    
           Alternatively, intervals may be specified in units of time.  There
           are two probe point variants similar to the jiffies timer:
    
                  timer.ms(N)
                  timer.ms(N).randomize(M)
    
           Here, N and M are specified in milliseconds, but the full options
           for units are seconds (s/sec), milliseconds (ms/msec), microsec‐
           onds (us/usec), nanoseconds (ns/nsec), and hertz (hz).  Randomiza‐
           tion is not supported for hertz timers.
    
           The actual resolution of the timers depends on the target kernel.
           For kernels prior to 2.6.17, timers are limited to jiffies resolu‐
           tion, so intervals are rounded up to the nearest jiffies interval.
           After 2.6.17, the implementation uses hrtimers for tighter preci‐
           sion, though the actual resolution will be arch-dependent.  In ei‐
           ther case, if the "randomize" component is given, then the random
           value will be added to the interval before any rounding occurs.
    
           Profiling timers are also available to provide probes that execute
           on all CPUs at the rate of the system tick (CONFIG_HZ) or at a
           given frequency (hz). On some kernels, this is a one-concurrent-
           user-only or disabled facility, resulting in error -16 (EBUSY)
           during probe registration.
    
                  timer.profile.tick
                  timer.profile.freq.hz(N)
    
           Full context information of the interrupted process is available,
           making this probe suitable for a time-based sampling profiler.
    
           It is recommended to use the tapset probe _timer.profile_ rather
           than timer.profile.tick. This probe point behaves identically to
           timer.profile.tick when the underlying functionality is available,
           and falls back to using perf.sw.cpu_clock on some recent kernels
           which lack the corresponding profile timer facility.
    
           Profiling timers with specified frequencies are only accurate up
           to around 100 hz. You may need to provide a larger value to
           achieve the desired rate.
    
           Note that if a timer probe is set to fire at a very high rate and
           if the probe body is complex, succeeding timer probes can get
           skipped, since the time for them to run has already passed. Nor‐
           mally systemtap reports missed probes, but it will not report
           these skipped probes.
    
       **DWARF**
           This family of probe points uses symbolic debugging information
           for the target kernel/module/program, as may be found in un‐
           stripped executables, or the separate _debuginfo_ packages.  They
           allow placement of probes logically into the execution path of the
           target program, by specifying a set of points in the source or ob‐
           ject code.  When a matching statement executes on any processor,
           the probe handler is run in that context.
    
           Probe points in the DWARF family can be identified by the target
           kernel module (or user process), source file, line number, func‐
           tion name, or some combination of these.
    
           Here is a list of DWARF probe points currently supported:
    
                  kernel.function(PATTERN)
                  kernel.function(PATTERN).call
                  kernel.function(PATTERN).callee(PATTERN)
                  kernel.function(PATTERN).callee(PATTERN).return
                  kernel.function(PATTERN).callee(PATTERN).call
                  kernel.function(PATTERN).callees(DEPTH)
                  kernel.function(PATTERN).return
                  kernel.function(PATTERN).inline
                  kernel.function(PATTERN).label(LPATTERN)
                  module(MPATTERN).function(PATTERN)
                  module(MPATTERN).function(PATTERN).call
                  module(MPATTERN).function(PATTERN).callee(PATTERN)
                  module(MPATTERN).function(PATTERN).callee(PATTERN).return
                  module(MPATTERN).function(PATTERN).callee(PATTERN).call
                  module(MPATTERN).function(PATTERN).callees(DEPTH)
                  module(MPATTERN).function(PATTERN).return
                  module(MPATTERN).function(PATTERN).inline
                  module(MPATTERN).function(PATTERN).label(LPATTERN)
                  kernel.statement(PATTERN)
                  kernel.statement(PATTERN).nearest
                  kernel.statement(ADDRESS).absolute
                  module(MPATTERN).statement(PATTERN)
                  process("PATH").function("NAME")
                  process("PATH").statement("*@FILE.c:123")
                  process("PATH").library("PATH").function("NAME")
                  process("PATH").library("PATH").statement("*@FILE.c:123")
                  process("PATH").library("PATH").statement("*@FILE.c:123").nearest
                  process("PATH").function("*").return
                  process("PATH").function("myfun").label("foo")
                  process("PATH").function("foo").callee("bar")
                  process("PATH").function("foo").callee("bar").return
                  process("PATH").function("foo").callee("bar").call
                  process("PATH").function("foo").callees(DEPTH)
                  process(PID).function("NAME")
                  process(PID).function("myfun").label("foo")
                  process(PID).plt("NAME")
                  process(PID).plt("NAME").return
                  process(PID).statement("*@FILE.c:123")
                  process(PID).statement("*@FILE.c:123").nearest
                  process(PID).statement(ADDRESS).absolute
                  debuginfod.process("PATH").**
    
           (See the USER-SPACE section below for more information on the
           process probes.)
    
           The list above includes multiple variants and modifiers which pro‐
           vide additional functionality or filters. They are:
    
                  **.function**
                         Places a probe near the beginning of the named func‐
                         tion, so that parameters are available as context
                         variables.
    
                  **.return**
                         Places a probe at the moment **after** the return from
                         the named function, so the return value is available
                         as the "$return" context variable.
    
                  **.inline**
                         Filters the results to include only instances of in‐
                         lined functions. Note that inlined functions do not
                         have an identifiable return point, so **.return** is not
                         supported on **.inline** probes.
    
                  **.call** Filters the results to include only non-inlined
                         functions (the opposite set of **.inline**)
    
                  **.exported**
                         Filters the results to include only exported func‐
                         tions.
    
                  **.statement**
                         Places a probe at the exact spot, exposing those lo‐
                         cal variables that are visible there.
    
                  **.statement.nearest**
                         Places a probe at the nearest available line number
                         for each line number given in the statement.
    
                  **.callee**
                         Places a probe on the callee function given in the
                         **.callee** modifier, where the callee must be a func‐
                         tion called by the target function given in **.func‐**
                         **tion**. The advantage of doing this over directly
                         probing the callee function is that this probe point
                         is run only when the callee is called from the tar‐
                         get function (add the -DSTAP_CALLEE_MATCHALL direc‐
                         tive to override this when calling stap(1)).
    
                         Note that only callees that can be statically deter‐
                         mined are available.  For example, calls through
                         function pointers are not available.  Additionally,
                         calls to functions located in other objects (e.g.
                         libraries) are not available (instead use another
                         probe point). This feature will only work for code
                         compiled with GCC 4.7+.
    
                  **.callees**
                         Shortcut for **.callee("*")** , which places a probe on
                         all callees of the function.
    
                  **.callees**(DEPTH)
                         Recursively places probes on callees. For example,
                         **.callees(2)** will probe both callees of the target
                         function, as well as callees of those callees. And
                         **.callees(3)** goes one level deeper, etc...  A callee
                         probe at depth N is only triggered when the N
                         callers in the callstack match those that were stat‐
                         ically determined during analysis (this also may be
                         overridden using -DSTAP_CALLEE_MATCHALL).
    
           In the above list of probe points, MPATTERN stands for a string
           literal that aims to identify the loaded kernel module of inter‐
           est. For in-tree kernel modules, the name suffices (e.g. "btrfs").
           The name may also include the "*", "[]", and "?" wildcards to
           match multiple in-tree modules. Out-of-tree modules are also sup‐
           ported by specifying the full path to the ko file. Wildcards are
           not supported. The file must follow the convention of being named
           <module_name>.ko (characters ',' and '-' are replaced by '_').
    
           LPATTERN stands for a source program label. It may also contain
           "*", "[]", and "?" wildcards. PATTERN stands for a string literal
           that aims to identify a point in the program.  It is made up of
           three parts:
    
           •   The first part is the name of a function, as would appear in
               the _nm_ program's output.  This part may use the "*" and "?"
               wildcarding operators to match multiple names.
    
           •   The second part is optional and begins with the "@" character.
               It is followed by the path to the source file containing the
               function, which may include a wildcard pattern, such as
               mm/slab*.  If it does not match as is, an implicit "*/" is op‐
               tionally added _before_ the pattern, so that a script need only
               name the last few components of a possibly long source direc‐
               tory path.
    
           •   Finally, the third part is optional if the file name part was
               given, and identifies the line number in the source file pre‐
               ceded by a ":" or a "+".  The line number is assumed to be an
               absolute line number if preceded by a ":", or relative to the
               declaration line of the function if preceded by a "+".  All
               the lines in the function can be matched with ":*".  A range
               of lines x through y can be matched with ":x-y". Ranges and
               specific lines can be mixed using commas, e.g. ":x,y-z".
    
           As an alternative, PATTERN may be a numeric constant, indicating
           an address.  Such an address may be found from symbol tables of
           the appropriate kernel / module object file.  It is verified
           against known statement code boundaries, and will be relocated for
           use at run time.
    
           In guru mode only, absolute kernel-space addresses may be speci‐
           fied with the ".absolute" suffix.  Such an address is considered
           already relocated, as if it came from **/proc/kallsyms** , so it cannot
           be checked against statement/instruction boundaries.
    
       **CONTEXT VARIABLES**
           Many of the source-level context variables, such as function para‐
           meters, locals, globals visible in the compilation unit, may be
           visible to probe handlers.  They may refer to these variables by
           prefixing their name with "$" within the scripts.  In addition, a
           special syntax allows limited traversal of structures, pointers,
           and arrays.  More syntax allows pretty-printing of individual
           variables or their groups.  See also **@cast**.  Note that variables
           may be inaccessible due to them being paged out, or for a few oth‐
           er reasons.  See also man error::fault(7stap).
    
           Functions called from DWARF class probe points and from
           process.mark probes may also refer to context variables.
    
           $var   refers to an in-scope variable or thread local storage
                  variable "var".  If it's an integer-like type, it will be
                  cast to a 64-bit int for systemtap script use.  String-like
                  pointers (char *) may be copied to systemtap string values
                  using the _kernel_string_ or _user_string_ functions.
    
           @var("varname")
                  an alternative syntax for _$varname_
    
           @var("varname","module")
                  The global variable or global thread local storage variable
                  in scope of the given module already loaded into the cur‐
                  rent probed process.  Useful to get an exported variable in
                  a shared library loaded into the process being probed, or a
                  global variable in a process while a shared library probe
                  is being executed.  For user-space modules only.  For exam‐
                  ple: _@var("_r_debug","/lib/ld-linux.so.2")_
    
           @var("varname@src/file.c")
                  refers to the global (either file local or external) vari‐
                  able _varname_ defined when the file _src/file.c_ was compiled.
                  The CU in which the variable is resolved is the first CU in
                  the module of the probe point which matches the given file
                  name at the end and has the shortest file name path (e.g.
                  given _@var("foo@bar/baz.c")_ and CUs with file name paths
                  _src/sub/module/bar/baz.c_ and _src/bar/baz.c_ the second CU
                  will be chosen to resolve the (file) global variable _foo_
    
           @var("varname@src/file.c","module")
                  The global variable in scope of the given CU, defined in
                  the given module, even if the variable is static (so the
                  name is not unique without the CU name).
    
           $var->field traversal via a structure's or a pointer's field.
           This
                  generalized indirection operator may be repeated to follow
                  more levels.  Note that the _._  operator is not used for
                  plain structure members, only _- >_ for both purposes.  (This
                  is because "." is reserved for string concatenation.) Also
                  note that for direct dereferencing of $var pointer {ker‐
                  nel,user}_{char,int,...}($var) should be used. (Refer to
                  stapfuncs(5) for more details.)
    
           $return
                  is available in return probes only for functions that are
                  declared with a return value, which can be determined using
                  @defined($return).
    
           $var[N]
                  indexes into an array.  The index given with a literal num‐
                  ber or even an arbitrary numeric expression.
    
           A number of operators exist for such basic context variable ex‐
           pressions:
    
           $$vars expands to a character string that is equivalent to
    
                  sprintf("parm1=%x ... parmN=%x var1=%x ... varN=%x",
                          parm1, ..., parmN, var1, ..., varN)
    
                  for each variable in scope at the probe point.  Some values
                  may be printed as _=?_  if their run-time location cannot be
                  found.
    
           $$locals
                  expands to a subset of $$vars for only local variables.
    
           $$parms
                  expands to a subset of $$vars for only function parameters.
    
           $$return
                  is available in return probes only.  It expands to a string
                  that is equivalent to sprintf("return=%x", $return) if the
                  probed function has a return value, or else an empty
                  string.
    
           & $EXPR
                  expands to the address of the given context variable ex‐
                  pression, if it is addressable.
    
           @defined($EXPR)
                  expands to 1 or 0 iff the given context variable expression
                  is resolvable, for use in conditionals such as
    
                  @defined($foo->bar) ? $foo->bar : 0
    
                  Use full $context variables or @cast() expressions in @de‐
                  fine parameters for unambiguous processing.  The TYPECAST
                  script-level variable facility is incompletely supported.
                  Simplify the parameters by reducing nesting or indirection,
                  if encountering "Collapsing unresolved @define to ..."
                  warnings.
    
           @probewrite($VAR)
                  see the PROBES section of stap(1).
    
           $EXPR$ expands to a string with all of $EXPR's members, equivalent
                  to
    
                  sprintf("{.a=%i, .b=%u, .c={...}, .d=[...]}",
                           $EXPR->a, $EXPR->b)
    
           $EXPR$$
                  expands to a string with all of $var's members and submem‐
                  bers, equivalent to
    
                  sprintf("{.a=%i, .b=%u, .c={.x=%p, .y=%c}, .d=[%i, ...]}",
                          $EXPR->a, $EXPR->b, $EXPR->c->x, $EXPR->c->y, $EXPR->d[0])
    
           @errno expands to the last value the C library global variable er‐
                  rno was set to.
    
       **MORE ON RETURN PROBES**
           For the kernel ".return" probes, only a certain fixed number of
           returns may be outstanding.  The default is a relatively small
           number, on the order of a few times the number of physical CPUs.
           If many different threads concurrently call the same blocking
           function, such as futex(2) or read(2), this limit could be exceed‐
           ed, and skipped "kretprobes" would be reported by "stap -t".  To
           work around this, specify a
    
                  probe FOO.return.maxactive(NNN)
    
           suffix, with a large enough NNN to cover all expected concurrently
           blocked threads.  Alternately, use the
    
                  stap -DKRETACTIVE=NNNN
    
           stap command line macro setting to override the default for all
           ".return" probes.
    
           For ".return" probes, context variables other than the "$return"
           may be accessible, as a convenience for a script programmer wish‐
           ing to access function parameters.  These values are **snapshots**
           taken at the time of function entry.  (Local variables within the
           function are **not** generally accessible, since those variables did
           not exist in allocated/initialized form at the snapshot moment.)
           These entry-snapshot variables should be accessed via _@en‐_
           _try($var)_.
    
           In addition, arbitrary entry-time expressions can also be saved
           for ".return" probes using the _@entry(expr)_ operator.  For exam‐
           ple, one can compute the elapsed time of a function:
    
                  probe kernel.function("do_filp_open").return {
                      println( get_timeofday_us() - @entry(get_timeofday_us()) )
                  }
    
           The following table summarizes how values related to a function
           parameter context variable, a pointer named **addr** , may be accessed
           from a _.return_ probe.
           **at-entry value   past-exit value**
    
           $addr            _not available_
           $addr->x->y      @cast(@entry($addr),"struct zz")->x->y
           $addr[0]         {kernel,user}_{char,int,...}(& $addr[0])
    
       **DWARFLESS**
           In absence of debugging information, entry & exit points of kernel
           & module functions can be probed using the "kprobe" family of
           probes.  However, these do not permit looking up the arguments /
           local variables of the function.  Following constructs are sup‐
           ported :
    
                  kprobe.function(FUNCTION)
                  kprobe.function(FUNCTION).call
                  kprobe.function(FUNCTION).return
                  kprobe.module(NAME).function(FUNCTION)
                  kprobe.module(NAME).function(FUNCTION).call
                  kprobe.module(NAME).function(FUNCTION).return
                  kprobe.statement(ADDRESS).absolute
    
           Probes of type **function** are recommended for kernel functions,
           whereas probes of type **module** are recommended for probing func‐
           tions of the specified module.  In case the absolute address of a
           kernel or module function is known, **statement** probes can be uti‐
           lized.
    
           Note that _FUNCTION_ and _MODULE_ names **must not** contain wildcards, or
           the probe will not be registered.  Also, statement probes must be
           run under guru-mode only.
    
       **USER-SPACE**
           Support for user-space probing is available for kernels that are
           configured with the utrace extensions, or have the uprobes facili‐
           ty in linux 3.5.  (Various kernel build configuration options need
           to be enabled; systemtap will advise if these are missing.)
    
           There are several forms.  First, a non-symbolic probe point:
    
                  process(PID).statement(ADDRESS).absolute
    
           is analogous to kernel.statement(ADDRESS).absolute in that both
           use raw (unverified) virtual addresses and provide no $variables.
           The target PID parameter must identify a running process, and AD‐
           DRESS should identify a valid instruction address.  All threads of
           that process will be probed.
    
           Second, non-symbolic user-kernel interface events handled by
           utrace may be probed:
    
                  process(PID).begin
                  process("FULLPATH").begin
                  process.begin
                  process(PID).thread.begin
                  process("FULLPATH").thread.begin
                  process.thread.begin
                  process(PID).end
                  process("FULLPATH").end
                  process.end
                  process(PID).thread.end
                  process("FULLPATH").thread.end
                  process.thread.end
                  process(PID).syscall
                  process("FULLPATH").syscall
                  process.syscall
                  process(PID).syscall.return
                  process("FULLPATH").syscall.return
                  process.syscall.return
    
           A **process.begin** probe gets called when new process described by
           PID or FULLPATH gets created.  In addition, it is called once from
           the context of each preexisting process, at systemtap script
           startup.  This is useful to track live processes.  A
           **process.thread.begin** probe gets called when a new thread described
           by PID or FULLPATH gets created.  A **process.end** probe gets called
           when process described by PID or FULLPATH dies.  A
           **process.thread.end** probe gets called when a thread described by
           PID or FULLPATH dies.  A **process.syscall** probe gets called when a
           thread described by PID or FULLPATH makes a system call.  The sys‐
           tem call number is available in the **$syscall** context variable, and
           the first 6 arguments of the system call are available in the
           **$argN**(ex. $arg1, $arg2, ...) context variable.  A
           **process.syscall.return** probe gets called when a thread described
           by PID or FULLPATH returns from a system call.  The system call
           number is available in the **$syscall** context variable, and the re‐
           turn value of the system call is available in the **$return** context
           variable.  A
    
           If a process probe is specified without a PID or FULLPATH, all
           user threads will be probed.  However, if systemtap was invoked
           with the _-c_ or _-x_ options, then process probes are restricted to
           the process hierarchy associated with the target process.  If a
           process probe is unspecified (i.e. without a PID or FULLPATH), but
           with the _-c_ option, the PATH of the _-c_ cmd will be heuristically
           filled into the process PATH. In that case, only command parame‐
           ters are allowed in the _-c_ command (i.e. no command substitution
           allowed and no occurrences of any of these characters:
           '|&;<>(){}').
    
           Third, symbolic static instrumentation compiled into programs and
           shared libraries may be probed:
    
                  process("PATH").mark("LABEL")
                  process("PATH").provider("PROVIDER").mark("LABEL")
                  process(PID).mark("LABEL")
                  process(PID).provider("PROVIDER").mark("LABEL")
    
           A **.mark** probe gets called via a static probe which is defined in
           the application by STAP_PROBE1(PROVIDER,LABEL,arg1), which are
           macros defined in **sys/sdt.h**.  The PROVIDER is an arbitrary appli‐
           cation identifier, LABEL is the marker site identifier, and arg1
           is the integer-typed argument.  STAP_PROBE1 is used for probes
           with 1 argument, STAP_PROBE2 is used for probes with 2 arguments,
           and so on.  The arguments of the probe are available in the con‐
           text variables $arg1, $arg2, ...  An alternative to using the
           STAP_PROBE macros is to use the dtrace script to create custom
           macros.  Additionally, the variables $$name and $$provider are
           available as parts of the probe point name.  The **sys/sdt.h** macro
           names DTRACE_PROBE* are available as aliases for STAP_PROBE*.
    
           Finally, full symbolic source-level probes in user-space programs
           and shared libraries are supported.  These are exactly analogous
           to the symbolic DWARF-based kernel/module probes described above.
           They expose the same sorts of context $variables for function pa‐
           rameters, local variables, and so on.
    
                  process("PATH").function("NAME")
                  process("PATH").statement("*@FILE.c:123")
                  process("PATH").plt("NAME")
                  process("PATH").library("PATH").plt("NAME")
                  process("PATH").library("PATH").function("NAME")
                  process("PATH").library("PATH").statement("*@FILE.c:123")
                  process("PATH").function("*").return
                  process("PATH").function("myfun").label("foo")
                  process("PATH").function("foo").callee("bar")
                  process("PATH").plt("NAME").return
                  debuginfod.process("PATH").**
                  process(PID).function("NAME")
                  process(PID).statement("*@FILE.c:123")
                  process(PID).plt("NAME")
    
           Note that for all process probes, _PATH_ names refer to executables
           that are searched the same way shells do: relative to the working
           directory if they contain a "/" character, otherwise in **$PATH**.  If
           PATH names refer to scripts, the actual interpreters (specified in
           the script in the first line after the #! characters) are probed.
           In the debuginfod probe family _PATH_ names likewise refer to exe‐
           cutables, but are searched for in the currently defined **$DEBUGIN‐**
           **FOD_URLS**.
    
           Tapset process probes placed in the special directory $pre‐
           fix/share/systemtap/tapset/PATH/ with relative paths will have
           their process parameter prefixed with the location of the tapset.
           For example,
    
                  process("foo").function("NAME")
    
           expands to
    
                  process("/usr/bin/foo").function("NAME")
    
           when placed in $prefix/share/systemtap/tapset/PATH/usr/bin/
    
           If PATH is a process component parameter referring to shared li‐
           braries then all processes that map it at runtime would be select‐
           ed for probing.  If PATH is a library component parameter refer‐
           ring to shared libraries then the process specified by the process
           component would be selected.  Note that the PATH pattern in a li‐
           brary component will always apply to libraries statically deter‐
           mined to be in use by the process. However, you may also specify
           the full path to any library file even if not statically needed by
           the process.
    
           A .plt probe will probe functions in the program linkage table
           corresponding to the rest of the probe point.  .plt can be speci‐
           fied as a shorthand for .plt("*").  The symbol name is available
           as a $$name context variable; function arguments are not avail‐
           able, since PLTs are processed without debuginfo.  A .plt.return
           probe places a probe at the moment **after** the return from the named
           function.
    
           If the PATH string contains wildcards as in the MPATTERN case,
           then standard globbing is performed to find all matching paths.
           In this case, the **$PATH** environment variable is not used.
    
           If systemtap was invoked with the _-c_ or _-x_ options, then process
           probes are restricted to the process hierarchy associated with the
           target process.
    
       **DEBUGINFOD**
           These probes take the form
    
                  debuginfod.process("PATH").**
    
           They are very similar to the process("PATH").** probe family.  The
           key difference is that the process probes search for _PATH_ in the
           host filesystem, while debuginfod probes search the current feder‐
           ation of debuginfod servers, using the currently defined **$DEBUGIN‐**
           **FOD_URLS**(see **debuginfod**(8) ).
    
           In order to probe the contents of one or more elf/archive files
           and/or elf/archive containing directories, the below will create a
           debuginfod server which will scan and process the elf files within
           and prepare them for systemtap.
    
                  $ debuginfod [options] [-F -R -Z etc.] /path1 /path2
                  $ env DEBUGINFOD_URLS=http://localhost:8002/ stap ...
    
       **JAVA**
           Support for probing Java methods is available using Byteman as a
           backend. Byteman is an instrumentation tool from the JBoss project
           which systemtap can use to monitor invocations for a specific
           method or line in a Java program.
    
           Systemtap does so by generating a Byteman script listing the
           probes to instrument and then invoking the Byteman _bminstall_ util‐
           ity.
    
           This Java instrumentation support is currently a prototype feature
           with major limitations.  Moreover, Java probing currently does not
           work across users; the stap script must run (with appropriate per‐
           missions) under the same user that the Java process being probed.
           (Thus a stap script under root currently cannot probe Java methods
           in a non-root-user Java process.)
    
           The first probe type refers to Java processes by the name of the
           Java process:
    
                  java("PNAME").class("CLASSNAME").method("PATTERN")
                  java("PNAME").class("CLASSNAME").method("PATTERN").return
    
           The PNAME argument must be a pre-existing jvm pid, and be identi‐
           fiable via a jps listing.
    
           The PATTERN parameter specifies the signature of the Java method
           to probe. The signature must consist of the exact name of the
           method, followed by a bracketed list of the types of the argu‐
           ments, for instance "myMethod(int,double,Foo)". Wildcards are not
           supported.
    
           The probe can be set to trigger at a specific line within the
           method by appending a line number with colon, just as in other
           types of probes: "myMethod(int,double,Foo):245".
    
           The CLASSNAME parameter identifies the Java class the method be‐
           longs to, either with or without the package qualification. By de‐
           fault, the probe only triggers on descendants of the class that do
           not override the method definition of the original class. However,
           CLASSNAME can take an optional caret prefix, as in _^org.my.My‐_
           _Class,_ which specifies that the probe should also trigger on all
           descendants of MyClass that override the original method. For in‐
           stance, every method with signature foo(int) in program
           org.my.MyApp can be probed at once using
    
                  java("org.my.MyApp").class("^java.lang.Object").method("foo(int)")
    
           The second probe type works analogously, but refers to Java
           processes by PID:
    
                  java(PID).class("CLASSNAME").method("PATTERN")
                  java(PID).class("CLASSNAME").method("PATTERN").return
    
           (PIDs for an already running process can be obtained using the
           **jps**(1) utility.)
    
           Context variables defined within java probes include _$arg1_ through
           _$arg10_ (for up to the first 10 arguments of a method), represented
           as character-pointers for the **toString()** form of each actual argu‐
           ment.  The _arg1_ through _arg10_ script variables provide access to
           these as ordinary strings, fetched via _user_string_warn()_.
    
           Prior to systemtap version 3.1, _$arg1_ through _$arg10_ could contain
           either integers or character pointers, depending on the types of
           the objects being passed to each particular java method.  This
           previous behaviour may be invoked with the _stap --compatible=3.0_
           flag.
    
       **PROCFS**
           These probe points allow procfs "files" in /proc/systemtap/MODNAME
           to be created, read and written using a permission that may be
           modified using the proper umask value. Default permissions are
           0400 for read probes, and 0200 for write probes. If both a read
           and write probe are being used on the same file, a default permis‐
           sion of 0600 will be used.  Using procfs.umask(0040).read would
           result in a 0404 permission set for the file.  (_MODNAME_ is the
           name of the systemtap module). The _proc_ filesystem is a pseudo-
           filesystem which is used as an interface to kernel data struc‐
           tures. There are several probe point variants supported by the
           translator:
    
                  procfs("PATH").read
                  procfs("PATH").umask(UMASK).read
                  procfs("PATH").read.maxsize(MAXSIZE)
                  procfs("PATH").umask(UMASK).maxsize(MAXSIZE)
                  procfs("PATH").write
                  procfs("PATH").umask(UMASK).write
                  procfs.read
                  procfs.umask(UMASK).read
                  procfs.read.maxsize(MAXSIZE)
                  procfs.umask(UMASK).read.maxsize(MAXSIZE)
                  procfs.write
                  procfs.umask(UMASK).write
    
           Note that there are a few differences when procfs probes are used
           in the stapbpf runtime.  FIFO special files are used instead of
           proc filesystem files.  These files are created in /var/tmp/sys‐
           temtap-USER/MODNAME.  (USER is the name of the user).  Additional‐
           ly, users cannot create both read and write probes on the same
           file.
    
           _PATH_ is the file name (relative to /proc/systemtap/MODNAME or
           /var/tmp/systemtap-USER/MODNAME) to be created.  If no _PATH_ is
           specified (as in the last two variants above), _PATH_ defaults to
           "command". The file name "__stdin" is used internally by systemtap
           for input probes and should not be used as a _PATH_ for procfs
           probes; see the input probe section below.
    
           When a user reads /proc/systemtap/MODNAME/PATH (normal runtime) or
           /var/tmp/systemtap-USER/MODNAME (stapbpf runtime), the correspond‐
           ing procfs _read_ probe is triggered.  The string data to be read
           should be assigned to a variable named _$value_ , like this:
    
                  procfs("PATH").read { $value = "100\n" }
    
           When a user writes into /proc/systemtap/MODNAME/PATH (normal run‐
           time) or /var/tmp/systemtap-USER/MODNAME (stapbpf runtime), the
           corresponding procfs _write_ probe is triggered.  The data the user
           wrote is available in the string variable named _$value_ , like this:
    
                  procfs("PATH").write { printf("user wrote: %s", $value) }
    
           _MAXSIZE_ is the size of the procfs read buffer.  Specifying _MAXSIZE_
           allows larger procfs output.  If no _MAXSIZE_ is specified, the
           procfs read buffer defaults to _STP_PROCFS_BUFSIZE_ (which defaults
           to _MAXSTRINGLEN_ , the maximum length of a string).  If setting the
           procfs read buffers for more than one file is needed, it may be
           easiest to override the _STP_PROCFS_BUFSIZE_ definition.  Here's an
           example of using _MAXSIZE_ :
    
                  procfs.read.maxsize(1024) {
                      $value = "long string..."
                      $value .= "another long string..."
                      $value .= "another long string..."
                      $value .= "another long string..."
                  }
    
       **INPUT**
           These probe points make input from stdin available to the script
           during runtime.  The translator currently supports two variants of
           this family:
    
                  input.char
                  input.line
    
           **input.char** is triggered each time a character is read from stdin.
           The current character is available in the string variable named
           _char_.  There is no newline buffering; the next character is read
           from stdin as soon as it becomes available.
    
           **input.line** causes all characters read from stdin to be buffered
           until a newline is read, at which point the probe will be trig‐
           gered. The current line of characters (including the newline) is
           made available in a string variable named _line_.  Note that no more
           than MAXSTRINGLEN characters will be buffered. Any additional
           characters will not be included in _line_.
    
           Input probes are aliases for **procfs("__stdin").write**.  Systemtap
           reconfigures stdin if the presence of this procfs probe is detect‐
           ed, therefore "__stdin" should not be used as a path argument for
           procfs probes.  Additionally, input probes will not work with the
           -F and --remote options.
    
       **NETFILTER HOOKS**
           These probe points allow observation of network packets using the
           netfilter mechanism. A netfilter probe in systemtap corresponds to
           a netfilter hook function in the original netfilter probes API. It
           is probably more convenient to use tapset::netfilter(3stap), which
           wraps the primitive netfilter hooks and does the work of extract‐
           ing useful information from the context variables.
    
           There are several probe point variants supported by the transla‐
           tor:
    
                  netfilter.hook("HOOKNAME").pf("PROTOCOL_F")
                  netfilter.pf("PROTOCOL_F").hook("HOOKNAME")
                  netfilter.hook("HOOKNAME").pf("PROTOCOL_F").priority("PRIORITY")
                  netfilter.pf("PROTOCOL_F").hook("HOOKNAME").priority("PRIORITY")
    
           _PROTOCOL_F_ is the protocol family to listen for, currently one of
           _NFPROTO_IPV4, NFPROTO_IPV6, NFPROTO_ARP,_ or _NFPROTO_BRIDGE._
    
           _HOOKNAME_ is the point, or 'hook', in the protocol stack at which
           to intercept the packet. The available hook names for each proto‐
           col family are taken from the kernel header files <linux/netfil‐
           ter_ipv4.h>, <linux/netfilter_ipv6.h>, <linux/netfilter_arp.h> and
           <linux/netfilter_bridge.h>. For instance, allowable hook names for
           _NFPROTO_IPV4_ are _NF_INET_PRE_ROUTING, NF_INET_LOCAL_IN, NF_IN‐_
           _ET_FORWARD, NF_INET_LOCAL_OUT,_ and _NF_INET_POST_ROUTING._
    
           _PRIORITY_ is an integer priority giving the order in which the
           probe point should be triggered relative to any other netfilter
           hook functions which trigger on the same packet. Hook functions
           execute on each packet in order from smallest priority number to
           largest priority number. If no _PRIORITY_ is specified (as in the
           first two probe point variants above), _PRIORITY_ defaults to "0".
    
           There are a number of predefined priority names of the form
           _NF_IP_PRI_*_ and _NF_IP6_PRI_*_ which are defined in the kernel head‐
           er files <linux/netfilter_ipv4.h> and <linux/netfilter_ipv6.h> re‐
           spectively. The script is permitted to use these instead of speci‐
           fying an integer priority. (The probe points for _NFPROTO_ARP_ and
           _NFPROTO_BRIDGE_ currently do not expose any named hook priorities
           to the script writer.)  Thus, allowable ways to specify the prior‐
           ity include:
    
                  priority("255")
                  priority("NF_IP_PRI_SELINUX_LAST")
    
           A script using guru mode is permitted to specify any identifier or
           number as the parameter for hook, pf, and priority. This feature
           should be used with caution, as the parameter is inserted verbatim
           into the C code generated by systemtap.
    
           The netfilter probe points define the following context variables:
    
           _$hooknum_
                  The hook number.
    
           _$skb_   The address of the sk_buff struct representing the packet.
                  See <linux/skbuff.h> for details on how to use this struct,
                  or alternatively use the tapset tapset::netfilter(3stap)
                  for easy access to key information.
    
           _$in_    The address of the net_device struct representing the net‐
                  work device on which the packet was received (if any). May
                  be 0 if the device is unknown or undefined at that stage in
                  the protocol stack.
    
           _$out_   The address of the net_device struct representing the net‐
                  work device on which the packet will be sent (if any). May
                  be 0 if the device is unknown or undefined at that stage in
                  the protocol stack.
    
           _$verdict_
                  (Guru mode only.) Assigning one of the verdict values de‐
                  fined in <linux/netfilter.h> to this variable alters the
                  further progress of the packet through the protocol stack.
                  For instance, the following guru mode script forces all
                  ipv6 network packets to be dropped:
    
                  probe netfilter.pf("NFPROTO_IPV6").hook("NF_IP6_PRE_ROUTING") {
                    $verdict = 0 /* nf_drop */
                  }
    
                  For convenience, unlike the primitive probe points dis‐
                  cussed here, the probes defined in tapset::netfilter(3stap)
                  export the lowercase names of the verdict constants (e.g.
                  NF_DROP becomes nf_drop) as local variables.
    
       **KERNEL TRACEPOINTS**
           This family of probe points hooks up to static probing tracepoints
           inserted into the kernel or modules.  As with markers, these tra‐
           cepoints are special macro calls inserted by kernel developers to
           make probing faster and more reliable than with DWARF-based
           probes, and DWARF debugging information is not required to probe
           tracepoints.  Tracepoints have an extra advantage of more strong‐
           ly-typed parameters than markers.
    
           Tracepoint probes look like: **kernel.trace("name")**.  The tracepoint
           name string, which may contain the usual wildcard characters, is
           matched against the names defined by the kernel developers in the
           tracepoint header files. To restrict the search to specific sub‐
           systems (e.g. sched, ext3, etc...), the following syntax can be
           used: **kernel.trace("system:name")**.  The tracepoint system string
           may also contain the usual wildcard characters.
    
           The handler associated with a tracepoint-based probe may read the
           optional parameters specified at the macro call site.  These are
           named according to the declaration by the tracepoint author.  For
           example, the tracepoint probe **kernel.trace("sched:sched_switch")**
           provides the parameters **$prev** and **$next**.  If the parameter is a
           complex type, as in a struct pointer, then a script can access
           fields with the same syntax as DWARF $target variables.  Also,
           tracepoint parameters cannot be modified, but in guru-mode a
           script may modify fields of parameters.
    
           The subsystem and name of the tracepoint are available in **$$system**
           and **$$name** and a string of name=value pairs for all parameters of
           the tracepoint is available in **$$vars** or **$$parms**.
    
       **KERNEL MARKERS (OBSOLETE)**
           This family of probe points hooks up to an older style of static
           probing markers inserted into older kernels or modules.  These
           markers are special STAP_MARK macro calls inserted by kernel de‐
           velopers to make probing faster and more reliable than with DWARF-
           based probes.  Further, DWARF debugging information is _not_ re‐
           quired to probe markers.
    
           Marker probe points begin with **kernel**.  The next part names the
           marker itself: **mark("name")**.  The marker name string, which may
           contain the usual wildcard characters, is matched against the
           names given to the marker macros when the kernel and/or module was
           compiled.    Optionally, you can specify **format("format")**.  Speci‐
           fying the marker format string allows differentiation between two
           markers with the same name but different marker format strings.
    
           The handler associated with a marker-based probe may read the op‐
           tional parameters specified at the macro call site.  These are
           named **$arg1** through **$argNN** , where NN is the number of parameters
           supplied by the macro.  Number and string parameters are passed in
           a type-safe manner.
    
           The marker format string associated with a marker is available in
           **$format**.  And also the marker name string is available in **$name**.
    
       **KERNEL HARDWARE BREAKPOINTS**
           This family of probes is used to set hardware watchpoints for a
           given
            (global) kernel symbol. The probes take three components as in‐
           puts :
    
           1. The **virtual** address / name of the kernel symbol to be traced is
           supplied as argument to this class of probes. ( Probes for only
           data segment variables are supported. Probing local variables of a
           function cannot be done.)
    
           2. Nature of access to be probed : a.  _.write_ probe gets triggered
           when a write happens at the specified address/symbol name.  b.  _rw_
           probe is triggered when either a read or write happens.
    
           3.  **.length**(optional) Users have the option of specifying the ad‐
           dress interval to be probed using "length" constructs. The user-
           specified length gets approximated to the closest possible address
           length that the architecture can support. If the specified length
           exceeds the limits imposed by architecture, an error message is
           flagged and probe registration fails.  Wherever 'length' is not
           specified, the translator requests a hardware breakpoint probe of
           length 1. It should be noted that the "length" construct is not
           valid with symbol names.
    
           Following constructs are supported :
    
                  probe kernel.data(ADDRESS).write
                  probe kernel.data(ADDRESS).rw
                  probe kernel.data(ADDRESS).length(LEN).write
                  probe kernel.data(ADDRESS).length(LEN).rw
                  probe kernel.data("SYMBOL_NAME").write
                  probe kernel.data("SYMBOL_NAME").rw
    
           This set of probes make use of the debug registers of the proces‐
           sor, which is a scarce resource. (4 on x86 , 1 on powerpc ) The
           script translation flags a warning if a user requests more hard‐
           ware breakpoint probes than the limits set by architecture. For
           example,a pass-2 warning is flashed when an input script requests
           5 hardware breakpoint probes on an x86 system while x86 architec‐
           ture supports a maximum of 4 breakpoints.  Users are cautioned to
           set probes judiciously.
    
           It is possible to specify userspace virtual memory addresses in
           this family of probes and the handlers would trigger upon the cor‐
           responding memory read/write events in those processes. But one
           cannot easily control which processes are monitored. Using `if
           (pid() == target())` is a workaround but it is inefficient. Better
           use the userland hardware breakpoint probes below instead.
    
       **USERLAND HARDWARE BREAKPOINTS**
           This family of probes is very similar to its kernel-space counter‐
           part but it targets the userland processes only.
    
           The following constructs are currently supported:
    
                  probe process.data(ADDRESS).write
                  probe process.data(ADDRESS).rw
                  probe process.data(ADDRESS).length(LEN).write
                  probe process.data(ADDRESS).length(LEN).rw
    
           Currently, only the target process specified by -x PID or -c CMD
           has the watchpoints registered. The ADDRESS must be a valid virtu‐
           al memory address in that process's address space.
    
       **PERF**
           This family of probe points interfaces to the kernel "perf event"
           infrastructure for controlling hardware performance counters.  The
           events being attached to are described by the "type", "config"
           fields of the _perf_event_attr_ structure, and are sampled at an in‐
           terval governed by the "sample_period" and "sample_freq" fields.
    
           These fields are made available to systemtap scripts using the
           following syntax:
    
                  probe perf.type(NN).config(MM).sample(XX)
                  probe perf.type(NN).config(MM).hz(XX)
                  probe perf.type(NN).config(MM)
                  probe perf.type(NN).config(MM).process("PROC")
                  probe perf.type(NN).config(MM).counter("COUNTER")
                  probe perf.type(NN).config(MM).process("PROC").counter("NAME")
    
           The systemtap probe handler is called once per XX increments of
           the underlying performance counter when using the .sample field or
           at a frequency in hertz when using the .hz field. When not speci‐
           fied, the default behavior is to sample at a count of 1000000.
           The range of valid type/config is described by the
           perf_event_open(2) system call, and/or the _linux/perf_event.h_
           file.  Invalid combinations or exhausted hardware counter re‐
           sources result in errors during systemtap script startup.  System‐
           tap does not sanity-check the values: it merely passes them
           through to the kernel for error- and safety-checking.  By default
           the perf event probe is systemwide unless .process is specified,
           which will bind the probe to a specific task.  If the name is
           omitted then it is inferred from the stap -c argument.   A perf
           event can be read on demand using .counter.  The body of the perf
           probe handler will not be invoked for a .counter probe; instead,
           the counter is read in a user space probe via:
    
              process("PROC").statement("func@file") {stat <<< @perf("NAME")}
    
       **PYTHON**
           Support for probing python 2 and python 3 function is available
           with the help of an extra python support module. Note that the de‐
           buginfo for the version of python being probed is required. To run
           a python script with the extra python support module you'd add the
           '-m HelperSDT' option to your python command, like this:
    
                  stap foo.stp -c "python -m HelperSDT foo.py"
    
           Python probes look like the following:
    
                  python2.module("MPATTERN").function("PATTERN")
                  python2.module("MPATTERN").function("PATTERN").call
                  python2.module("MPATTERN").function("PATTERN").return
                  python3.module("MPATTERN").function("PATTERN")
                  python3.module("MPATTERN").function("PATTERN").call
                  python3.module("MPATTERN").function("PATTERN").return
    
           The list above includes multiple variants and modifiers which pro‐
           vide additional functionality or filters. They are:
    
                  **.function**
                         Places a probe at the beginning of the named func‐
                         tion by default, unless modified by PATTERN. Parame‐
                         ters are available as context variables.
    
                  **.call** Places a probe at the beginning of the named func‐
                         tion. Parameters are available as context variables.
    
                  **.return**
                         Places a probe at the moment **before** the return from
                         the named function. Parameters and local/global
                         python variables are available as context variables.
    
           PATTERN stands for a string literal that aims to identify a point
           in the python program.  It is made up of three parts:
    
           •   The first part is the name of a function (e.g. "foo") or class
               method (e.g. "bar.baz"). This part may use the "*" and "?"
               wildcarding operators to match multiple names.
    
           •   The second part is optional and begins with the "@" character.
               It is followed by the path to the source file containing the
               function, which may include a wildcard pattern. The python
               path is searched for a matching filename.
    
           •   Finally, the third part is optional if the file name part was
               given, and identifies the line number in the source file pre‐
               ceded by a ":" or a "+".  The line number is assumed to be an
               absolute line number if preceded by a ":", or relative to the
               declaration line of the function if preceded by a "+".  All
               the lines in the function can be matched with ":*".  A range
               of lines x through y can be matched with ":x-y". Ranges and
               specific lines can be mixed using commas, e.g. ":x,y-z".
    
           In the above list of probe points, MPATTERN stands for a python
           module or script name that names the python module of interest.
           This part may use the "*" and "?" wildcarding operators to match
           multiple names. The python path is searched for a matching file‐
           name.
    


## EXAMPLES



           Here are some example probe points, defining the associated
           events.
    
           begin, end, end
                  refers to the startup and normal shutdown of the session.
                  In this case, the handler would run once during startup and
                  twice during shutdown.
    
           timer.jiffies(1000).randomize(200)
                  refers to a periodic interrupt, every 1000 +/- 200 jiffies.
    
           kernel.function("*init*"), kernel.function("*exit*")
                  refers to all kernel functions with "init" or "exit" in the
                  name.
    
           kernel.function("*@kernel/time.c:240")
                  refers to any functions within the "kernel/time.c" file
                  that span line 240.  Note that this is **not** a probe at the
                  statement at that line number.  Use the kernel.statement
                  probe instead.
    
           kernel.trace("sched_*")
                  refers to all scheduler-related (really, prefixed) trace‐
                  points in the kernel.
    
           kernel.mark("getuid")
                  refers to an obsolete STAP_MARK(getuid, ...) macro call in
                  the kernel.
    
           module("usb*").function("*sync*").return
                  refers to the moment of return from all functions with
                  "sync" in the name in any of the USB drivers.
    
           kernel.statement(0xc0044852)
                  refers to the first byte of the statement whose compiled
                  instructions include the given address in the kernel.
    
           kernel.statement("*@kernel/time.c:296")
                  refers to the statement of line 296 within "kernel/time.c".
    
           kernel.statement("bio_init@fs/bio.c+3")
                  refers to the statement at line bio_init+3 within
                  "fs/bio.c".
    
           kernel.data("pid_max").write
                  refers to a hardware breakpoint of type "write" set on
                  pid_max
    
           syscall.*.return
                  refers to the group of probe aliases with any name in the
                  third position
    


* * *