# pmfault

## NAME



           **__pmFaultInject** , **__pmFaultSummary** , **PM_FAULT_POINT** ,
           **PM_FAULT_RETURN** , **PM_FAULT_CHECK** , **PM_FAULT_CLEAR** - Fault Injection
           Infrastructure for QA
    


## C SYNOPSIS



           **#include <pcp/pmapi.h>**
           **#include <pcp/fault.h>**
    
           **void __pmFaultInject(const char ***_ident_**, int** _class_**);**
           **void __pmFaultSummary(FILE ***_f_**);**
    
           **PM_FAULT_POINT(**_ident_**,**_class_**);**
           **PM_FAULT_RETURN(**_retvalue_**);**
           **PM_FAULT_CHECK;**
           **PM_FAULT_CLEAR;**
    
           **cc -DPM_FAULT_INJECTION=1 ... -lpcp_fault**
    


## DESCRIPTION



           As part of the coverage-driven changes to QA in PCP 3.6, it became
           apparent  that we needed someway to exercise the ``uncommon'' code
           paths associated with error detection and recovery.
    
           The facilities described below provide a basic fault injection in‐
           frastructure (for _libpcp_ only at this stage, although  the  mecha‐
           nism is far more general and could easily be extended).
    
           A special build is required to create _libpcp_fault_ and the associ‐
           ated  _< pcp/fault.h>_  header file.  Once this has been done, new QA
           applications may be built with **-DPM_FAULT_INJECTION=1** and/or  ex‐
           isting  applications  can be exercised in presence of fault injec‐
           tion by forcing _libpcp_fault_ to be used in preference to _libpcp_ as
           described below.
    
           In the code to be tested, **__pmFaultInject** defines a fault point at
           which a fault of type _class_ may be injected.  _ident_ is a string to
           uniquely identify the fault point across all  of  the  PCP  source
           code,  so something like "libpcp/" __FILE__ ":<number>" works just
           fine.  The _ident_ string also determines if a fault will be inject‐
           ed at run-time or not - refer to the **RUN-TIME CONTROL** section  be‐
           low.  _class_ selects a failure type, using one of the following de‐
           fined values (this list may well grow over time):
    
           **PM_FAULT_ALLOC**
                  Will cause the **next** call to malloc(3), realloc(3) or
                  strdup(3) to fail, returning NULL and setting _errno_ to
                  **ENOMEM**.  We could extend the coverage to all of the malloc-
                  related routines, but these three are sufficient to cover
                  the vast majority of the uses within _libpcp_.
    
           **PM_FAULT_CALL**
                  Will cause the **next** call to an instrumented routine to fail
                  by returning an error code (possibly the new **PM_ERR_FAULT**
                  code).  The actual error code is defined in the
                  **PM_FAULT_RETURN** macro at the head of an instrumented rou‐
                  tine.  Initially, only **__pmRegisterAnon**(3) (returns
                  **PM_ERR_FAULT**), **__pmGetPDU**(3) (returns **PM_ERR_TIMEOUT**) and
                  **__pmAllocResult**(3) (returns **NULL**) were instrumented as a
                  proof of concept for this part of the facility, however
                  other routines may have this fault injection capability
                  added over time.
    
           **PM_FAULT_MISC**
                  The ``other'' class, currently used with **PM_FAULT_CHECK** as
                  described below.
    
           To allow fault injection to co-exist within the production source
           code, **PM_FAULT_POINT** is a macro that emits no code by default, but
           when **PM_FAULT_INJECTION** is defined this becomes a call to **__pm‐**
           **FaultInject**.  Throughout _libpcp_ we use **PM_FAULT_POINT** and **not**
           **__pmFaultInject** so that both _libpcp_ and _libpcp_fault_ can be built
           from the same source code.
    
           Similarly, the macro **PM_FAULT_RETURN** emits no code unless
           **PM_FAULT_INJECTION** is defined, in which case if a fault of type
           **PM_FAULT_CALL** has been armed with **__pmFaultInject** then, the en‐
           closing routine return with the function value _retvalue_.
    
           The **PM_FAULT_CHECK** macro returns a value that may be 0 or 1.  If
           **PM_FAULT_INJECTION** is defined then if a fault of type
           **PM_FAULT_MISC** has been armed with **__pmFaultInject** then the value
           is 1 else it is 0.
    
           **PM_FAULT_CHECK** is most often used in concert with the
           **PM_FAULT_POINT** macro with the **PM_FAULT_MISC** class to potentially
           arm a trigger, then test **PM_FAULT_CHECK** and if this has the value
           1, then the **PM_FAULT_CLEAR** macro is used to clear any armed
           faults, and the fault injection code is executed.
    
           This is illustrated in the example below from _src/libpcp/src/ex‐_
           _ec.c_ :
    
               pid = fork();
    
               /* begin fault-injection block */
               PM_FAULT_POINT("libpcp/" __FILE__ ":4", PM_FAULT_MISC);
               if (PM_FAULT_CHECK) {
                PM_FAULT_CLEAR;
                if (pid > (pid_t)0)
                    kill(pid, SIGKILL);
                setoserror(EAGAIN);
                pid = -1;
               }
               /* end fault-injection block */
    
           A summary of fault points seen and faults injected is produced on
           stdio stream _f_ by **__pmFaultSummary**.
    
           Additional tracing (via **-Dfault** or **pmDebugOptions.fault**) and a new
           PMAPI error code (**PM_ERR_FAULT**) are also defined, although these
           will only ever be seen or used in _libpcp_fault_.  If **pmDebugOp‐**
           **tions.fault** is set the first time **__pmFaultInject** is called, then
           **__pmFaultSummary** will be called automatically to report on _stderr_
           when the application exits (via atexit(3)).
    
           Fault injection cannot be nested.  Each call to **__pmFaultInject**
           clears any previous fault injection that has been armed, but not
           yet executed.
    
           The fault injection infrastructure is **not** thread-safe and should
           only be used with applications that are known to be single-thread‐
           ed.
    


## RUN-TIME CONTROL



           By default, no fault injection is enabled at run-time, even when
           **__pmFaultInject** is called.
    
           Faults are selectively enabled using a control file, identified by
           the environment variable **$PM_FAULT_CONTROL** ; if this is not set, no
           faults are enabled.
    
           The control file (if it exists) is read the first time **__pmFault‐**
           **Inject** is called, and contains lines of the form:
                   _ident op number_
           that define fault injection guards.
    
           _ident_ is a fault point string (as defined by a call to **__pmFault‐**
           **Inject** , or more usually the **PM_FAULT_POINT** macro).  So one needs
           access to the _libpcp_ source code to determine the available _ident_
           strings and their semantics.
    
           _op_ is one of the C-style operators **> =**, **>** , **==** , **<** , **< =**, **!=** or **%** and
           _number_ is an unsigned integer.  _op number_ is optional and the de‐
           fault is **> 0**
    
           The semantics of the fault injection guards are that each time
           **__pmFaultInject** is called for a particular _ident_ , a trip count is
           incremented (the first trip is 1); if the C-style expression _trip‐_
           _count op number_ has the value 1 (so **true** for most _op_ s, or the re‐
           mainder equals 1 for the **%**_op_), then a fault of the _class_ defined
           for the fault point associated with _ident_ will be armed, and exe‐
           cuted as soon as possible.
    
           Within the control file, blank lines are ignored and lines begin‐
           ning with # are treated as comments.
    
           For an existing application linked with _libpcp_ fault injection may
           still be used by forcing _libpcp_fault_ to be used in the place of
           _libpcp_.  The following example shows how this might be done.
    
           $ export PM_FAULT_CONTROL=/tmp/control
           $ cat $PM_FAULT_CONTROL
           # ok for 2 trips, then inject errors
           libpcp/events.c:1  >2
    
           $ export LD_PRELOAD=/usr/lib/libpcp_fault.so
           $ pmevent -Dfault -s 3 sample.event.records
           host:      localhost
           samples:   3
           interval:  1.00 sec
           sample.event.records[fungus]: 0 event records
           __pmFaultInject(libpcp/events.c:1) ntrip=1 SKIP
           sample.event.records[bogus]: 2 event records
             10:46:12.413 --- event record [0] flags 0x1 (point) ---
               sample.event.param_string "fetch #0"
             10:46:12.413 --- event record [1] flags 0x1 (point) ---
               sample.event.param_string "bingo!"
           __pmFaultInject(libpcp/events.c:1) ntrip=2 SKIP
           sample.event.records[fungus]: 1 event records
             10:46:03.416 --- event record [0] flags 0x1 (point) ---
           __pmFaultInject(libpcp/events.c:1) ntrip=3 INJECT
           sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory
           __pmFaultInject(libpcp/events.c:1) ntrip=4 INJECT
           sample.event.records[fungus]: pmUnpackEventRecords: Cannot allocate memory
           __pmFaultInject(libpcp/events.c:1) ntrip=5 INJECT
           sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory
           === Fault Injection Summary Report ===
           libpcp/events.c:1: guard trip>2, 5 trips, 3 faults
    


## EXAMPLES



           Refer to the PCP and PCP QA source code.
    
           The macro definitions are in _src/include/pcp/fault.h_.
    
           _src/libpcp/src/fault.c_ contains all of the the underlying imple‐
           mentation.
    
           _src/libpcp_fault_ and _src/libpcp_fault/src_ contains the recipe and
           Makefiles for creating and installing _libpcp_fault.so_ and
           _< pcp/fault.h>_.
    
           **PM_FAULT_RETURN** was initiallly used in the following _libpcp_ source
           files: _derive_parser.y.in_ , _pdu.c_ and _result.c_.
    
           **PM_FAULT_POINT**.  was initiallly used in the following _libpcp_
           source files: _derive_parser.y.in_ , _desc.c_ , _e_indom.c_ , _e_labels.c_ ,
           _err.c_ , _events.c_ , _exec.c_ , _fetch.c_ , _help.c_ , _instance.c_ , _interp.c_ ,
           _labels.c_ , _logmeta.c_ , _pmns.c_ , _p_profile.c_ and _store.c_.
    
           The ``fault'' group of QA tests show examples of control file use.
           To see which tests are involved
    
           $ cd qa
           $ check -n -g fault
    


## DIAGNOSTICS



           Some non-recoverable errors are reported on _stderr_.
    


## ENVIRONMENT



           **PM_FAULT_CONTROL**
                  Full path to the fault injection control file.
    
           **LD_PRELOAD**
                  Force _libpcp_fault_ to be used in preference to _libpcp_.
    


* * *