# pmdaopenmetrics

## NAME



           **pmdaopenmetrics** - OpenMetrics PMDA
    


## SYNOPSIS



           **$PCP_PMDAS_DIR/openmetrics/pmdaopenmetrics**[**-D**] [**-n**] [**-c** _config_]
           [**-d** _domain_] [**-l** _logfile_] [**-r** _root_] [**-t** _timeout_] [**-u** _user_]
    


## DESCRIPTION



           **pmdaopenmetrics** is a Performance Metrics Domain Agent (PMDA) which
           dynamically creates PCP metrics from configured OpenMetrics
           endpoints, which provide HTTP based access to application metrics.
           The PMDA essentially implements a bridge between **Prometheus** and
           **PCP** , allowing PCP to easily ingest performance data from more than
           650 registered end-points and many other application specific end-
           points.
    
           The default _config_ directory is
           **$PCP_PMDAS_DIR/openmetrics/config.d/** , see ``CONFIGURATION
           SOURCES'' below.  The default URL fetch _timeout_ is **2** seconds.  The
           default _user_ , if not specified with the **-u** option, is the current
           user.  If the **-n** option is given, the list of configuration files
           will not be sorted prior to processing.  This list is sorted by
           default but that can be expensive if there are a large number of
           configuration files (URLs and/or scripts).
    
           If the **-D** option is given, additional diagnostic messages will be
           written to the PMDA log file, which is
           **$PCP_LOG_DIR/pmcd/openmetrics.log** by default (see also **-l** below).
           In addition, the metric **openmetrics.control.debug** controls the
           same debug flag and can be set with the following command:
                **pmstore openmetrics.control.debug** _value_
           where _value_ is either **1**(to enable verbose log messages) or **0**(to
           disable verbose log messages).  This is particularly useful for
           examining the http headers passed to each fetch request, filter
           settings and other processing details that are logged when the
           debugging flag is enabled.
    
           The **-d** option may be used to override the default performance
           metrics _domain_ number, which defaults to **144.** It is strongly
           recommended not to change this.  The _domain_ number should be
           different for every PMDA on the one host, and the same _domain_
           number should be used for **pmdaopenmetrics** PMDA on all hosts.  See
           also the **-r** option, which allows the root of the dynamic namespace
           to be changed from the default **openmetrics**.
    
           The **-l** option may be used to specify _logfile_ as the destination
           for PMDA messages instead of the default,
           **$PCP_LOG_DIR/pmcd/openmetrics.log**.  As a special case, _logfile_ may
           be **"-"** to send messages to the **stderr** stream instead, e.g.  **-l-**.
           This would normally be the **stderr** stream for the parent process,
           pmcd(1), which may itself have redirected **stderr**.  This
           redirection is normally most useful in a containerized
           environment, or when using dbpmda(1).
    
           The **-r** option allows the root of the dynamic namespace to be
           changed to _root_ from the default, **openmetrics**.  In conjunction
           with other command line options, this allows **pmdaopenmetrics** to be
           deployed as a different PMDA with distinct metrics namespace and
           metrics domain on the same host system.  Note that all PMDAs
           require a unique domain number so the **-d** option must also be
           specified.  Use of the **-r** option may also change the defaults for
           some other command line options, e.g. the default log file name
           and the default configuration directory.
    


## CONFIGURATION SOURCES



           As it runs, **pmdaopenmetrics** periodically recursively scans the
           **$PCP_PMDAS_DIR/openmetrics/config.d** directory (or the directory
           specified with the **-c** option), looking for source URL files
           (_*.url_) and executable scripts or binaries.  Any files that do not
           have the **.url** suffix or are not executable, are ignored - this
           allows documentation files such as "README" and non-executable
           "common" script function definitions to be present without being
           considered as config files.
    
           A remote server does not have to be up or stay running - the PMDA
           tolerates remote URLs that may come and go over time.  The PMDA
           will relay data and metadata when/if they are available, and will
           return errors when/if they are down.  PCP metric IDs, internal and
           external instance domain identifiers are persisted and will be
           restored when individual metric sources become available and/or
           when the PMDA is restarted.  In addition, the PMDA checks
           directory modification times and will rescan for new or changed
           configuration files dynamically.  It is _not_ necessary to restart
           the PMDA when adding, removing or changing configuration files.
    


## URL SOURCES



           Each file with the _.url_ suffix found in the config directory or
           sub-directory contains one complete HTTP or HTTPS URL at which
           **pmdaopenmetrics** can reach a OpenMetrics endpoint.  Local file
           access is also supported with a conventional
           _file:///somepath/somefile_ URL, in which case _somepath/somefile_
           should contain openmetrics formatted metric data.
    
           The first line of a _.url_ config file should be the URL, as
           described above.  Subsequent lines, if any, are prefixed with a
           keyword that can be used to alter the http GET request.  A keyword
           must end with **':'**(colon) and the text extends to the end of the
           line.  Comment lines that start with **#** and blank lines are
           ignored.  The only currently supported keywords are **HEADER:** and
           **FILTER:**.
    
           **HEADER:**_headername_**:**_value ... to end of line_
           Adds _headername_ and its value to the headers passed in the http
           GET request for the configured URL.  An example configuration file
           that provides 3 commonly used headers and an authentication token
           might be :
    
              http://somehost/path/endpoint.html
              # this is a comment
              HEADER: Accept: text/html
              HEADER: Keep-Alive: 300
              HEADER: Connection: keep-alive
              HEADER: Authorization: token ABCDEF1234567890
    
           As mentioned above, header values extend to the end of the line.
           They may contain any valid characters, including colons.  Multiple
           spaces will be collapsed to a single space, and leading and
           trailing spaces are trimmed.  A common use for headers is to
           configure a proxy agent and the assorted parameters it may
           require.
    


## METRIC FILTERING



           Metric filtering is a configuration file feature that allows
           ingested metrics to be included or excluded, i.e. filtered.  This
           is useful because most end-points return multiple metrics, and
           usually only some are interesting for monitoring purposes.  The
           syntax is:
           **FILTER: INCLUDE METRIC** _regex_
           or
           **FILTER: EXCLUDE METRIC** _regex_
           Dynamically created metric names that match _regex_ will be either
           included or excluded in the name space, as specified.  Note that
           only the PMNS leaf component of the metric name (as ingested from
           the URL source) is compared with the _regex_ pattern.  The simple
           rule is that the _first matching_ filter regex for a particular
           metric leaf name is the rule that prevails.  If no filter regex
           matches (or there are no filters), then the metric is included by
           default, i.e. the default filter if none are specified is **FILTER:**
           **INCLUDE METRIC .*** This is backward compatible with older versions
           of the configuration file that did not support filters.  Multiple
           **FILTER:** lines would normally be used, e.g. to include some metrics
           but exclude all others, use **FILTER: EXCLUDE METRIC .*** as the last
           of several filters that include the desired metrics.  Conversely,
           to exclude some metrics but include all others, use **FILTER:**
           **EXCLUDE METRIC** _regex_.  In this case it's not necessary (though
           doesn't hurt) to specify the final **FILTER: INCLUDE METRIC .***
           because, as stated above, any metric that does not match any
           filter regex will be included by default.
    


## LABEL FILTERING



           Label filtering uses similar **FILTER:** syntax and semantics as
           metric filtering.  **FILTER: EXCLUDE LABEL** _regex_ will delete all
           labels with label name matching _regex_ from all metrics defined by
           the configuration file.  The same rules as for metric filters
           apply for label filters too - an implicit rule: **FILTER: INCLUDE**
           **LABEL .*** applies to all labels that do not match any earlier
           label filter rule.  **FILTER: OPTIONAL LABEL** _regex_ specifies that
           matching label names are to be included in the returned metric
           labelsets (i.e. included), but are **not** to be used as part of the
           the external instance names.  All included labels that are not
           optional (i.e. the _intrinsic_ labels) will be concatenated together
           and used for external instance naming.  In addition, non-intrinsic
           labels (i.e. labels tagged as **OPTIONAL**) will have the
           **PM_LABEL_OPTIONAL** flag set in the labelsets returned by **notes**
           callbacks.  This flag affects how the labels are used in certain
           clients.  For further details, see pmLookupLabels(3) and related
           man pages for further details.  Note that external instance names
           begin with the unique numeric internal instance identifier
           followed by a space, so external instance names are always unique.
    
           Caution is needed with label filtering because by default, all
           labels are used to construct the PCP instance name.  By excluding
           some labels (or changing them to optional), the instance names
           will change.  In addition, excluding all labels for a particular
           metric changes that metric to be singular, i.e. have no instance
           domain.  By excluding some labels, different instances returned by
           the URL or scripted configuration entry for the same metric may
           become duplicates.  When such duplicates occur, the last duplicate
           instance returned by the end-point URL or script prevails over any
           earlier instances.  For these reasons, it is recommended that
           label filtering rules be configured when the configuration file is
           first defined, and not changed thereafter.  If a label filtering
           change is required, the configuration file should be renamed,
           which effectively defines a new metric (or set of peer metrics as
           returned by the URL or script), with the new (or changed) instance
           naming.
    
           Unrecognized keywords in configuration files are reported in the
           PMDA log file but otherwise ignored.
    


## SCRIPTED SOURCES



           Executable scripts present in the
           _$PCP_PMDAS_DIR/openmetrics/config.d_ directory or sub-directories
           will be executed and the **stdout** stream containing openmetrics
           formatted metric data will be parsed as though it had come from a
           URL or file.  The **stderr** stream from a script will be sent to the
           PMDA log file, which by default can be found in
           **$(PCP_LOG_DIR)/pmcd/openmetrics.log**.
    
           Note that scripted sources do not support label or metric
           filtering (as described above for URL sources) - they can simply
           do their own filtering in the script itself with sed(1), **awk**(1),
           or whatever tool is desired.
    
           A simple example of a scripted config entry follows:
    
              #! /bin/sh
              awk '{
                  print("# HELP loadavg local load average")
                  print("# TYPE loadavg gauge")
                  printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
                  printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
                  printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
              }' /proc/loadavg
    
           This script produces the following OpenMetrics-formatted metric
           data when run:
    
              # HELP loadavg local load average
              # TYPE loadavg gauge
              loadavg {interval="1-minute"} 0.12
              loadavg {interval="5-minute"} 0.27
              loadavg {interval="15-minute"} 0.54
    
           If the above script was saved and made executable in a file named
           _$PCP_PMDAS_DIR/openmetrics/config.d/local/system.sh_ then this
           would result in a new PCP metric named
           **openmetrics.local.system.loadavg** which would have three instances
           for the current load average values: **1-minute** , **5-minute** and
           **15-minute**.
    
           Scripted config entries may produce more than one PCP leaf metric
           name.  For example, the above "system.sh" script could also export
           other metrics such as CPU statistics, by reading _/proc/stat_ on the
           local system.  Such additional metrics would appear as peer
           metrics in the same PCP metric subtree.  In the case of CPU
           counters, the metric type definition should be **counter** , not **gauge**.
           For full details of the openmetrics exposition formats, see
           _https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md _.
    


## SELinux CONSIDERATIONS



           Scripted config files are executed by the **pmdaopenmetrics** PMDA
           with the same SELinux context and policy as the local pmcd(1).
           For simple scripts, such as the load average example described
           above, this is normally fine.  However AVC errors may result for
           scripts that make library or system calls that are restricted by
           the prevailing SELinux context and policies.  In these cases it is
           not feasible to unilaterally grant **pmcd** or it's PMDAs an
           unconfined execution policy.  In these site specific cases it will
           be necessary to create a local SELinux policy module.  This can be
           done by capturing the AVC record(s) from the local audit log,
           generate a local policy module using **audit2allow** , and then load
           the new module using **semodule** , e.g. as follows :
    
                  $ sudo grep '^type=AVC.*pcp' /var/log/audit/audit.log \
                  | audit2allow -M mypolicy
                  $ sudo semodule -i mypolicy.pp
    
           If these local policies need to be persistent across reboots, then
           a scriptlet similar to the above example may be added to the local
           pmcd RC file (typically **/etc/pcp/pmcd/rc.local**).  For further
           details, see audit2allow(1) and **semodule**(1).
    


## METRIC NAMING



           All metrics from a file named _JOB_.*  will be exported as PCP
           metrics with the _openmetrics.JOB_ metric name prefix.  Therefore,
           the JOB name must be a valid non-leaf name for PCP PMNS metric
           names.  If the _JOB_ name has multiple dot-separated components, the
           resulting PMNS names will include those components and care is
           needed to ensure there are no overlapping definitions, e.g.
           metrics returned by **JOB.response** may overlap or conflict with
           metrics returned by **JOB.response.time**.
    
           Config file entries (URLs or scripts) found in subdirectories of
           the config directory will also result in hierarchical metric
           names.  For example, a config file named
           **$PCP_PMDAS_DIR/openmetrics/config.d/mysource/latency/get.url** will
           result in metrics being created (by fetching that source URL)
           below **openmetrics.mysource.latency.get** in the PCP namespace.
           Scripts found in subdirectories of the config directory similarly
           result in hierarchical PCP metric names.
    


## DYNAMIC METRIC NAMES



           As described above, changes and new additions can be made to files
           in the configuration directory without having to restart the PMDA.
           These changes are detected automatically and the PCP metric names
           below **openmetrics** in the PMNS will be updated accordingly, i.e.
           new metrics will be dynamically added and/or existing metrics
           removed.  In addition, **pmdaopenmetrics** honors the
           PMCD_NAMES_CHANGE pmFetch(3) protocol that was introduced in PCP
           version 4.0.  In particular, if **openmetrics** metrics are being
           logged by a PCP version 4.0 or later pmlogger(1), new metrics that
           appear as a result of changes in the PMDA configuration directory
           will automatically start to be logged, provided the root of the
           **openmetrics** PMDA namespace is configured for logging in the
           **pmlogger** configuration file.  See pmlogger(1) for details.  An
           example of such a **pmlogger** configuration file is :
    
              log mandatory on 2 second {
                   # log all metrics below the root of the openmetrics namespace
                   openmetrics
              }
    


## METADATA



           Metric data returned by URL or scripted configuration files may
           contain metadata that can be used by the **openmetrics** PMDA to
           specify the semantics, data type, scaling and units of dynamically
           created metrics.  This metadata is prefixed with **# PCP5** or **# PCP**
           in the ingested metric data.  For additional information about PCP
           metadata, see pmLookupDesc(3) and pmParseUnitsStr(3) and examples
           in shipped configuration files.
    
           In-line "PCP5" metadata must be supplied by the metrics source
           end-point (URL or script).  An alternative is to specify this in
           the URL configuration file directly, which has the advantage of
           not having to modify the source/end-point if the metadata is
           incorrect or missing.  Metadata specified in the URL configuration
           file over-rides any in-line metadata.
    
           The configuration file syntax for specifying metadata is:
           **METADATA:**_regex type indom semantics units_ ... to EOL
           Where:
           **METADATA:** is literal
           _regex_ is an extended regular expression to match one or more
           metric names returned by the URL,
           _type_ is one of the PCP numeric types (**32** , **u32** , **64** , **u64** , **float** or
           **double**),
           _indom_ is an arbitrary instance domain name, or **PM_INDOM_NULL** ,
           _semantics_ is either **counter** , **instant** or **discrete** and
           _units_ is either **none** or a string extending to end of line that is
           parsable by pmParseUnitsStr(3), i.e. the units, dimensions and
           scaling to be used for matching metrics.
    
           An example configuration file that ingests metrics from the
           Grafana /metrics end-point on localhost, filters out all metrics
           returned by that URL _except_ for **grafana_api_response_status_total**
           and then specifies the metric _type_ is an unsigned 32 bit integer
           with a non-singular instance domain named **response** and **counter**
           semantics with _units_ of **count**.
    
           **http://localhost:3000/metrics **
           **FILTER: INCLUDE METRIC grafana_api_response_status_total**
           **FILTER: EXCLUDE METRIC .***
           **METADATA: grafana_api_response_status_total u32 response counter**
           **count**
    
           Note that the name in the _indom_ field is presently ignored unless
           it is **PM_INDOM_NULL** , in which case the metric has no instance
           domain (i.e. singular), even if it has labels which would
           otherwise be used for instance naming.
    


## CONTROL METRICS



           The PMDA maintains special control metrics, as described below.
           Apart from **openmetrics.control.debug** , each of these metrics has
           one instance for each configured metric source.  All of these
           metrics have integer values with counter semantics, except
           **openmetrics.control.status** , which has a string value.  It is
           important to note that fetching any of the **openmetrics.control**
           metrics will only update the counters and status values if the
           corresponding URL is actually fetched.  If the source URL is not
           fetched, the control metric values do not trigger a refresh and
           the control values reported represent the most recent fetch of
           each corresponding source.
    
           The instance domain for the **openmetrics.control** metrics is
           adjusted dynamically as new sources are discovered.  If there are
           no sources configured, the metric names are still defined but the
           instance domain will be empty and a fetch will return no values.
    
           **openmetrics.control.status**
                  A string representing the status of the last fetch of the
                  corresponding source.  This will generally be **success** for
                  an http response code of 200.  This metric can be used for
                  service availability monitoring - provided, as stated
                  above, the corresponding source URL is fetched too.
    
           **openmetrics.control.status_code**
                  This metric is similar to **openmetrics.control.status** except
                  that it is the integer response code of the last fetch.  A
                  value of **200** usually signifies success and any other value
                  failure.  This metric can also be used for service
                  availability monitoring, with the same caveats as
                  **openmetrics.control.status**.
    
           **openmetrics.control.calls**
                  total number of times each configured metric source has
                  been fetched (if it's a URL) or executed (if it's a
                  script), since the PMDA started.  This metric has counter
                  semantics and would normally be converted to a rate/second
                  by client tools.
    
           **openmetrics.control.fetch_time**
                  Total time in milliseconds that each configured metric
                  source has taken to return a document, excluding the time
                  to parse the document.  This metric has counter semantics
                  and would normally be rate converted by client tools but is
                  also useful in raw form as the accumulated parse time since
                  the PMDA was started.
    
           **openmetrics.control.parse_time**
                  Total time in milliseconds that each configured metric
                  source has taken to parse each document, excluding the time
                  to fetch the document.  This metric has counter semantics
                  and would normally be rate converted by client tools but is
                  also useful in raw form as the accumulated parse time since
                  the PMDA was started.
    
           When converted to a rate, the **calls** metric represents the average
           fetch rate of each source over the sampling interval (time delta
           between samples).  The **fetch_time** and **parse_time** counters, when
           converted to a rate, represent the average fetch and parsing
           latency (respectfully), during the sampling interval.
    
           The **openmetrics.control.debug** metric has a singular value,
           defaulting to **0**.  If a non-zero value is stored into this metric
           using pmstore(1), additional debug messages will be written to the
           PMDA log file.
    


## LIMITATIONS



           **pmdaopenmetrics** and **libpcp** internals impose some numerical
           constraints about the number of sources (4095), metrics (1024)
           within each source, and instances for each metric (4194304).
    


## INSTALLATION



           Install the OpenMetrics PMDA by using the Install script as root:
    
               **# cd $PCP_PMDAS_DIR/openmetrics**
               **# ./Install**
    
           To uninstall, the following must be done as root:
    
               **# cd $PCP_PMDAS_DIR/openmetrics**
               **# ./Remove**
    
           **pmdaopenmetrics** is launched by pmcd(1) and should never be
           executed directly.  The Install and Remove scripts notify **pmcd**
           when the agent is installed or removed.
    
           When scripts and _.url_ files are added, removed or changed in the
           configuration directory, it is usually not necessary to restart
           the PMDA - the changes will be detected and managed on subsequent
           requests to the PMDA.
    


## FILES



           **$PCP_PMDAS_DIR/openmetrics/Install**
               installation script for the **pmdaopenmetrics** agent
    
           **$PCP_PMDAS_DIR/openmetrics/Remove**
               undo installation script for the **pmdaopenmetrics** agent
    
           **$PCP_PMDAS_DIR/openmetrics/config.d/**
               contains URLs and scripts used by the **pmdaopenmetrics** agent as
               sources of openmetrics metric data.
    
           **$PCP_LOG_DIR/pmcd/openmetrics.log**
               default log file for error messages from **pmdaopenmetrics**
    
           **$PCP_VAR_DIR/config/144.***
               files containing internal tables for metric and instance ID
               number persistence (domain 144).
    


## PCP ENVIRONMENT



           Environment variables with the prefix **PCP_** are used to
           parameterize the file and directory names used by **PCP**.  On each
           installation, the file _/etc/pcp.conf_ contains the local values for
           these variables.  The **$PCP_CONF** variable may be used to specify an
           alternative configuration file, as described in pcp.conf(5).
    


* * *