# pcp2elasticsearch

## NAME



           **pcp2elasticsearch** - pcp-to-elasticsearch metrics exporter
    


## SYNOPSIS



           **pcp2elasticsearch**[**-5CGHIjLmnrRvV?**]  [**-4** _action_] [**-8** |**-9** _limit_] [**-a**
           _archive_] [**-A** _align_] [**--archive-folio** _folio_] [**-b** |**-B** _space-scale_]
           [**-c** _config_] [**--container** _container_] [**-D** _debug_] [**--daemonize**] [**-e**
           _derived_] [**-g** _URL_] [**-h** _host_] [**-i** _instances_] [**-J** _rank_] [**-K** _spec_] [**-N**
           _predicate_] [**-O** _origin_] [**-p** _type_] [**-P** |**-0** _precision_] [**-q** |**-Q** _count-_
           _scale_] [**-s** _samples_] [**-S** _starttime_] [**-t** _interval_] [**-T** _endtime_] [**-x**
           _index_] [**-X** _hostid_] [**-y** |**-Y** _time-scale_] _metricspec_ [...]
    


## DESCRIPTION



           **pcp2elasticsearch** is a customizable performance metrics exporter
           tool from PCP to Elasticsearch.  Any available performance metric,
           live or archived, system and/or application, can be selected for
           exporting using either command line arguments or a configuration
           file.
    
           **pcp2elasticsearch** is a close relative of pmrep(1).  Refer to
           pmrep(1) for the _metricspec_ description accepted on
           **pcp2elasticsearch** command line.  See pmrep.conf(5) for description
           of the **pcp2elasticsearch.conf** configuration file syntax.  This
           page describes **pcp2elasticsearch** specific options and
           configuration file differences with pmrep.conf(5).  pmrep(1) also
           lists some usage examples of which most are applicable with
           **pcp2elasticsearch** as well.
    
           Only the command line options listed on this page are supported,
           other options available for pmrep(1) are not supported.
    
           Options via environment values (see pmGetOptions(3)) override the
           corresponding built-in default values (if any).  Configuration
           file options override the corresponding environment variables (if
           any).  Command line options override the corresponding
           configuration file options (if any).
    


## CONFIGURATION FILE



           **pcp2elasticsearch** uses a configuration file with syntax described
           in pmrep.conf(5).  The following options are common with
           **pmrep.conf** : **version** , **source** , **speclocal** , **derived** , **header** , **globals** ,
           **samples** , **interval** , **type** , **type_prefer** , **ignore_incompat** ,
           **names_change** , **instances** , **live_filter** , **rank** , **limit_filter** ,
           **limit_filter_force** , **invert_filter** , **predicate** , **omit_flat** ,
           **include_labels** , **precision** , **precision_force** , **count_scale** ,
           **count_scale_force** , **space_scale** , **space_scale_force** , **time_scale** ,
           **time_scale_force**.  The rest of the **pmrep.conf** options are
           recognized but ignored for compatibility.
    
       **pcp2elasticsearch specific options**
           es_server (string)
               Specify the destination Elasticsearch server metric URL.
               Corresponding command line option is **-g**.  Defaults to
               **http://localhost:9200/ **.
    
           es_auth (string)
               Specify the user name to use with Elasticsearch requests.
               Defaults to none.
    
           es_password (string)
               Specify the pass word to use with Elasticsearch requests.
               Defaults to none.
    
           es_index (string)
               Specify the Elasticsearch index for metric names.
               Corresponding command line option is **-x**.  Defaults to **pcp**.
    
           es_hostid (string)
               Specify the Elasticsearch host-id for measurements.
               Corresponding command line option is **-X**.  Defaults to the
               metrics source host.
    
           es_search_type (string)
               Specify the Elasticsearch search type for measurements.
               Corresponding command line option is **-p**.  Defaults to
               **pcp-metric**.
    


## OPTIONS



           The available command line options are:
    
           **-0** _precision_ , **--precision-force** =_precision_
                Like **-P** but this option _will_ override per-metric
                specifications.
    
           **-4** _action_ , **--names-change** =_action_
                Specify which _action_ to take on receiving a metric names
                change event during sampling.  These events occur when a PMDA
                discovers new metrics sometime after starting up, and informs
                running client tools like **pcp2elasticsearch**.  Valid values
                for _action_ are **update**(refresh metrics being sampled), **ignore**
                (do nothing - the default behaviour) and **abort**(exit the
                program if such an event occurs).
    
           **-5** , **--ignore-unknown**
                Silently ignore any metric name that cannot be resolved.  At
                least one metric must be found for the tool to start.
    
           **-8** _limit_ , **--limit-filter** =_limit_
                Limit results to instances with values above/below _limit_.  A
                positive integer will include instances with values at or
                above the limit in reporting.  A negative integer will
                include instances with values at or below the limit in
                reporting.  A value of zero performs no limit filtering.
                This option will _not_ override possible per-metric
                specifications.  See also **-J** and **-N**.
    
           **-9** _limit_ , **--limit-filter-force** =_limit_
                Like **-8** but this option _will_ override per-metric
                specifications.
    
           **-a** _archive_ , **--archive** =_archive_
                Performance metric values are retrieved from the set of
                Performance Co-Pilot (PCP) archive files identified by the
                _archive_ argument, which is a comma-separated list of names,
                each of which may be the base name of an archive or the name
                of a directory containing one or more archives.
    
           **-A** _align_ , **--align** =_align_
                Force the initial sample to be aligned on the boundary of a
                natural time unit _align_.  Refer to PCPIntro(1) for a complete
                description of the syntax for _align_.
    
           **--archive-folio** =_folio_
                Read metric source archives from the PCP archive _folio_
                created by tools like pmchart(1) or, less often, manually
                with mkaf(1).
    
           **-b** _scale_ , **--space-scale** =_scale_
                _Unit/scale_ for space (byte) metrics, possible values include
                **bytes** , **Kbytes** , **KB** , **Mbytes** , **MB** , and so forth.  This option
                will _not_ override possible per-metric specifications.  See
                also pmParseUnitsStr(3).
    
           **-B** _scale_ , **--space-scale-force** =_scale_
                Like **-b** but this option _will_ override per-metric
                specifications.
    
           **-c** _config_ , **--config** =_config_
                Specify the _config_ file or directory to use.  In case _config_
                is a directory all files in it ending **.conf** will be included.
                The default is the first found of: _./pcp2elasticsearch.conf_ ,
                $HOME _/.pcp2elasticsearch.conf_ ,
                $HOME _/pcp/pcp2elasticsearch.conf_ , and
                $PCP_SYSCONF_DIR _/pcp2elasticsearch.conf_.  For details, see
                the above section and pmrep.conf(5).
    
           **--container** =_container_
                Fetch performance metrics from the specified _container_ ,
                either local or remote (see **-h**).
    
           **-C** , **--check**
                Exit before reporting any values, but after parsing the
                configuration and metrics and printing possible headers.
    
           **--daemonize**
                Daemonise on startup.
    
           **-e** _derived_ , **--derived** =_derived_
                Specify _derived_ performance metrics.  If _derived_ starts with
                a slash (``/'') or with a dot (``.'') it will be interpreted
                as a PCP derived metrics configuration file, otherwise it
                will be interpreted as comma- or semicolon-separated derived
                metric expressions.  For complete description of derived
                metrics and PCP derived metrics configuration files see
                pmLoadDerivedConfig(3) and pmRegisterDerived(3).
                Alternatively, using pmrep.conf(5) configuration syntax
                allows defining derived metrics as part of metricsets.
    
                In case of issues with derived metrics, review the
                aforementioned manual pages in detail and ensure all the
                required metrics are available, especially when using
                archives.  Use **-Dderive** to see additional debug information
                about parsing derived metrics.
    
           **-g** _URL_ , **--es-host** =_URL_
                Destination Elasticsearch server metric _URL_.
    
           **-G** , **--no-globals**
                Do not include global metrics in reporting (see
                pmrep.conf(5)).
    
           **-h** _host_ , **--host** =_host_
                Fetch performance metrics from pmcd(1) on _host_ , rather than
                from the default localhost.
    
           **-H** , **--no-header**
                Do not print any headers.
    
           **-i** _instances_ , **--instances** =_instances_
                Retrieve and report only the specified metric _instances_.  By
                default all instances, present and future, are reported.
    
                Refer to pmrep(1) for complete description of this option.
    
           **-I** , **--ignore-incompat**
                Ignore incompatible metrics.  By default incompatible metrics
                (that is, their type is unsupported or they cannot be scaled
                as requested) will cause **pcp2elasticsearch** to terminate with
                an error message.  With this option all incompatible metrics
                are silently omitted from reporting.  This may be especially
                useful when requesting non-leaf nodes of the PMNS tree for
                reporting.
    
           **-j** , **--live-filter**
                Perform instance live filtering.  This allows capturing all
                named instances even if processes are restarted at some point
                (unlike without live filtering).  Performing live filtering
                over a huge number of instances will add some internal
                overhead so a bit of user caution is advised.  See also **-n**.
    
           **-J** _rank_ , **--rank** =_rank_
                Limit results to highest/lowest _rank_ ed instances of set-
                valued metrics.  A positive integer will include highest
                valued instances in reporting.  A negative integer will
                include lowest valued instances in reporting.  A value of
                zero performs no ranking.  Ranking does not imply sorting,
                see **-6**.  See also **-8**.
    
           **-K** _spec_ , **--spec-local** =_spec_
                When fetching metrics from a local context (see **-L**), the **-K**
                option may be used to control the DSO PMDAs that should be
                made accessible.  The _spec_ argument conforms to the syntax
                described in pmSpecLocalPMDA(3).  More than one **-K** option may
                be used.
    
           **-L** , **--local-PMDA**
                Use a local context to collect metrics from DSO PMDAs on the
                local host without PMCD.  See also **-K**.
    
           **-m** , **--include-labels**
                Include PCP metric labels in the output.
    
           **-n** , **--invert-filter**
                Perform ranking before live filtering.  By default instance
                live filtering (when requested, see **-j**) happens before
                instance ranking (when requested, see **-J**).  With this option
                the logic is inverted and ranking happens before live
                filtering.
    
           **-N** _predicate_ , **--predicate** =_predicate_
                Specify a comma-separated list of _predicate_ filter reference
                metrics.  By default ranking (see **-J**) happens for each metric
                individually.  With predicates, ranking is done only for the
                specified predicate metrics.  When reporting, rest of the
                metrics sharing the same _instance domain_ (see PCPIntro(1)) as
                the predicate will include only the highest/lowest ranking
                instances of the corresponding predicate.  Ranking does not
                imply sorting, see **-6**.
    
                So for example, using **proc.memory.rss**(resident memory size
                of process) as the _predicate_ metric together with
                **proc.io.total_bytes** and **mem.util.used** as metrics to be
                reported, only the processes using most/least (as per **-J**)
                memory will be included when reporting total bytes written by
                processes.  Since **mem.util.used** is a single-valued metric
                (thus not sharing the same instance domain as the process
                related metrics), it will be reported as usual.
    
           **-O** _origin_ , **--origin** =_origin_
                When reporting archived metrics, start reporting at _origin_
                within the time window (see **-S** and **-T**).  Refer to PCPIntro(1)
                for a complete description of the syntax for _origin_.
    
           **-p** _type_ , **--es-search-type** =_type_
                Elasticsearch search _type_ for measurements.
    
           **-P** _precision_ , **--precision** =_precision_
                Use _precision_ for numeric non-integer output values.  The
                default is to use 3 decimal places (when applicable).  This
                option will _not_ override possible per-metric specifications.
    
           **-q** _scale_ , **--count-scale** =_scale_
                _Unit/scale_ for count metrics, possible values include **count x**
                **10^-1** , **count** , **count x 10** , **count x 10^2** , and so forth from
                **10^-8** to **10^7**.  (These values are currently space-sensitive.)
                This option will _not_ override possible per-metric
                specifications.  See also pmParseUnitsStr(3).
    
           **-Q** _scale_ , **--count-scale-force** =_scale_
                Like **-q** but this option _will_ override per-metric
                specifications.
    
           **-r** , **--raw**
                Output raw metric values, do not convert cumulative counters
                to rates.  This option _will_ override possible per-metric
                specifications.
    
           **-R** , **--raw-prefer**
                Like **-r** but this option will _not_ override per-metric
                specifications.
    
           **-s** _samples_ , **--samples** =_samples_
                The _samples_ argument defines the number of samples to be
                retrieved and reported.  If _samples_ is 0 or **-s** is not
                specified, **pcp2elasticsearch** will sample and report
                continuously (in real time mode) or until the end of the set
                of PCP archives (in archive mode).  See also **-T**.
    
           **-S** _starttime_ , **--start** =_starttime_
                When reporting archived metrics, the report will be
                restricted to those records logged at or after _starttime_.
                Refer to PCPIntro(1) for a complete description of the syntax
                for _starttime_.
    
           **-t** _interval_ , **--interval** =_interval_
                Set the reporting _interval_ to something other than the
                default 1 second.  The _interval_ argument follows the syntax
                described in PCPIntro(1), and in the simplest form may be an
                unsigned integer (the implied units in this case are
                seconds).  See also the **-T** option.
    
           **-T** _endtime_ , **--finish** =_endtime_
                When reporting archived metrics, the report will be
                restricted to those records logged before or at _endtime_.
                Refer to PCPIntro(1) for a complete description of the syntax
                for _endtime_.
    
                When used to define the runtime before **pcp2elasticsearch** will
                exit, if no _samples_ is given (see **-s**) then the number of
                reported samples depends on _interval_ (see **-t**).  If _samples_ is
                given then _interval_ will be adjusted to allow reporting of
                _samples_ during runtime.  In case all of **-T** , **-s** , and **-t** are
                given, _endtime_ determines the actual time **pcp2elasticsearch**
                will run.
    
           **-v** , **--omit-flat**
                Report only set-valued metrics with instances (e.g.
                disk.dev.read) and omit single-valued ``flat'' metrics
                without instances (e.g.  kernel.all.sysfork).  See **-i** and **-I**.
    
           **-V** , **--version**
                Display version number and exit.
    
           **-x** _index_ , **--es-index** =_index_
                Elasticsearch _index_ for metric names.
    
           **-X** _hostid_ , **--es-hostid** =_hostid_
                Elasticsearch _hostid_ for measurements.
    
           **-y** _scale_ , **--time-scale** =_scale_
                _Unit/scale_ for time metrics, possible values include **nanosec** ,
                **ns** , **microsec** , **us** , **millisec** , **ms** , and so forth up to **hour** , **hr**.
                This option will _not_ override possible per-metric
                specifications.  See also pmParseUnitsStr(3).
    
           **-Y** _scale_ , **--time-scale-force** =_scale_
                Like **-y** but this option _will_ override per-metric
                specifications.
    
           **-?** , **--help**
                Display usage message and exit.
    


## OUTPUT FORMAT



           First, **pcp2elasticsearch** sends this type definition to
           Elasticsearch:
    
                   {
                       "mappings": {
                           "properties": {
                               "@timestamp": {
                                  "type": "long"
                               },
                               "host-id": {
                               "type": "text"
                               }
                           }
                       }
                   }
    
           The actual metrics data document is similar to this example:
    
                   {
                       "@host-id": "bozo",
                       "@timestamp": 1507637520,
                       "mem": {
                           "util": {
                               "used": 4525820,
                               "free": 3344000
                           }
                       },
                       "disk": {
                           "partitions": {
                               "@instances": [
                                   {
                                       "read": 1121,
                                       "@id": "sda1"
                                   },
                                   {
                                       "read": 77306,
                                       "@id": "sda2"
                                   },
                                   {
                                       "read": 131,
                                       "@id": "sda3"
                                   }
                               ]
                           }
                       }
                   }
    


## FILES



           _pcp2elasticsearch.conf_
                **pcp2elasticsearch** configuration file (see **-c**)
    
           $PCP_SYSCONF_DIR _/pmrep/*.conf_
                system provided default **pmrep** configuration files
    


## PCP ENVIRONMENT



           Environment variables with the prefix **PCP_** are used to
           parameterize the file and directory names used by PCP.  On each
           installation, the file _/etc/pcp.conf_ contains the local values for
           these variables.  The **$PCP_CONF** variable may be used to specify an
           alternative configuration file, as described in pcp.conf(5).
    
           For environment variables affecting PCP tools, see
           pmGetOptions(3).
    
           Of particular note, **PCP_DISCRETE_ONCE** can be set to ensure that
           discrete metric values are reported only once, unless they change
           at some point.
    


## DEBUGGING OPTIONS



           The **-D** or **--debug** option enables the output of additional
           diagnostics on _stderr_ to help triage problems, although the
           information is sometimes cryptic and primarily intended to provide
           guidance for developers rather end-users.  _debug_ is a comma
           separated list of debugging options; use pmdbg(1) with the **-l**
           option to obtain a list of the available debugging options and
           their meaning.
    


* * *