# pcp2spark

## NAME



           **pcp2spark** - pcp-to-spark metrics exporter
    


## SYNOPSIS



           **pcp2spark**[**-5CGHIjLmnrRvV?**]  [**-4** _action_] [**-8** |**-9** _limit_] [**-a**
           _archive_] [**-A** _align_] [**--archive-folio** _folio_] [**-b** |**-B** _space-scale_]
           [**-c** _config_] [**--container** _container_] [**-D** _debug_] [**--daemonize**] [**-e**
           _derived_] [**-g** _server_] [**-h** _host_] [**-i** _instances_] [**-J** _rank_] [**-K** _spec_]
           [**-N** _predicate_] [**-O** _origin_] [**-p** _port_] [**-P** |**-0** _precision_] [**-q** |**-Q**
           _count-scale_] [**-s** _samples_] [**-S** _starttime_] [**-t** _interval_] [**-T**
           _endtime_] [**-y** |**-Y** _time-scale_] _metricspec_ [...]
    


## DESCRIPTION



           **pcp2spark** is a customizable performance metrics exporter tool from
           PCP to Apache Spark.  Any available performance metric, live or
           archived, system and/or application, can be selected for exporting
           using either command line arguments or a configuration file.
    
           **pcp2spark** acts as a bridge which provides a network socket stream
           on a given address/port which an Apache Spark worker task can
           connect to and pull the configured PCP metrics from **pcp2spark**
           exporting them using the streaming extensions of the Apache Spark
           API.
    
           **pcp2spark** is a close relative of pmrep(1).  Refer to pmrep(1) for
           the _metricspec_ description accepted on **pcp2spark** command line.
           See pmrep.conf(5) for description of the **pcp2spark.conf**
           configuration file syntax.  This page describes **pcp2spark** specific
           options and configuration file differences with pmrep.conf(5).
           pmrep(1) also lists some usage examples of which most are
           applicable with **pcp2spark** as well.
    
           Only the command line options listed on this page are supported,
           other options available for pmrep(1) are not supported.
    
           Options via environment values (see pmGetOptions(3)) override the
           corresponding built-in default values (if any).  Configuration
           file options override the corresponding environment variables (if
           any).  Command line options override the corresponding
           configuration file options (if any).
    


## GENERAL USAGE



           A general setup for making use of **pcp2spark** would involve the user
           configuring **pcp2spark** for the PCP metrics to export followed by
           starting the **pcp2spark** application. The **pcp2spark** application will
           then wait and listen on the given address/port for a connection
           from an Apache Spark worker thread to be started.  The worker
           thread will then connect to **pcp2spark.**
    
           When an Apache Spark worker thread has connected, **pcp2spark** will
           begin streaming PCP metric data to Apache Spark until the worker
           thread completes or the connection is interrupted.  If the
           connection is interrupted or the socket is closed from the Apache
           Spark worker thread **pcp2spark** will exit.
    
           For an example Apache Spark worker job which will connect to an
           **pcp2spark** instance on a given address/port and pull in PCP metric
           data see the example provided in the PCP examples directory for
           **pcp2spark**(often provided by the PCP development package) or the
           online version at
           _https://github.com/performancecopilot/pcp/blob/main/src/pcp2spark/ _.
    


## CONFIGURATION FILE



           **pcp2spark** uses a configuration file with syntax described in
           pmrep.conf(5).  The following options are common with **pmrep.conf** :
           **version** , **source** , **speclocal** , **derived** , **header** , **globals** , **samples** ,
           **interval** , **type** , **type_prefer** , **ignore_incompat** , **names_change** ,
           **instances** , **live_filter** , **rank** , **limit_filter** , **limit_filter_force** ,
           **invert_filter** , **predicate** , **omit_flat** , **include_labels** , **precision** ,
           **precision_force** , **count_scale** , **count_scale_force** , **space_scale** ,
           **space_scale_force** , **time_scale** , **time_scale_force**.  The rest of the
           **pmrep.conf** options are recognized but ignored for compatibility.
    
       **pcp2spark specific options**
           spark_server (string)
               Specify the address on which **pcp2spark** will listen for
               connections from an Apache Spark worker thread.  Corresponding
               command line option is **-g**.  Defaults to **127.0.0.1**.
    
           spark_port (integer)
               Specify the port on which **pcp2spark** will listen for
               connections.  Corresponding command line option is **-p**.
               Defaults to **44325**.
    


## OPTIONS



           The available command line options are:
    
           **-0** _precision_ , **--precision-force** =_precision_
                Like **-P** but this option _will_ override per-metric
                specifications.
    
           **-4** _action_ , **--names-change** =_action_
                Specify which _action_ to take on receiving a metric names
                change event during sampling.  These events occur when a PMDA
                discovers new metrics sometime after starting up, and informs
                running client tools like **pcp2spark**.  Valid values for _action_
                are **update**(refresh metrics being sampled), **ignore**(do
                nothing - the default behaviour) and **abort**(exit the program
                if such an event occurs).
    
           **-5** , **--ignore-unknown**
                Silently ignore any metric name that cannot be resolved.  At
                least one metric must be found for the tool to start.
    
           **-8** _limit_ , **--limit-filter** =_limit_
                Limit results to instances with values above/below _limit_.  A
                positive integer will include instances with values at or
                above the limit in reporting.  A negative integer will
                include instances with values at or below the limit in
                reporting.  A value of zero performs no limit filtering.
                This option will _not_ override possible per-metric
                specifications.  See also **-J** and **-N**.
    
           **-9** _limit_ , **--limit-filter-force** =_limit_
                Like **-8** but this option _will_ override per-metric
                specifications.
    
           **-a** _archive_ , **--archive** =_archive_
                Performance metric values are retrieved from the set of
                Performance Co-Pilot (PCP) archive files identified by the
                _archive_ argument, which is a comma-separated list of names,
                each of which may be the base name of an archive or the name
                of a directory containing one or more archives.
    
           **-A** _align_ , **--align** =_align_
                Force the initial sample to be aligned on the boundary of a
                natural time unit _align_.  Refer to PCPIntro(1) for a complete
                description of the syntax for _align_.
    
           **--archive-folio** =_folio_
                Read metric source archives from the PCP archive _folio_
                created by tools like pmchart(1) or, less often, manually
                with mkaf(1).
    
           **-b** _scale_ , **--space-scale** =_scale_
                _Unit/scale_ for space (byte) metrics, possible values include
                **bytes** , **Kbytes** , **KB** , **Mbytes** , **MB** , and so forth.  This option
                will _not_ override possible per-metric specifications.  See
                also pmParseUnitsStr(3).
    
           **-B** _scale_ , **--space-scale-force** =_scale_
                Like **-b** but this option _will_ override per-metric
                specifications.
    
           **-c** _config_ , **--config** =_config_
                Specify the _config_ file or directory to use.  In case _config_
                is a directory all files in it ending **.conf** will be included.
                The default is the first found of: _./pcp2spark.conf_ ,
                $HOME _/.pcp2spark.conf_ , $HOME _/pcp/pcp2spark.conf_ , and
                $PCP_SYSCONF_DIR _/pcp2spark.conf_.  For details, see the above
                section and pmrep.conf(5).
    
           **--container** =_container_
                Fetch performance metrics from the specified _container_ ,
                either local or remote (see **-h**).
    
           **-C** , **--check**
                Exit before reporting any values, but after parsing the
                configuration and metrics and printing possible headers.
    
           **--daemonize**
                Daemonize on startup.
    
           **-e** _derived_ , **--derived** =_derived_
                Specify _derived_ performance metrics.  If _derived_ starts with
                a slash (``/'') or with a dot (``.'') it will be interpreted
                as a PCP derived metrics configuration file, otherwise it
                will be interpreted as comma- or semicolon-separated derived
                metric expressions.  For complete description of derived
                metrics and PCP derived metrics configuration files see
                pmLoadDerivedConfig(3) and pmRegisterDerived(3).
                Alternatively, using pmrep.conf(5) configuration syntax
                allows defining derived metrics as part of metricsets.
    
                In case of issues with derived metrics, review the
                aforementioned manual pages in detail and ensure all the
                required metrics are available, especially when using
                archives.  Use **-Dderive** to see additional debug information
                about parsing derived metrics.
    
           **-g** _server_ , **--spark-server** =_server_
                **pcp2spark** local _server_ address.
    
           **-G** , **--no-globals**
                Do not include global metrics in reporting (see
                pmrep.conf(5)).
    
           **-h** _host_ , **--host** =_host_
                Fetch performance metrics from pmcd(1) on _host_ , rather than
                from the default localhost.
    
           **-H** , **--no-header**
                Do not print any headers.
    
           **-i** _instances_ , **--instances** =_instances_
                Retrieve and report only the specified metric _instances_.  By
                default all instances, present and future, are reported.
    
                Refer to pmrep(1) for complete description of this option.
    
           **-I** , **--ignore-incompat**
                Ignore incompatible metrics.  By default incompatible metrics
                (that is, their type is unsupported or they cannot be scaled
                as requested) will cause **pcp2spark** to terminate with an error
                message.  With this option all incompatible metrics are
                silently omitted from reporting.  This may be especially
                useful when requesting non-leaf nodes of the PMNS tree for
                reporting.
    
           **-j** , **--live-filter**
                Perform instance live filtering.  This allows capturing all
                named instances even if processes are restarted at some point
                (unlike without live filtering).  Performing live filtering
                over a huge number of instances will add some internal
                overhead so a bit of user caution is advised.  See also **-n**.
    
           **-J** _rank_ , **--rank** =_rank_
                Limit results to highest/lowest _rank_ ed instances of set-
                valued metrics.  A positive integer will include highest
                valued instances in reporting.  A negative integer will
                include lowest valued instances in reporting.  A value of
                zero performs no ranking.  Ranking does not imply sorting,
                see **-6**.  See also **-8**.
    
           **-K** _spec_ , **--spec-local** =_spec_
                When fetching metrics from a local context (see **-L**), the **-K**
                option may be used to control the DSO PMDAs that should be
                made accessible.  The _spec_ argument conforms to the syntax
                described in pmSpecLocalPMDA(3).  More than one **-K** option may
                be used.
    
           **-L** , **--local-PMDA**
                Use a local context to collect metrics from DSO PMDAs on the
                local host without PMCD.  See also **-K**.
    
           **-m** , **--include-labels**
                Include PCP metric labels in the output.
    
           **-n** , **--invert-filter**
                Perform ranking before live filtering.  By default instance
                live filtering (when requested, see **-j**) happens before
                instance ranking (when requested, see **-J**).  With this option
                the logic is inverted and ranking happens before live
                filtering.
    
           **-N** _predicate_ , **--predicate** =_predicate_
                Specify a comma-separated list of _predicate_ filter reference
                metrics.  By default ranking (see **-J**) happens for each metric
                individually.  With predicates, ranking is done only for the
                specified predicate metrics.  When reporting, rest of the
                metrics sharing the same _instance domain_ (see PCPIntro(1)) as
                the predicate will include only the highest/lowest ranking
                instances of the corresponding predicate.  Ranking does not
                imply sorting, see **-6**.
    
                So for example, using **proc.memory.rss**(resident memory size
                of process) as the _predicate_ metric together with
                **proc.io.total_bytes** and **mem.util.used** as metrics to be
                reported, only the processes using most/least (as per **-J**)
                memory will be included when reporting total bytes written by
                processes.  Since **mem.util.used** is a single-valued metric
                (thus not sharing the same instance domain as the process
                related metrics), it will be reported as usual.
    
           **-O** _origin_ , **--origin** =_origin_
                When reporting archived metrics, start reporting at _origin_
                within the time window (see **-S** and **-T**).  Refer to PCPIntro(1)
                for a complete description of the syntax for _origin_.
    
           **-p** _port_ , **--spark-port** =_port_
                **pcp2spark** local _port_.
    
           **-P** _precision_ , **--precision** =_precision_
                Use _precision_ for numeric non-integer output values.  The
                default is to use 3 decimal places (when applicable).  This
                option will _not_ override possible per-metric specifications.
    
           **-q** _scale_ , **--count-scale** =_scale_
                _Unit/scale_ for count metrics, possible values include **count x**
                **10^-1** , **count** , **count x 10** , **count x 10^2** , and so forth from
                **10^-8** to **10^7**.  (These values are currently space-sensitive.)
                This option will _not_ override possible per-metric
                specifications.  See also pmParseUnitsStr(3).
    
           **-Q** _scale_ , **--count-scale-force** =_scale_
                Like **-q** but this option _will_ override per-metric
                specifications.
    
           **-r** , **--raw**
                Output raw metric values, do not convert cumulative counters
                to rates.  This option _will_ override possible per-metric
                specifications.
    
           **-R** , **--raw-prefer**
                Like **-r** but this option will _not_ override per-metric
                specifications.
    
           **-s** _samples_ , **--samples** =_samples_
                The _samples_ argument defines the number of samples to be
                retrieved and reported.  If _samples_ is 0 or **-s** is not
                specified, **pcp2spark** will sample and report continuously (in
                real time mode) or until the end of the set of PCP archives
                (in archive mode).  See also **-T**.
    
           **-S** _starttime_ , **--start** =_starttime_
                When reporting archived metrics, the report will be
                restricted to those records logged at or after _starttime_.
                Refer to PCPIntro(1) for a complete description of the syntax
                for _starttime_.
    
           **-t** _interval_ , **--interval** =_interval_
                Set the reporting _interval_ to something other than the
                default 1 second.  The _interval_ argument follows the syntax
                described in PCPIntro(1), and in the simplest form may be an
                unsigned integer (the implied units in this case are
                seconds).  See also the **-T** option.
    
           **-T** _endtime_ , **--finish** =_endtime_
                When reporting archived metrics, the report will be
                restricted to those records logged before or at _endtime_.
                Refer to PCPIntro(1) for a complete description of the syntax
                for _endtime_.
    
                When used to define the runtime before **pcp2spark** will exit,
                if no _samples_ is given (see **-s**) then the number of reported
                samples depends on _interval_ (see **-t**).  If _samples_ is given
                then _interval_ will be adjusted to allow reporting of _samples_
                during runtime.  In case all of **-T** , **-s** , and **-t** are given,
                _endtime_ determines the actual time **pcp2spark** will run.
    
           **-v** , **--omit-flat**
                Report only set-valued metrics with instances (e.g.
                disk.dev.read) and omit single-valued ``flat'' metrics
                without instances (e.g.  kernel.all.sysfork).  See **-i** and **-I**.
    
           **-V** , **--version**
                Display version number and exit.
    
           **-y** _scale_ , **--time-scale** =_scale_
                _Unit/scale_ for time metrics, possible values include **nanosec** ,
                **ns** , **microsec** , **us** , **millisec** , **ms** , and so forth up to **hour** , **hr**.
                This option will _not_ override possible per-metric
                specifications.  See also pmParseUnitsStr(3).
    
           **-Y** _scale_ , **--time-scale-force** =_scale_
                Like **-y** but this option _will_ override per-metric
                specifications.
    
           **-?** , **--help**
                Display usage message and exit.
    


## FILES



           _pcp2spark.conf_
                **pcp2spark** configuration file (see **-c**)
    
           $PCP_SYSCONF_DIR _/pmrep/*.conf_
                system provided default **pmrep** configuration files
    


## PCP ENVIRONMENT



           Environment variables with the prefix **PCP_** are used to
           parameterize the file and directory names used by PCP.  On each
           installation, the file _/etc/pcp.conf_ contains the local values for
           these variables.  The **$PCP_CONF** variable may be used to specify an
           alternative configuration file, as described in pcp.conf(5).
    
           For environment variables affecting PCP tools, see
           pmGetOptions(3).
    
           Of particular note, **PCP_DISCRETE_ONCE** can be set to ensure that
           discrete metric values are reported only once, unless they change
           at some point.
    


## DEBUGGING OPTIONS



           The **-D** or **--debug** option enables the output of additional
           diagnostics on _stderr_ to help triage problems, although the
           information is sometimes cryptic and primarily intended to provide
           guidance for developers rather end-users.  _debug_ is a comma
           separated list of debugging options; use pmdbg(1) with the **-l**
           option to obtain a list of the available debugging options and
           their meaning.
    


* * *