# pmseries

## NAME



           **pmseries** - display information about performance metric timeseries
    


## SYNOPSIS



           **pmseries**[**-adFiIlLmMnqsStvV?**]  [**-c** _config_] [**-D** _debug_] [**-g** _pattern_]
           [**-h** _host_] [**-p** _port_] [**-w** _window_] [**-Z** _timezone_] [_query_ | _labels_ ...
           | _series_ ... | _source_ ... ]
    


## DESCRIPTION



           **pmseries** displays various types of information about performance
           metrics available through the scalable timeseries facilities of
           the Performance Co-Pilot (PCP) and a distributed key-value data
           store such as **https://valkey/io/ **..
    
           By default **pmseries** communicates with a local key-value server
           however the **-h** and **-p** options can be used to specify an alternate
           key-value server.  If this instance is a node within a cluster,
           all other instances in the cluster will be discovered and used
           automatically.
    
           **pmseries** runs in several different modes - either querying
           timeseries identifiers, metadata or values (already stored in the
           key-value store), or manually loading timeseries.  The latter mode
           is generally only used for loading previously collected (inactive)
           archives, since pmproxy(1) automatically performs this function
           for "live" (actively growing) local pmlogger(1) instances, when
           running in its default time series mode.  See the **TIMESERIES**
           **LOADING** section below and the **-L** option for further details.
    
           Without command line options specifying otherwise, **pmseries** will
           issue a timeseries query to find matching timeseries and values.
           All timeseries are identified using a unique SHA-1 hash which is
           always displayed in a 40-hexdigit human readable form.  These
           hashes are formed using the metadata associated with every metric.
    
           Importantly, this includes all metric metadata (labels, names,
           descriptors).  Metric labels in particular are (as far as
           possible) unique for every machine - on Linux for example the
           labels associated with every metric include the unique
           _/etc/machine-id_ , the hostname, domainname, and other automatically
           generated machine labels, as well as any administrator-defined
           labels from _/etc/pcp/labels_.  These labels can be reported with
           pminfo(1) and the _pmcd.labels_ metric.
    
           See pmLookupLabels(3), pmLookupInDom(3), pmLookupName(3) and
           pmLookupDesc(3) for detailed information about metric labels and
           other metric metadata used in each timeseries identifier hash
           calculation.
    
           The timeseries identifiers provide a higher level (and machine
           independent) identifier than the traditional PCP performance
           metric identifiers (pmID), instance domain identifiers (pmInDom)
           and metric names.  See PCPIntro(1) for more details about these
           traditional identifiers.  However, **pmseries** uses timeseries
           identifiers in much the same way that pminfo(1) uses the lower
           level indom, metric identifiers and metric names.
    
           The default mode of **pmseries** operation (i.e. with no command line
           options) depends on the arguments it is presented.  If all non-
           option arguments appear to be timeseries identifiers (in 40 hex
           digit form) **pmseries** will report metadata for these timeseries -
           refer to the **-a** option for details.  Otherwise, the parameters
           will be treated as a timeseries query.
    


## TIMESERIES QUERIES



           Query expressions are formed using the **pmseries** query language
           described below, but can be as simple as a metric name.
    
           The following is an example of querying timeseries from all hosts
           that match a metric name pattern (globbed):
    
             $ pmseries kernel.all.cpu*
             1d7b0bb3f6aec0f49c54f5210885464a53629b60
             379db729afd63fb9eff436625bd6c55a7adc5cfd
             3dd3b45bb05f96636043e5d58b52b441ce542285
             [...]
             ed2bf325ff6dc7589ec966698e5404b67252306a
             dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
    
           To identify timeseries expression operands, the query language us‐
           es the general syntax:
    
             [_metric.name_] '**{**_metadata qualifiers_**}** ' '**[**_time specification_**]** '
    
           The _metric.name_ component restricts the timeseries query to any
           matching PCP metric name (the list of metric names for a PCP
           archive or live host is reported by pminfo(1) with no arguments
           beyond --**host** or --**archive**).  The **pmseries** syntax extends on that
           of **pminfo** and allows for glob(7) based pattern matching within the
           metric name.  The above describes operands available as the leaves
           of **pmseries** expressions, which may include functions, arithmetic
           operators and other features.  See the **EXPRESSIONS** section below
           for further details.
    


## METADATA QUALIFIERS AND METADATA OPERATORS



           Metadata qualifiers are enclosed by ``curly'' braces (**{}**), and
           further restrict the query results to timeseries operands with
           various metadata properties.  These qualifiers are based on metric
           or instance names, and metric label values, and take the general
           form _metadata.name_ OPERATOR _value_ , such as:
    
             instance.name == "cpu0"
             metric.name != "kernel.all.pswitch"
    
           When using label names, the metadata qualifier is optional and can
           be dropped, such as:
    
             label.hostname == "www.acme.com"
             hostname == "www.acme.com"
    
           For metric and instance names only the string operators apply, but
           for metric label values all operators are available.  The set of
           available operators is:
    
       **Boolean operators**
           All string (label, metrics and instances) and numeric (label) val‐
           ues can be tested for equality ("==") or inequality ("!=").
    
       **String operators**
           Strings can be subject to pattern matching in the form of glob
           matching ("~~"), regular expression matching ("=~"), and regular
           expression non-matching ("!~").  The ":" operator is equivalent to
           "~~" - i.e., glob matching.
    
       **Relational operators (numeric label values only)**
           Numeric label values can be subject to the less than ("<"),
           greater than (">"), less than or equal ("<="), greater than or
           equal (">="), equal ("==") and not equal ("!=") operators.
    
       **Logical operators**
           Multiple metadata qualifiers can be combined with the logical op‐
           erators for AND ("&&") and OR ("||") as in many programming lan‐
           guages.  The comma (",") character is equivalent to logical AND
           ("&&").
    


## TIME SPECIFICATION



           The final (optional) component of a query allows the user to spec‐
           ify a specific time window of interest.  Any time specification
           will result in values being returned for all matching timeseries
           only for the time window specified.
    
           The specification is ``square'' bracket (**[]**) enclosed, and con‐
           sists of one or more comma-separated components.  Each component
           specifies some aspect related to time, taking the general form:
           **keyword** : _value_ , such as:
    
             samples:10
    
       **Sample count**
           The number of samples to return, specified via either the **samples**
           or (equivalent) **count** keyword.  The _value_ provided must be a posi‐
           tive integer.  If no end time is explicitly set (see ``Time win‐
           dow'' later) then the most recent samples will be returned.
    
       **Sample interval**
           An interval between successive samples can be requested using the
           **interval** or (equivalent) **delta** keyword.  The _value_ provided should
           be either a numeric or string value that will be parsed by
           pmParseInterval(3), such as **5**(seconds) or **2min**(minutes).
    
       **Time window**
           Start and end times, and alignments, affecting the returned val‐
           ues.  The keywords match the parameters to the
           pmParseTimeWindow(3) function which will be used to parse them,
           and are: **start** or (equivalent) **begin** , **finish** or (equivalent) **end** ,
           **align** and **offset**.
    
       **Time zones**
           The resulting timestamps can be returned having been evaluated for
           a specific timezone, using the **timezone** or **hostzone** keywords.  The
           _value_ associated with **timezone** will be interpreted by
           pmNewZone(3).  A **true** or **false** value should be associated with
           **hostzone** , and when set to **true** this has the same effect as de‐
           scribed by pmNewContextZone(3).
    


## EXPRESSIONS



           As described above, operands are the leaves of a query expression
           tree.
    
             [_metric.name_] '**{**_metadata qualifiers_**}** ' '**[**_time specification_**]** '
           Note in most of the query expression examples below, the _metadata_
           _qualifiers_ have been omitted for brevity.  In all cases, multiple
           time series may qualify, particularly for the **hostname** label.
    
           In the simple case, a query expression consists of a single
           operand and may just be a metric name.  In the more general case,
           a query expression is either an operand or the argument to a func‐
           tion, or two operands in a binary arithmetic or logical expres‐
           sion.  Most functions take a single argument (an expression),
           though some require additional arguments, e.g.  **rescale**.
    
             _operand_ | _expr operator expr_ | _func_(_expr_[, _arg_])
    
           This grammar shows expressions may be nested, e.g. using the addi‐
           tion (**+**) operator as an example,
    
             _func1_**(**_func2_**(**_expr_**))**
             _func1_**(**_expr_**) +**_func2_**(**_expr_**)**
             _expr_ **+**_func_**(**_expr_**)**
             _func_**(**_expr_**) +**_expr_
             _expr_ **+**_expr_
    
           Rules governing compatibility of operands in an expression gener‐
           ally depend on the function and/or operators and are described be‐
           low individually.  An important rule is that if any time windows
           are specified, then all operands must cover the same number of
           samples, though the time windows may differ individually.  If no
           time windows or sample counts are given, then **pmseries** will return
           a series identifier (SID) instead of a series of timestamps and
           values.  This SID may be used in subsequent **/series/values?se‐**
           **ries=**_SID_ RESTAPI calls, along with a specific time window.
    
       **Arithmetic Operators**
           **pmseries** support addition, subtraction, division and multiplica‐
           tion on each value in the time series of a binary pair of
           operands.  No unary or ternary operators are supported (yet).  In
           all cases, the instance domain and the number of samples of time
           series operands must be the same.  The metadata (units and dimen‐
           sions) must also be compatible.  Depending on the function, the
           result will usually have the same instance domain and (unless not‐
           ed otherwise), the same units as the operands.  The metadata di‐
           mensions (space, time, count) of the result may differ (see be‐
           low).
    
           Expression operands may have different qualifiers, e.g. you can
           perform binary arithmetic on metrics qualified by different labels
           (such as **hostname**), or metric names.  For example, to add the two
           most recent samples of the process context switch (pswitch)
           counter metric for hosts **node88** and **node89** , and then perform rate
           conversion:
    
             $ pmseries 'rate(kernel.all.pswitch{hostname:**node88**}[count:2] +
                              kernel.all.pswitch{hostname:**node89**}[count:2])'
             1cf1a85d5978640ef94c68264d3ae8866cc11f7c
                 [Tue Nov 10 14:39:48.771868000 2020] 71.257509 8e0a59304eb99237b89593a3e839b5bb8b9a9924
    
           Note the resulting time series of values has one less sample than
           the expression operand passed to the **rate** function.
    
           Other rules for arithmetic expressions:
    
           1. if both operands have the semantics of a counter, then only ad‐
           dition and subtraction are allowed
    
           2. if the left operand is a counter and the right operand is not,
           then only multiplication or division are allowed
    
           3. if the left operand is not a counter and the right operand is a
           counter, then only multiplication is allowed.
    
           4. addition and subtraction - the dimensions of the result are the
           same as the dimensions of the operands.
    
           5. multiplication - the dimensions of the result are the sum of
           the dimensions of the operands.
    
           6. division - the dimensions of the result are the difference of
           the dimensions of the operands.
    
       **Functions**
           Expression functions operate on vectors of time series values, and
           may be nested with other functions or expressions as described
           above.  When an operand has multiple instances, there will gener‐
           ally be one result for each series of instances.  For example, the
           result for
    
             $ pmseries 'min_sample(kernel.all.load[count:100])'
    
           will be the smallest value of the 100 most recent samples, treat‐
           ing each of the three load average instances as a separate time
           series.  As an example, for the two most recent samples for each
           of the three instances of the load average metric:
    
             $ pmseries 'kernel.all.load[count:2]'
             726a325c4c1ba4339ecffcdebd240f441ea77848
                 [Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
                 [Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
                 [Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
                 [Tue Nov 10 11:52:20.827134000 2020] 1.120000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
                 [Tue Nov 10 11:52:20.827134000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
                 [Tue Nov 10 11:52:20.827134000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
    
           Using the **min_sample** function :
    
             $ pmseries 'min_sample(kernel.all.load[count:2])'
             11b965bc5f9598034ed9139fb3a78c6c0b7065ba
                 [Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
                 [Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
                 [Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
    
           For singular metrics (with no instance domain), a single value
           will result, e.g. for the five most recent samples of the context
           switching metric:
    
             $ pmseries 'kernel.all.pswitch[count:5]'
             d7832c4fba33bcc980b1a1b614e0508043288480
                 [Tue Nov 10 12:44:59.380666000 2020] 460774294
                 [Tue Nov 10 12:44:49.382070000 2020] 460747232
                 [Tue Nov 10 12:44:39.378545000 2020] 460722370
                 [Tue Nov 10 12:44:29.379029000 2020] 460697388
                 [Tue Nov 10 12:44:19.379096000 2020] 460657412
    
             $ pmseries 'min_sample(kernel.all.pswitch[count:5])'
             1b6e92fb5bc012372f54452734dd03f0f131fa06
                 [Tue Nov 10 12:44:19.379096000 2020] 460657412 d7832c4fba33bcc980b1a1b614e0508043288480
    
           Some **pmseries** functions provide operations across both time and
           instances domain. For example, **max_inst**(_expr_) finds the maximum
           value across instances while **max_sample**(_expr_) finds the maximum
           value across time.
    
           Future versions of **pmseries** may provide functions that perform ag‐
           gregation, interpolation, filtering or transforms in other ways.
    
       **Function Reference**
           **max_inst**(_expr_) the maximum value in the time series for each in‐
           stance of _expr_.  For backwards compatibility, the synonym **max** is
           equivalent to **max_inst**.
    
           **max_sample(**_expr_**)** the maximum value in the time series for each
           sample of _expr_ across time.
    
           **min_inst(**_expr_**)** the minimum value in the time series for each in‐
           stance of _expr_.  For backwards compatibility, the synonym **min** is
           equivalent to **min_inst**.
    
           **min_sample(**_expr_**)** the minimum value in the time series for each
           sample of _expr_ across time.
    
           **sum_inst(**_expr_**)** sum of the values in the time series for each in‐
           stance of _expr_.  For backwards compatibility, the synonym **sum** is
           equivalent to **sum_inst**.
    
           **sum_sample(**_expr_**)** sum of the values in the time series for each
           sample of _expr_ across time.
    
           **avg_inst(**_expr_**)** average of the values in the time series for each
           instance of _expr_.  For backwards compatibility, the synonym **avg** is
           equivalent to **avg_inst**.
    
           **avg_sample(**_expr_**)** average of the values in the time series for each
           sample of _expr_ across time.
    
           **rate(**_expr_**)** the rate with respect to time of each sample.  The giv‐
           en _expr_ must have **counter** semantics and the result will have **in‐**
           **stant** semantics (the time dimension reduced by one).  In addition,
           the result will have one less sample than the operand - this is
           because the first sample cannot be rate converted (two samples are
           required).
    
           **rescale(**_expr_**,**_scale_) rescale the values in the time series for each
           instance of _expr_ to _scale_ (units).  Note that _expr_ should have **in‐**
           **stant** or **discrete** semantics (not **counter** - rate conversion should
           be done first if needed).  The time, space and count dimensions
           between _expr_ and _scale_ must be compatible.  Example: rate convert
           the read throughput counter for each disk instance and then
           rescale to mbytes per second.  Note the native units of
           **disk.dev.read_bytes** is a **counter** of kbytes read from each device
           instance since boot.
    
             $ pmseries 'rescale(rate(disk.dev.read_bytes[count:4]), "mbytes/s")'
    
           **abs(**_expr_**)** the absolute value of each value in the time series for
           each instance of _expr_.  This has no effect if the type of _expr_ is
           unsigned.
    
           **floor(**_expr_**)** rounded down to the nearest integer value of the time
           series for each instance of _expr_.
    
           **round(**_expr_**)** rounded up or down to the nearest integer for each
           value in the time series for each instance of _expr_.
    
           **log(**_expr_**)** logarithm of the values in the time series for each in‐
           stance of _expr_
    
           **sqrt(**_expr_**)** square root of the values in the time series for each
           instance of _expr_
    
           **stdev_inst(**_expr_**)** standard deviation of the values in the time se‐
           ries for each instance of _expr_.
    
           **stdev_sample(**_expr_**)** standard deviation of the values in the time
           series for each sample of _expr_ across time.
    
           **topk_inst(**_expr_**,**_k_) the top _k_ values in the time series for each in‐
           stance of _expr_.
    
           **topk_sample(**_expr_**,**_k_) the top _k_ values in the time series for each
           sample of _expr_ across time.
    
           **nth_percentile_inst(**_expr_**,**_percentile_value_) the nth percentile of
           the values in the time series for each instance of _expr_. Note that
           _percentile_value_ has value in the range 0 to 100.
    
           **nth_percentile_sample(**_expr_**,**_percentile_value_) the nth percentile of
           the values in the time series for each sample of _expr_ across time.
           Note that _percentile_value_ has value in the range 0 to 100.
    
       **Compatibility**
           All operands in an expression must have the same number of sam‐
           ples, but not necessarily the same time window. e.g. you could
           subtract some metric time series from today from that of yesterday
           by giving different time windows and different metrics or quali‐
           fiers, ensuring the same number of samples are given as the
           operands.
    
           Operands in an expression must either all have a time window, or
           none.  If no operands have a time window, then instead of a series
           of time stamps and values, the result will be a time series iden‐
           tifier (_SID_) that may be passed to the **/series/values?series=**_SID_
           REST API function, along with a time window.  For further details,
           see PMWEBAPI(3).
    
           If the semantics of both operands in an arithmetic expression are
           not counter (i.e. **PM_SEM_INSTANT** or **PM_SEM_DISCRETE**) then the re‐
           sult will have semantics **PM_SEM_INSTANT** unless both operands are
           **PM_SEM_DISCRETE** in which case the result is also **PM_SEM_DISCRETE**.
    


## TIMESERIES METADATA



           Using command line options, **pmseries** can be requested to provide
           metadata (metric names, instance names, labels, descriptors) asso‐
           ciated with either individual timeseries or a group of timeseries,
           for example:
    
             $ pmseries -a dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
    
             dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
                 PMID: 60.0.21
                 Data Type: 64-bit unsigned int  InDom: PM_INDOM_NULL 0xffffffff
                 Semantics: counter  Units: millisec
                 Source: f5ca7481da8c038325d15612bb1c6473ce1ef16f
                 Metric: kernel.all.cpu.nice
                 labels {"agent":"linux","domainname":"localdomain",\
                         "groupid":1000,"hostname":"shard",\
                         "latitude":-25.28496,"longitude":152.87886,\
                         "machineid":"295b16e3b6074cc8bdbda8bf96f6930a",\
                         "userid":1000}
    
           The complete set of **pmseries** metadata reporting options are:
    
           **-a** , **--all**
                Convenience option to report all metadata for the given time‐
                series, equivalent to **-deilms**.
    
           **-d** , **--desc**
                Metric descriptions detailing the PMID, data type, data se‐
                mantics, units, scale and associated instance domain.  This
                option has a direct pminfo(1) equivalent.
    
           **-F** , **--fast**
                Query or load series metadata only, not values.
    
           **-g** _pattern_ , **--glob** =_pattern_
                Provide a glob(7) _pattern_ to restrict the report provided by
                the **-i** , **-l** , **-m** , and **-S**.
    
           **-i** , **--instances**
                Metric descriptions detailing the PMID, data type, data se‐
                mantics, units, scale and associated instance domain.
    
           **-I** , **--fullindom**
                Print the InDom in verbose mode.  This option has a direct
                pminfo(1) equivalent.
    
           **-l** , **--labels**
                Print label sets associated with metrics and instances.  La‐
                bels are optional metric metadata described in detail in
                pmLookupLabels(3).  This option has a direct pminfo(1) equiv‐
                alent.
    
           **-m** , **--metrics**
                Print metric names.
    
           **-M** , **--fullpmid**
                Print the PMID in verbose mode.  This option has a direct
                pminfo(1) equivalent.
    
           **-n** , **--names**
                Print comma-separated label names only (not values) for the
                labels associated with metrics and instances.
    
           **-s** , **--series**
                Print timeseries identifiers associated with metrics, in‐
                stances and sources.  These unique identifiers are calculated
                from intrinsic (non-optional) labels and other metric metada‐
                ta associated with each PMAPI context (sources), metrics and
                instances.  Archive, local context or pmcd(1) connections for
                the same host all produce the same source identifier.  This
                option has a direct pminfo(1) equivalent.  See also
                pmLookupLabels(3) and the **-l/--labels** option.
    


## TIMESERIES SOURCES



           A source is a unique identifier (represented externally as a
           40-byte hexadecimal SHA-1 hash) that represents both the live host
           and/or archives from which each timeseries originated.  The con‐
           text for a source identifier (obtained with **-s**) can be reported
           with:
    
           **-S** , **--sources**
                Print names for timeseries sources.  These names are either
                hostnames or fully qualified archive paths.
    
           It is important to note that live and archive sources can and will
           generate the same SHA-1 source identifier hash, provided that the
           context labels remain the same for that host (labels are stored in
           PCP archives and can also be fetched live from pmcd(1)).
    


## TIMESERIES LOADING



           Timeseries metadata and data are loaded either automatically by a
           local pmproxy(1), or manually using a specially crafted **pmseries**
           query and the **-L** /**--load** option:
    
             $ pmseries --load "{source.path: \"$PCP_LOG_DIR/pmlogger/acme\"}"
             pmseries: [Info] processed 2275 archive records from [...]
    
           This query must specify a source archive path, but can also re‐
           strict the import to specific timeseries (using metric names, la‐
           bels, etc) and to a specific time window using the time specifica‐
           tion component of the query language.
    
           As a convenience, if the argument to load is a valid file path as
           determined by access(2), then a short-hand form can be used:
    
             $ pmseries --load $PCP_LOG_DIR/pmlogger/acme.0
    
           NOTE: Timeseries loading is **append-only**(timestamp-wise) and if
           more than **stream.maxlen** entries (defined in _$PCP_SYSCONF_DIR/pm‐_
           _series/pmseries.conf_) are loaded for a given metric, the oldest
           entries are dropped.
    


## OPTIONS



           The available command line options, in addition to timeseries
           metadata and sources options described above, are:
    
           **-c** _config_ , **--config** =_config_
                Specify the _config_ file to use.
    
           **-h** _host_ , **--host** =_host_
                Connect to the key-value server at _host_ , rather than the one
                the localhost.
    
           **-L** , **--load**
                Load timeseries metadata and data into the key-value cluster.
    
           **-p** _port_ , **--port** =_port_
                Connect to the key-value server at _port_ , rather than the de‐
                fault **6379**.
    
           **-q** , **--query**
                Perform a timeseries query.  This is the default action.
    
           **-t** , **--times**
                Report time stamps numerically (in milliseconds) instead of
                the default human readable form.
    
           **-v** , **--values**
                Report all of the known values for given _label_ name(s), or
                report values for the given _series_ identifiers.
    
           **-w** , **--window**
                Provide a time specification that will be applied to values
                being returned when returning values via use of _series_ iden‐
                tifiers (i.e. when not using a _query_ string).  The time spec‐
                ification uses the same square-bracket enclosed form de‐
                scribed earlier in the ``TIME SPECIFICATION'' section.
    
           **-V** , **--version**
                Display version number and exit.
    
           **-Z** _timezone_ , **--timezone** =_timezone_
                Use _timezone_ for the date and time.  _Timezone_ is in the for‐
                mat of the environment variable **TZ** as described in
                environ(7).
    
           **-?** , **--help**
                Display usage message and exit.
    


## EXAMPLES



           The following sample query shows several fundamental aspects of
           the **pmseries** query language:
    
             $ pmseries 'kernel.all.load{hostname:"toium"}[count:2]'
    
             eb713a9cf472f775aa59ae90c43cd7f960f7870f
                 [Thu Nov 14 05:57:06.082861000 2019] 1.0e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
                 [Thu Nov 14 05:57:06.082861000 2019] 6.8e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
                 [Thu Nov 14 05:57:06.082861000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
                 [Thu Nov 14 05:57:16.091546000 2019] 1.6e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
                 [Thu Nov 14 05:57:16.091546000 2019] 6.7e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
                 [Thu Nov 14 05:57:16.091546000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
    
           This query returns the two most recent values for all instances of
           the **kernel.all.load** metric with a _label.hostname_ matching the reg‐
           ular expression "toium".  This is a set-valued metric (i.e., a
           metric with an ``instance domain'' which in this case consists of
           three instances: 1, 5 and 15 minute averages).  The first column
           returned is a timestamp, then a floating point value, and finally
           an instance identifier timeseries hash (two values returned for
           three instances, so six rows are returned).  The metadata for
           these timeseries can then be further examined:
    
             $ pmseries -a eb713a9cf472f775aa59ae90c43cd7f960f7870f
    
             eb713a9cf472f775aa59ae90c43cd7f960f7870f
                 PMID: 60.2.0
                 Data Type: float  InDom: 60.2 0xf000002
                 Semantics: instant  Units: none
                 Source: 0e89c1192db79326900d82131c31399524f0b3ee
                 Metric: kernel.all.load
                 inst [1 or "1 minute"] series b84040ffccd54f839b65140cf139bab51cbbcf62
                 inst [5 or "5 minute"] series a60b5b3bf25e71071c41934fa4d7d251f765f30c
                 inst [15 or "15 minute"] series e1974a062375e6e62370ffadf5b0650dad739480
                 inst [1 or "1 minute"] labels {"agent":"linux","hostname":"toium"}
                 inst [5 or "5 minute"] labels {"agent":"linux","hostname":"toium"}
                 inst [15 or "15 minute"] labels {"agent":"linux","hostname":"toium"}
    


## PCP ENVIRONMENT



           Environment variables with the prefix **PCP_** are used to parameter‐
           ize the file and directory names used by PCP.  On each installa‐
           tion, the file _/etc/pcp.conf_ contains the local values for these
           variables.  The **$PCP_CONF** variable may be used to specify an al‐
           ternative configuration file, as described in pcp.conf(5).
    
           For environment variables affecting PCP tools, see
           pmGetOptions(3).
    


## DEBUGGING OPTIONS



           The **-D** or **--debug** option enables the output of additional diagnos‐
           tics on _stderr_ to help triage problems, although the information
           is sometimes cryptic and primarily intended to provide guidance
           for developers rather end-users.  _debug_ is a comma separated list
           of debugging options; use pmdbg(1) with the **-l** option to obtain a
           list of the available debugging options and their meaning.
    


* * *