# pmlogreduce

## NAME



           **pmlogreduce** - temporal reduction of Performance Co-Pilot archives
    


## SYNOPSIS



           **pmlogreduce**[**-z?**]  [**-A** _align_] [**-D** _debug_] [**-s** _samples_] [**-S**
           _starttime_] [**-t** _interval_] [**-T** _endtime_] [**-v** _volsamples_] [**-Z**
           _timezone_] _input output_
    


## DESCRIPTION



           **pmlogreduce** reads one set of Performance Co-Pilot (PCP) archives
           identified by _input_ and creates a temporally reduced PCP archive
           in _output_.  _input_ is a comma-separated list of names, each of
           which may be the base name of an archive or the name of a
           directory containing one or more archives.  The data reduction
           involves statistical and temporal reduction of samples with an
           output sampling interval defined by the **-t** option in the _output_
           archive (independent of the sampling intervals in the _input_
           archives), and is further controlled by other command line
           arguments.
    
           For some metrics, temporal data reduction is not going to be
           helpful, so for metrics with types **PM_TYPE_AGGREGATE** or
           **PM_TYPE_EVENT** , a warning is issued if these metrics are found in
           _input_ and they will be skipped and not appear in the _output_
           archive.
    


## OPTIONS



           The available command line options are:
    
           **-A** _align_ , **--align** =_align_
                Specify a ``natural'' alignment of the output sample times;
                refer to PCPIntro(1).
    
           **-s** _samples_ , **--samples** =_samples_
                The argument _samples_ defines the number of samples to be
                written to _output_.  If _samples_ is 0 or **-s** is not specified,
                **pmlogreduce** will sample until the end of the set of PCP
                archives, or the end of the time window as specified by **-T** ,
                whichever comes first.  The **-s** option will override the **-T**
                option if it occurs sooner.
    
           **-S** _starttime_ , **--start** =_starttime_
                Define the start of a time window to restrict the samples
                retrieved from the _input_ archives; refer to PCPIntro(1).
    
           **-t** _interval_ , **--interval** =_interval_
                Consecutive samples in the _output_ archive will appear with a
                time delta defined by _interval_ ; refer to PCPIntro(1).  Note
                the default value is 600 (seconds, i.e. 10 minutes).
    
           **-T** _endtime_ , **--finish** =_endtime_
                Define the termination of a time window to restrict the
                samples retrieved from the _input_ archives; refer to
                PCPIntro(1).
    
           **-v** _volsamples_
                The _output_ archive is potentially a multi-volume data set,
                and the **-v** option causes **pmlogreduce** to start a new volume
                after _volsamples_ records have been written to the _output_
                archive.
    
                  Independent of any **-v** option, each volume of an archive is
                  limited to no more than 2^31 bytes, so _pmlogreduce_ will
                  automatically create a new volume for the archive before
                  this limit is reached.
    
           **-z** , **--hostzone**
                Use the local timezone of the host from the _input_ archives
                when displaying the date and time, or interpreting the **-S** and
                **-T** options.  The default is to initially use the timezone of
                the local host.
    
           **-Z** _timezone_ , **--timezone** =_timezone_
                Use _timezone_ when displaying the date and time, or
                interpreting the **-S** and **-T** options.  _Timezone_ is in the
                format of the environment variable **TZ** as described in
                environ(7).
    
           **-?** , **--help**
                Display usage message and exit.
    


## DATA REDUCTION



           The statistical and temporal reduction follows the following
           rules:
    
           1.  Consecutive records from _input_ are read without interpolation,
               and at most one output record is written for each _interval_ ,
               summarizing the performance data over that period.
    
           2.  If the semantics of a metric indicates it is **instantaneous** or
               **discrete** then _output_ value is computed as the arithmetic mean
               of the observations (if any) over each _interval_.
    
           3.  If the semantics of a metric indicates it is a **counter** then
               the following transformations are applied:
               a)  Metrics with 32-bit precision are promoted to 64-bit
                   precision.
               b)  Any counter wrap (overflow) is noted, and appropriate
                   adjustment made in the value of the metric over each
                   _interval_.  This will be correct in the case of a single
                   counter wrap, but will silently **underestimate** in the case
                   where more than one counter wrap occurs between
                   consecutive observations in the _input_ archives, and
                   silently **overestimate** in the case where a counter reset
                   occurs between consecutive observations in the _input_
                   archives; unfortunately these situations cannot be
                   detected, but are believed to be rare events for the sort
                   of production monitoring environments where **pmlogreduce** is
                   most likely to be deployed.
    
           4.  Any changes in instance domains, and indeed all metadata, is
               preserved.
    
           5.  Any ``mark'' records in the _input_ archives (as created by
               pmlogextract(1)) will be preserved in the _output_ archive, so
               periods where no data is available are maintained, and data
               interpolation will **not** occur across these periods when the
               _output_ archive is subsequently processed with PCP
               applications.
    


## CAVEATS



           The preamble metrics (pmcd.pmlogger.archive, pmcd.pmlogger.host,
           and pmcd.pmlogger.port), which are automatically recorded by
           **pmlogger** at the start of the archive, may not be present in the
           archive output by **pmlogreduce**.  These metrics are only relevant
           while the archive is being created, and have no significance once
           recording has finished.
    


## DIAGNOSTICS



           All error conditions detected by **pmlogreduce** are reported on
           _stderr_ with textual (if sometimes terse) explanation.
    
           Should the _input_ archives be corrupted (this can happen if the
           **pmlogger** instance writing the archive suddenly dies), then
           **pmlogreduce** will detect and report the position of the corruption
           in the file, and any subsequent information from the _input_
           archives will not be processed.
    
           If any error is detected, **pmlogreduce** will exit with a non-zero
           status.
    


## FILES



           For each of the _input_ and _output_ archives, several physical files
           are used.
    
           _archive_**.meta**
                metadata (metric descriptions, instance domains, etc.) for
                the archive
    
           _archive_**.0**
                initial volume of metrics values (subsequent volumes have
                suffixes **1** , **2** , ...) - for _input_ these files may have been
                previously compressed with **bzip2**(1) or **gzip**(1) and thus may
                have an additional **.bz2** or **.gz** suffix.
    
           _archive_**.index**
                temporal index to support rapid random access to the other
                files in the archive.
    


## PCP ENVIRONMENT



           Environment variables with the prefix **PCP_** are used to
           parameterize the file and directory names used by PCP.  On each
           installation, the file _/etc/pcp.conf_ contains the local values for
           these variables.  The **$PCP_CONF** variable may be used to specify an
           alternative configuration file, as described in pcp.conf(5).
    


## DEBUGGING OPTIONS



           The **-D** or **--debug** option enables the output of additional
           diagnostics on _stderr_ to help triage problems, although the
           information is sometimes cryptic and primarily intended to provide
           guidance for developers rather end-users.  _debug_ is a comma
           separated list of debugging options; use pmdbg(1) with the **-l**
           option to obtain a list of the available debugging options and
           their meaning.
    
           Debugging options specific to **pmlogger** are as follows:
           ┌────────┬───────────────────────────────────────────────────────┐
           │ **Option** │ **Description** │
           ├────────┼───────────────────────────────────────────────────────┤
           │ **appl0** │ initialization and time window                        │
           ├────────┼───────────────────────────────────────────────────────┤
           │ **appl1** │ report new metric-instance observations               │
           ├────────┼───────────────────────────────────────────────────────┤
           │ **appl2** │ report scan start and end times, dump input and       │
           │        │ output records                                        │
           └────────┴───────────────────────────────────────────────────────┘
    


* * *