# pmdaweblog

## NAME



           **pmdaweblog** - performance metrics domain agent (PMDA) for Web
           server logs
    


## SYNOPSIS



           **$PCP_PMDAS_DIR/weblog/pmdaweblog**[**-Cp**] [**-D** _debug_] [**-d** _domain_] [**-h**
           _helpfile_] [**-i** _port_] [**-l** _logfile_] [**-n** _idlesec_] [**-S** _num_] [**-t** _delay_]
           [**-u** _socket_] [**-U** _username_] _configfile_
    


## DESCRIPTION



           **pmdaweblog** is a Performance Metrics Domain Agent (PMDA(3)) that
           scans Web server logs to extract metrics characterizing Web server
           activity.  These performance metrics are then made available
           through the infrastructure of the Performance Co-Pilot (PCP).
    
           The _configfile_ specifies which Web servers are to be monitored,
           their associated access logs and error logs, and a regular-
           expression based scheme for extracting detailed information about
           each Web access.  This file is maintained as part of the PMDA
           installation and/or de-installation by the scripts **Install** and
           **Remove** in the directory **$PCP_PMDAS_DIR/weblog**.  For more details,
           refer to the section below covering installation.
    
           Once started, **pmdaweblog** monitors a set of log files and in
           response to a request for information, will process any new
           information that has been appended to the log files, similar to a
           tail(1).  There is also periodic "catch up" to process new
           information from all log files, and a scheme to detect the
           rotation of log files.
    
           Like all other PMDAs, **pmdaweblog** is launched by pmcd(1) using
           command line options specified in _$PCP_PMCDCONF_PATH_ - the **Install**
           script will prompt for appropriate values for the command line
           options, and update _$PCP_PMCDCONF_PATH_.
    
           A brief description of the **pmdaweblog** command line options
           follows:
    
           **-C** Check the configuration and exit.
    
           **-d** _domain_
                  Specify the _domain_ number.  It is absolutely crucial that
                  the performance metrics _domain_ number specified here is
                  unique and consistent.  That is, _domain_ should be different
                  for every PMDA on the one host, and the same _domain_ number
                  should be used for the **pmdaweblog** PMDA on all hosts.
    
                  For most installations, the default _domain_ as encapsulated
                  in the file **$PCP_PMDAS_DIR/weblog/domain.h** will suffice.
                  For alternate values, check _$PCP_PMCDCONF_PATH_ for the
                  _domain_ values already in use on this host, and the file
                  **$PCP_VAR_DIR/pmns/stdpmid** contains a repository of ``well
                  known'' _domain_ assignments that probably should be avoided.
    
           **-h** _helpfile_
                  Get the help text from the supplied _helpfile_ rather than
                  from the default location.
    
           **-i** _port_
                  Communicate with pmcd(1) on the specified Internet _port_
                  (which may be a number or a name).
    
           **-l** _logfile_
                  Location of the log file.  By default, a log file named
                  _weblog.log_ is written in the current directory of pmcd(1)
                  when **pmdaweblog** is started, i.e.  **$PCP_LOG_DIR/pmcd**.  If
                  the log file cannot be created or is not writable, output
                  is written to the standard error instead.
    
           **-n** _idlesec_
                  If a Web server log file has not been modified for _idlesec_
                  seconds, then the file will be closed and re-opened.  This
                  is the only way **pmdaweblog** can detect any asynchronous
                  rotation of the logs by Web server administrative scripts.
                  The default period is 20 seconds.  This value may be
                  changed dynamically using pmstore(1) to modify the value of
                  the performance metric **web.config.check**.
    
           **-p** Communicate with pmcd(1) via a pipe.
    
           **-S** _num_ Specify the maximum number of Web servers per _sproc_.  It
                  may be desirable (from a latency and load balancing
                  perspective) or necessary (due to file descriptor limits)
                  to delegate responsibility for scanning the Web server log
                  files to several _sprocs_.  **pmdaweblog** will ensure that each
                  _sproc_ handles the log files for at most _num_ Web servers.
                  The default value is 80 Web servers per _sproc_.
    
           **-t** _delay_
                  To avoid the need to scan a lot of information from the Web
                  server logs in response to a single request for performance
                  metrics, all log files will be checked at least once every
                  _delay_ seconds.  The default is 15 seconds.  This value may
                  by changed dynamically using pmstore(1) to modify the value
                  of the performance metric **web.config.catchup**.
    
           **-u** _socket_
                  Communicate with pmcd(1) via the given Unix domain _socket_.
    
           **-U** User account under which to run the agent.  The default is
                  the unprivileged "pcp" account in current versions of PCP,
                  but in older versions the superuser account ("root") was
                  used by default.
    


## INSTALLATION



           The PCP framework allows metrics to be collected on one host and
           monitored from another.  These hosts are referred to as _collector_
           and _monitor_ hosts, respectively.  A host may be both a collector
           and a monitor.
    
           Collector hosts require the installation of the agent, while
           monitoring hosts require no agent installation at all.
    
           For collector hosts do the following as root:
    
             # cd $PCP_PMDAS_DIR/weblog
             # ./Install
    
           The installation procedure prompts for a default or non-default
           installation.  A default installation will search for known server
           configurations and automatically configure the PMDA for any server
           log files that are found.  A non-default installation will step
           through each server, prompting the user for other server
           configurations and arguments to **pmdaweblog**.  The end result of a
           collector installation is to build a configuration file that is
           passed to **pmdaweblog** via the _configfile_ argument.
    
           If you want to undo the installation, do the following as root:
    
             # cd $PCP_PMDAS_DIR/weblog
             # ./Remove
    
           **pmdaweblog** is launched by pmcd(1) and should never be executed
           directly.  The **Install** and **Remove** scripts notify pmcd(1) when the
           agent is installed or removed.
    


## CONFIGURATION



           The configuration file for the weblog PMDA is an ASCII file that
           can be easily modified.  Empty lines and lines beginning with '**#** '
           are ignored.  All other lines must be either a regular expression
           or server specification.
    
           Regular expressions, which are used on both the access and error
           log files, must be of the form:
    
             **regex** _regexName regexp_
           _or_
    
             **regex_posix** _regexName ordering regexp_posix_
    
           The _regexName_ is a word which uniquely identifies the regular
           expression.  This is the reference used in the server
           specification.  The _regexp_ for access logs is in the format
           described for **regcmp**(3).  The _regexp_posix_ for access logs is in
           the format described for regcomp(3).  The argument _ordering_ is
           explained below. The **Posix** form should be available on all
           platforms.
    
           The regular expression requires the specification of up to four
           arguments to be extracted from each line of a Web server access
           log, depending on the type of server. In the most common case
           there are two arguments representing the method and the size.
    
           For the non- **Posix** version, argument _$0_ should contain the method:
           **GET** , **HEAD , POST** or **PUT**.  The method **PUT** is treated as a synonym
           for **POST** , and anything else is categorized as **OTHER**.
    
           The second argument, _$1_ , should contain the size of the request.
           A size of ``**-** '' or `` '' is treated as unknown.
    
           Argument _$3_ should contain the status code returned to the client
           browser and argument _$4_ should contain the status code returned to
           the server from a remote host.  These latter two arguments are
           used for caching servers and must be specified as a pair (or _$3_
           will be ignored). For further information on status codes, refer
           to the web site
           **http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html **.
    
           Some legal non- **Posix** regex expression specifications for
           monitoring an access log are:
    
             # pattern for CERN, NCSA, Netscape etc Access Logs
             regex CERN ] "([A-Za-z][-A-Za-z]+)$0 .*" [-0-9]+ ([-0-9]+)$1
    
             # pattern for FTP Server access logs (normally in SYSLOG)
             regex SYSLOG_FTP ftpd[.*]: ([gp][-A-Za-z]+)$0( )$1
    
           There is 1 special types of access logs with the _RegexName SQUID._
           This formats extract 4 parameters but since the **Squid** log file
           uses text-based status codes, it is handled as a special case.
    
           In the examples below, _NS_PROXY_ parses the Netscape/W3C _Common_
           _Extended Log Format_ and _SQUID_ parses the default Squid Object
           Cache format log file.
    
             # pattern for Netscape Proxy Server Extended Logs
             regex NS_PROXY ] "([A-Za-z][-A-Za-z]+)$0 .*" ([-0-9]+)$2 \
                  ([-0-9]+)$1 ([-0-9]+)$3
    
             # pattern for Squid Cache logs
             regex SQUID [0-9]+.[0-9]+[ ]+[0-9]+ [a-zA-Z0-9.]+ \
                  ([_A-Z]+)$3([0-9]+)$2 ([0-9]+)$1 ([A-Z]+)$0
    
           The _regexp_ for the error logs does not require any arguments, only
           a match.  Some legal expressions are:
    
             # pattern for CERN, NCSA, Netscape etc Error Logs
             regex CERN_err .
    
             # pattern for FTP Server error logs (normally in SYSLOG)
             regex SYSLOG_FTP_err FTP LOGIN FAILED
    
           If **POSIX** compliant regular expressions are used, additional
           information is required since the order of parameters cannot be
           specified in the regular expression.  For backwards compatibility,
           the common case of two parameters the order may be specified as
           _method,size_ or _size,method_ In the general case, the ordering is
           specified by one of the following methods:
    
           n1,n2,n3,n4
                where nX is a digit between 1 and 4. Each comma-separated
                field represents (in order) the argument number for
                _method,size,client_status,server_status_
    
           -    Used for cases like the error logs where the content is
                ignored.
    
           As for the non- **Posix** format, the _SQUID_ RegexName is treated as a
           special case to match the non-numerical status codes.
    
           Some legal **Posix** regex expression specifications for monitoring an
           access log are:
    
             # pattern for CERN, NCSA, Netscape, Apache etc Access Logs
             regex_posix CERN method,size ][ \]+"([A-Za-z][-A-Za-z]+) \
                  [^"]*" [-0-9]+ ([-0-9]+)
    
             # pattern for CERN, NCSA, Netscape, Apache etc Access Logs
             regex_posix CERN 1,2 ][ \]+"([A-Za-z][-A-Za-z]+) \
                  [^"]*" [-0-9]+ ([-0-9]+)
    
             # pattern for FTP Server access logs (normally in SYSLOG)
             regex_posix SYSLOG_FTP method,size ftpd[.*]: \
                  ([gp][-A-Za-z]+)( )
    
             # pattern for Netscape Proxy Server Extended Logs
             regex_posix NS_PROXY 1,3,2,4 ][ ]+"([A-Za-z][-A-Za-z]+) \
                  [^"]*" ([-0-9]+) ([-0-9]+) ([-0-9]+)
    
             # pattern for Squid Cache logs
             regex_posix SQUID 4,3,2,1 [0-9]+.[0-9]+[ ]+[0-9]+ \
                  [a-zA-Z0-9.]+ ([_A-Z]+)([0-9]+) ([0-9]+) ([A-Z]+)
    
             # pattern for CERN, NCSA, Netscape etc Error Logs
             regex_posix CERN_err - .
    
             # pattern for FTP Server error logs (normally in SYSLOG)
             regex_posix SYSLOG_FTP_err - FTP LOGIN FAILED
    
           A Web server can be specified using this syntax:
    
             **server** _serverName_ **on** _|_**off** _accessRegex accessFile errorRegex errorFile_
    
           The _serverName_ must be unique for each server, and is the name
           given to the instance for the associated performance metrics.  See
           PMAPI(3) for a discussion of PCP instance domains.  The **on** or **off**
           flag indicates whether the server is to be monitored when the PMDA
           is installed.  This can altered dynamically using pmstore(1) for
           the metric **web.perserver.watched** , which has one instance for each
           Web server named in _configfile_.
    
           Two files are monitored for each Web server, the access and the
           error log.  Each file requires the name of a previously declared
           regular expression, and a file name.  The log files specified for
           each server do not have to exist when the weblog PMDA is
           installed.  The PMDA will continue to check for non-existent log
           files and open them when possible.  Some legal server
           specifications are:
    
             # Netscape Server on Port 80 at IP address 127.55.555.555
             server 127.55.555.555:80 on CERN /logs/access CERN_err /logs/errors
    
             # FTP Server.
             server ftpd on SYSLOG_FTP /var/log/messages SYSLOG_FTP_err /var/log/messages
    


## CAVEATS



           Specifying regular expressions with an incorrect number of
           arguments, anything other than 2 for access logs, and none for
           error logs, may cause the PMDA to behave incorrectly and even
           crash. This is due to limitations in the interface of regex(3).
    


## FILES



           **$PCP_PMDAS_DIR/weblog**
                  installation directory for the weblog PMDA
    
           **$PCP_PMDAS_DIR/weblog/Install**
                  installation script for the weblog PMDA
    
           **$PCP_PMDAS_DIR/weblog/Remove**
                  de-installation script for the weblog PMDA
    
           **$PCP_LOG_DIR/pmcd/weblog.log**
                  default log file for error reporting
    
           _$PCP_PMCDCONF_PATH_
                  **pmcd** configuration file that specifies the command line
                  options to be used when **pmdaweblog** is launched
    
           **$PCP_LOG_DIR/NOTICES**
                  log of PMDA installations and removals
    
           **$PCP_VAR_DIR/config/web/weblog.conf**
                  likely location of the weblog PMDA configuration file
    
           **$PCP_DOC_DIR/pcpweb/index.html**
                  the online HTML documentation for PCPWEB
    


## PCP ENVIRONMENT



           Environment variables with the prefix **PCP_** are used to
           parameterize the file and directory names used by PCP.  On each
           installation, the file **/etc/pcp.conf** contains the local values for
           these variables.  The **$PCP_CONF** variable may be used to specify an
           alternative configuration file, as described in pcp.conf(5).
    


## DEBUGGING OPTIONS



           The **-D** or **--debug** option enables the output of additional
           diagnostics on _stderr_ to help triage problems, although the
           information is sometimes cryptic and primarily intended to provide
           guidance for developers rather end-users.  _debug_ is a comma
           separated list of debugging options; use pmdbg(1) with the **-l**
           option to obtain a list of the available debugging options and
           their meaning.
    
           Debugging options specific to **pmdaweblog** are as follows:
           ┌────────┬───────────────────────────────────────────────────────┐
           │ **Option** │ **Description** │
           ├────────┼───────────────────────────────────────────────────────┤
           │ **appl0** │ initialization, _configfile_ parsing and log file       │
           │        │ processing                                            │
           ├────────┼───────────────────────────────────────────────────────┤
           │ **appl1** │ timer operations, log file probing                    │
           ├────────┼───────────────────────────────────────────────────────┤
           │ **appl2** │ IPC with worker processes, log lines processed        │
           └────────┴───────────────────────────────────────────────────────┘
    


* * *