YugabyteDB advanced analysis: log inspection
Tags: yugabytedb, internals, analysis
Published: 2023-01-30 12:55:06

This blog is about an analysis and troubleshooting option that the tool `yb_stats` provides for YugabyteDB clusters: looking at the logging output. 

## Two log inspection options
`yb_stats` offers two ways to inspect the logs of a YugabyteDB cluster: one is to print the entire (last one megabyte) of the logging from all servers, and the other one is to show only logging lines as they get created, alike the `tail -f` command.

## print-log
The first switch is the print-log option: `yb_stats --print-log`. This switch quite simply takes the logs of all the servers that have a /log endpoint that match the hosts and ports set, orders it by time (!), and then prints the logs. 

## tail-log
The second switch is the tail-log option: `yb_stats --tail-log`. This switch also takes the logs of all the servers that have a /log endpoint that match the hosts and ports set, but after it fetched the logs, the tool remains active, and only prints shows log lines which are produced after the first fetch. This is similar to how `tail -f` works on files, but now for all the servers in scope, over the network.

## Enhanced output
YugabyteDB processes can produce a lot of log information. In order to help log inspection, a number enhancements are made:

1. Timestamps are in a easy to understand format, and the timezone (UTC) is clearly indicated:
    ```
    2023-01-30 11:18:35.746704 UTC
    ```
2. Severity is indicated, and coloured:
    ![Example of partial log output showing severity letters in colour](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8hwwyab8hy77mfdl8stp.png)
    - An informal message is shown with a green 'I'.
    - A warning message is shown with a yellow 'W'.
    - An error message is shown with a red 'E'.
    - A fatal message is shown with a purple 'F'.
    This allows to spot different severities more easy.

## Filtering
Outside of these enhancements, the log output can still get overwhelming, if only because we combine the log output of multiple servers at a single place.

For that reason, a number of filtering options are available:
1. `--log-severity`: By default, the I (informal) messages are not shown. However, it's perfectly possible to add the informal messages, if that is necessary by setting the argument of `--log-severity` to `'IWEF'`. It's also possible to filter for only error and fatal errors by setting it to `'EF'`.
2. `--hostname-match`: By default, the log lines of all hosts that provide glog format errors at /logs at the specified hosts and ports are shown. This might be too much: for example: the YCQL port (12000) will show the /logs output identical to the tablet server http port (9000). It's easy to set `--hostname-match` to `'(7000|9000)'` for only the master and tablet server endpoints, for example.
3. `--stat-name-match`: If you are looking for specific code locations (such as 'log.cc'), or for specific names in the log lines (such as tablet UUID), you can use `--stat-name-match` to filter on these.

If you using YugabyteDB and want to give `yb_stats` a spin: the installation instructions can be found in the [yb_stats book](https://fritshoogland-yugabyte.github.io/yb_stats-book/), in the [install chapter](https://fritshoogland-yugabyte.github.io/yb_stats-book/install.html).