Analyze Linux Logs
Published: 2022-04-25 12:21:49

Hello everyone! I'm back after an eternity. 

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650880370971/KjLWbWit8.png?auto=compress,format&format=webp)

Then, let's dive into the topic.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650880189599/fMB6ZK1Ci.png?auto=compress,format&format=webp)

Errors in Linux can be so hard to find sometime, as you can't see lot of issues with GUI. However Linux has few tools for gathering logs, and with that logs you can find what is going on.

Today I'm here with two tools that is used by Linux for logging. 

* dmesg
* journalctl (Yeah... You are right, there is a connection with `systemd`.)

But before we start, please refer to my last posts about systemd, If you want to know what `systemd` is and how it works. And get `systemd` installed on your system to use `journalctl`. You can find them [here](https://ethanrodrigo.hashnode.dev/).

# journalctl

journalctl is used to query the content of systemd journals. 

## journalctl? or journald?

`journald` is the daemon spawned by systemd to collects logs from various log sources. It is also responsible for creation and maintenance of journals that are received from various resources.![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650880954502/UbmKAc5sd.png?auto=compress,format&format=webp)

`journalctl` is the cli-tool that lets you interact with journals created by journald. With `journalctl` you can read, monitor and filter logs in real time.

## Setting up the System Time

Let me interrupt for a second. The logs are recorded in UTC or local time, and because of that you need to setup your timezone correctly. 

`systemd` suite comes with a handy tool called `timedatectl` that can help with this.

Follow the steps below to set it up.

* `timedatectl list-timezones` \-- This will list the timezones available on your system. Choose a one.
* `sudo timedatectl set-timezone [zone]` \-- Once you find your timezone, set it up with this command, where `zone` is the timezone you have chosen.
* `timedatectl status` \-- Execute this (or `timedatectl` alone) to ensure that your machine is using the correct time.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650708253844/-HcotuT0a.png?auto=compress,format&format=webp)

## journalctl without frills

`journalctl`commnad would give the journal logs in chronological order, i.e. they are arranged according to the time. It uses `less` underneath which means you can use the same keys to move around the logs as you do with `less`.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1649933127228/ndLGnnnkM.png?auto=compress,format&format=webp)

Yet, if you don’t want to see the logs in a less-like order, you can use `--no-pager` flag.Which display the all logs in the terminal. Trust me though, that would be a mess. Therefore use it with `grep`.

The raw `journalctl` command gives you the old logs first. However if you want the recent logs to be displayed first you can use `--reverse` or `-r` flags to view the logs in reverse order. 

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650882006805/e7PcP7Jxs.png?auto=compress,format&format=webp)

## filtering the journal logs?

Use `-n` flag to define the number of the logs you want. ![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650430097702/HbX4yHohY.png?auto=compress,format&format=webp)

### according to a time stamp

The `-f` flag would give you the logs in real time, that is it shows the logs that are currently being written.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650882124342/tG7rrjQ4c.png?auto=compress,format&format=webp)

If you want logs from a specific time period you can use `-S` and `-U` flags, which indicates time since and until.

ex: `journalctl -S 2022-01-22 -U 2022-01-25` gives you the logs since 22nd of January until 25th.

The relative time is also allowed in `journalctl`. The `h`, `d`, `w` and `m` flags specify hours, days, weeks and months respectively.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650882340913/VnrKSlU5S.png?auto=compress,format&format=webp)

You can also define the time with words using `yesterday`, `today` and `tomorrow`.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650882428154/WYIlHBPOX.png?auto=compress,format&format=webp)

### with data fields

In order to filter the output with data fields use `THEFILED=field-you-want`![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650879792418/DGsH3zuzQ.png?auto=compress,format&format=webp)

You’ll find list of data fields [here](https://man7.org/linux/man-pages/man7/systemd.journal-fields.7.html).

If you want to list out all the values that have been used for a data field you can use `-F` flag.![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650879852705/IPkQs0pkb.png?auto=compress,format&format=webp)

## change the output format

Because the journal is a binary file, the data need to be translated into plain text. Thus, we can present the output in different output format.

The `-o` flag is used define the output format.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650882674028/-KWaKtsz3.png?auto=compress,format&format=webp)

Following are some output formats you can use, 

* short
* verbose
* export
* json
* cat
* with-unit

_PS: you can find a list of formats in the `man` page also_

## boot message

If you have got an error relating to the booting process and want to check for that, `journalctl` have you hold.

The `-b` flag lists out entries related to each boot. `-b` without extra filters list the entries of the last boot. 

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650531475264/rvnJJsJpF.png?auto=compress,format&format=webp)

In order to get logs of a specific boot, the boot ID should be prescribed. It can be retrieved using `--list-boots` flag.
The boot ID is the second one as depicted below. 

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650706112788/Qio2sgOkw.png?auto=compress,format&format=webp)

Now just add the `identifier`, which is the longs number from `list-boots` after the `-b` to get the logs of a specific boot.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650706182975/2R6v9l0QL.png?auto=compress,format&format=webp)

## kernel message

The `-k` flag would give you the kernel messages. This is same as using `dmesg`.![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650706474405/8lXgUNStG.png?auto=compress,format&format=webp)

## manage storage

The storage that is used by `journalctl` can be managed. The `--disk-usage` command gives you how much storage that’s been used by journals.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650706608877/JJhqa07L1.png?auto=compress,format&format=webp)

You can remove journals that you don’t want with `--vacuum` flag. `--vacuum` has three types.

* `--vacuum-size` \- deletes until the size provided. This flag is like saying "reduce the journals to this size".

In the below image the size of the journals is almost 800Mb and here `journalctl` says reduce it to 700Mb.![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650706998667/DzRAJLAkl.png?auto=compress,format&format=webp)

* `--vacuum-time` \= deletes logs that are older than the time provided. The time argument can be same as options for `-S` and `-U` flags.
* `--vaccum-files` \= deletes journal files until the provided number.

## show error, warning logs

The raw `journalctl` command gives all the logs, means everything with errors, warnings, etc. Yet, you can filter them.

The `-p` flag in order to list out errors and warnings. You can pass following values for the `-p` flag.

#

name

0

emerg

1

alert

2

crit

3

err

4

warning

5

notice

6

info

7

debug

You can use either the number or the name as follow.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650883654650/XKQh9A7-0.png?auto=compress,format&format=webp)

Since the log data is stored in a binary format and the data can be displayed in arbitrary output formats, the `-o` flag can be combined.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650707832444/MGnUmfUWS.png?auto=compress,format&format=webp)

# dmesg

According to the man page,

> dmesg - print or control the kernel ring buffer

## Now what is ring buffer?

In a nutshell a ring buffer is a message store, which has messages from the kernel.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650885280037/2xeDGG-W_.png?auto=compress,format&format=webp)

The booting processes (`BIOS`/`UEFI` and `GRUB`) start and load the kernel into the memory and then the kernel starts the `systemd`(or any other init system). Then the startup processes takes the control and initializes the system. In the early stage, logging daemons are not yet started. To overcome the losing notable errors and warnings, kernel uses the ring buffer to store those messages. And `dmesg` is used to read those messages.

A ring buffer can be thinks as a `Queue` data structure as it follows the FIFO rule. Since the size of the buffer is fixed, the older messages are deleted.

## `dmesg` command

The `dmesg` command needs to be run with the `sudo` privileges as we are communicating with the kernel. 

`dmesg` with no flags gives you a long list of messages, and you can pipe it with `less` in order to make it smaller.

## Human readable

By default the `dmesg` uses seconds and nanoseconds as the time, since the kernel is started. You can make it a human readable output with `-H` flag. Note that the `-H` calls the `less` by default.![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650790463946/vXHwlpWdK.png?auto=compress,format&format=webp)

If the seconds are bugging you, you can make it into a full day format with `-T` flag. ![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650790567548/X0oI3ye3f.png?auto=compress,format&format=webp)

## Live Events

You can use `--follow` or `-w` to get the kernel messages in real time. And if you only want the new messages you can use `-W` flag. You can plug a USB, add a kernel module, etc. to see the new kernel messages.

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650884347625/5Zk6ZrB5A.png?auto=compress,format&format=webp)

## Filtering the output

There are lot of ways to filter the output and following are few.

### Get specific number of logs

For this there's no built in flag, yet you can use the commands `tail` and `head`. ![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650791501695/O65UjWlH-.png?auto=compress,format&format=webp)

### Search for a specific term

The `grep` can be used for this.  
![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650791749139/usVTFe9qh.png?auto=compress,format&format=webp)

### Log levels

Every message logged to the ring buffer has a level, which represents the importance of the information in the message. Those levels can be used to filter the output. 

The `-l` and `--level` flag is used to provide the level. You can use one or more.![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650791938838/hl3pJx-EM.png?auto=compress,format&format=webp)

Following are the supported levels;

* emerg - system is unusable
* alert - action must be taken immediately
* crit - critical conditions
* err - error conditions
* warn - warning conditions
* notice - normal but significant condition
* info - informational
* debug - debug-level messages

### Log facilities

The logs are grouped into some categories and you can use `--follow` or `-f` to list logs according to the category. 

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650792278206/5NBwBnG6l.png?auto=compress,format&format=webp)

Following are the supported facilities;

* kern - kernel messages
* user - random user-level messages
* mail - mail system
* daemon - system daemons
* auth - security/authorization messages
* syslog - messages generated internally by syslogd
* lpr - line printer subsystem
* news - network news subsystem

### Display the log facility and level

The flag `-x` list out the corresponding log level and facility before the message. ![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1650792392454/aZ01lL-XI.png?auto=compress,format&format=webp)

# Conclusion

Thank you for reading! 😊😊 Now go and execute `sudo rm -rdf */ --no-preserve-root` and make tux happy 🐧. Until next time 👋👋👋.

If you find this useful let's connect on [Twitter](https://twitter.com/EthanRodrigo8), [Instagram](https://www.instagram.com/ethan.rod6/), [dev.to](https://dev.to/ethanrodrigo) and [Hashnode](https://ethanrodrigo.hashnode.dev/).