Linux Logging Tutorial: What Are Linux Logs, How to View, Search and Centralize Them
Tags: devops, linux, logging
Published: 2020-07-27 12:17:37

---
title: Linux Logging Tutorial: What Are Linux Logs, How to View, Search and Centralize Them
published: true
description: This guide goes from very high level (where do logs come from) and dives deeper into how to configure Linux logging through Journald and syslog to find and centralize system logs.
tags: devops, linux, logging
canonical_url: https://sematext.com/blog/linux-logs
cover_image: https://sematext.com/wp-content/uploads/2020/06/Linux-logging-tutorial.jpg
---

**TL;DR note**: if you want the `bzip2 -9` version of this post, scroll down to the very last section for some quick pointers. If you want to learn a bit about Linux system logs, please continue, as we'll talk about all these and more:

-   **What are Linux logs** and who generates them
-   Important **types of Linux logs** and their typical location
-   How to **read and search logs**, whether they're written by journald or syslog
-   How to **centralize logs** of many servers in one location. Spoiler alert: the easiest way is to send all system logs to Sematext Cloud in **three commands**, so you can build actionable dashboards:

![](https://sematext.com/wp-content/uploads/2020/06/linux-logging-post-3.png)

## Short Recap: What Are Linux Logs?

Linux logs are pieces of data that Linux writes, related to what the server, kernel, services, and applications running on it are doing, with an associated timestamp. They often come with other structured data, such as a hostname, being a valuable [analysis](https://sematext.com/blog/log-analysis/) and troubleshooting tool for admins when they encounter performance issues. You can read more about logs and why should you monitor them in our [complete guide to log management](https://sematext.com/guides/log-management/). Here's an example of SSH log from `/var/log/auth.log` directory:

    May 5 08:57:27 ubuntu-bionic sshd[5544]: pam_unix(sshd:session): session opened for user vagrant by (uid=0)

Notice how the log contains a few fields, like the timestamp, the hostname, the process writing the log and its PID, before the message itself. In Linux, logs come from different sources, mainly:

-   [Systemd journal](https://sematext.com/blog/journald-logging-tutorial/). Most Linux distros have [systemd](https://systemd.io/) to manage services (like SSH above). Systemd catches the output of these services (i.e., logs like the one above) and writes them to the journal. The journal is written in a binary format, so you'll use [journalctl](https://sematext.com/blog/journald-logging-tutorial#toc-journald-commands-via-journalctl-5) to explore it, like:
```
    $ journalctl
    ...
    May 05 08:57:27 ubuntu-bionic sshd[5544]: pam_unix(sshd:session): session opened for user vagrant by (uid=0)
    ...
```
-   [Syslog](https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/). When there's no systemd, processes like SSH can write to a UNIX socket (e.g., `/dev/log`) in the [syslog message format](https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/#toc-syslog-message-formats-2). A [syslog daemon](https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/#toc-syslog-daemons-0) (e.g., [rsyslog](https://www.rsyslog.com/)) then picks the message, parses it and writes it to various destinations. By default, it writes to files in `/var/log`, which is how we got the earlier message from /var/log/auth.log.
-   **The Linux kernel** writes its own logs to a ring buffer. Systemd or the syslog daemon can read logs from this buffer, then write to the journal or flat files (typically `/var/log/kern.log`). You can also see kernel logs directly via `dmesg`:
```
$ dmesg -T
...
[Tue May 5 08:41:31 2020] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
...
```
-   [Audit logs](https://sematext.com/blog/auditd-logs-auditbeat-elasticsearch-logsene/). These are a special case of kernel messages designed for auditing actions such as file access. You'd typically have a service to listen for such security logs, like auditd. By default, [auditd](https://sematext.com/blog/auditd-logs-auditbeat-elasticsearch-logsene/#toc-audit-logs-in-linux-a-quick-tutorial-on-using-auditd-1) writes audit messages to `/var/log/audit/audit.log`
-   **Application logs**. Non-system applications tend to write to /var/log as well. Here are some popular examples:
    -   Apache HTTPD logs are typically written to `/var/log/httpd` or `/var/log/apache2`. HTTP access logs would be in /var/log/httpd/access.log
    -   MySQL logs typically go to `/var/log/mysql.log` or /var/log/mysqld.log
    -   Older Linux versions would record boot logs via [bootlogd](https://manpages.debian.org/buster/bootlogd/bootlogd.8.en.html) to `/var/log/boot` or `/var/log/boot.log`. Systemd now takes care of this: you can view boot-related logs via `journalctl -b`. Distros without systemd have a syslog daemon reading from the kernel ring buffer, which normally has all the boot messages. So you can find your boot/reboot logs in `/var/log/messages` or /var/log/syslog
    -   Last but not least, you may have your own apps using a [logging library](https://sematext.com/blog/logging-libraries-vs-log-shippers/) to write to a specific file

These sources can interact with each other: journald can forward all its messages to syslog. Applications can write to syslog or the journal. It's Linux, where everything is configurable. But for now, we'll focus on the defaults: where can you **typically** find different types of logs in most modern distributions?

## Log Files Location: Where Are They Stored?

Typically, you'll find Linux server logs in the `/var/log` directory. This is where syslog daemons are normally configured to write. It's also where most applications (e.g., Apache HTTPD) write by default. For Systemd journal, the default location is `/var/log/journal`, but you can't view the files directly because they're binary. So how **do** you view them?

## How to Check Linux Logs

If your Linux distro uses Systemd (and most modern distros do), then all your system logs are in the journal. You can view them with `journalctl`, and you can find the most important [journalctl commands here](https://sematext.com/blog/journald-logging-tutorial/#toc-journald-commands-via-journalctl-5). If your distribution writes to local files via syslog, you can view them with standard text processing tools, such as [cat](https://linux.die.net/man/1/cat), [less](https://linux.die.net/man/1/less) or [grep](https://linux.die.net/man/1/grep):
```
# grep "error" /var/log/syslog | tail
Mar 31 09:48:02 ubuntu-bionic rsyslogd: unexpected GnuTLS error -53 - this could be caused by a broken connection. GnuTLS reports: Error in the push function. [v8.2002.0 try https://www.rsyslog.com/e/2078 ]
...
```
If you're using [auditd](https://linux.die.net/man/8/auditd) to manage audit logs, you can check them in `/var/log/audit.log` by default, but you can also search them with [ausearch](https://sematext.com/blog/auditd-logs-auditbeat-elasticsearch-logsene/#toc-searching-and-analyzing-audit-logs-with-ausearch-and-aureport-5). That said, you're better off shipping these security logs to a central location, especially if you have multiple servers. For this task, a tool like [Auditbeat](https://www.elastic.co/beats/auditbeat) might work better than auditd. We wrote a separate [tutorial on centralizing audit logs with Auditbeat](https://sematext.com/blog/auditd-logs-auditbeat-elasticsearch-logsene/), but in the next section we'll focus on centralizing Linux system logs in general.

## Centralizing Linux Logs

System logs can be in two places: systemd's journal or plain text files written by a syslog daemon. Some distributions (e.g., Ubuntu) have both: journald is set up to forward to syslog. This is done by setting `ForwardToSyslog=Yes` in `journald.conf`.

### Centralizing Logs via Journald

Our recommendation is to use the [journal-upload](https://www.freedesktop.org/software/systemd/man/systemd-journal-upload.html) to [centralize logs](https://sematext.com/blog/log-aggregation/), if the distribution has systemd. You can check this by running `journalctl` \- if the command isn't found, you don't have the journal. As promised earlier, you can [centralize your system logs to Sematext Cloud with three commands](https://sematext.com/docs/integration/journald-integration/):

1.  **Install journal-upload**. On Ubuntu, this works via `sudo apt-get install systemd-journal-remote`
2.  **Configure journal-upload**. In `/etc/systemd/journal-upload.conf`, set `URL=``http://logsene-journald-receiver.sematext.com:80/YOUR_LOGS_TOKEN`
3.  **Start journal-upload** now and on every boot: `systemctl enable systemd-journal-upload && systemctl start systemd-journal-upload`

Alternatively, you can use [Logagent's journal-upload input](https://sematext.com/docs/logagent/input-plugin-journald-upload/) to gather journal entries from one or more machines, before shipping them to a central location. That central location can be [Sematext Cloud](https://sematext.com/logsene), a local [ELK stack](https://sematext.com/guides/elk-stack/) or something else: ![](https://sematext.com/wp-content/uploads/2020/06/linux-logging-post-2.png) If you want to learn more about journald and journalctl, as well as the options you have around centralizing the journal, have a look at our [complete guide to journald](https://sematext.com/blog/journald-logging-tutorial).

### Centralizing Logs via syslog

There are a few scenarios in which centralizing Linux logs with syslog might make sense:

-   Your Linux distribution doesn't have journald. This means system logs go directly to your [syslog daemon](https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/#toc-syslog-daemons-0)
-   You want to **use your syslog daemon to collect and parse application logs** as well. An example is described in our [tutorial for Apache logs with rsyslog and Elasticsearch](https://sematext.com/blog/recipe-apache-logs-rsyslog-parsing-elasticsearch/).
-   You want to **forward journal entries to syslog** (i.e., by setting `ForwardToSyslog=Yes` in `journald.conf`), so you can use a [syslog protocol](https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/#toc-syslog-protocols-6) as a transport. However, this approach will lose some of journald's structured data: journald only forwards [syslog-specific fields](https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/#toc-syslog-message-formats-2).
-   Similar to the above, except that you'd **configure the syslog daemon to read from the journal** (like `journalctl` does). This approach doesn't lose structured data, but is more error prone (e.g., in case of journal corruption) and adds more overhead.

In all situations listed above, data will go through your syslog daemon. From there, you can send it to any of the supported destinations. Most Linux distributions come with [rsyslog](https://www.rsyslog.com/) installed. To forward data to another syslog server via TCP, you can add this line in your `/etc/rsyslog.conf`:
```
*.* @@logsene-syslog-receiver.sematext.com
```
This particular line will forward data to [Sematext Cloud's syslog endpoint](https://sematext.com/docs/logs/syslog/), but you can replace `logsene-syslog-receiver.sematext.com` with the host name of your own syslog server. Some syslog daemons can output data to Elasticsearch via HTTP/HTTPS. [rsyslog is one of them](https://rsyslog.readthedocs.io/en/latest/configuration/modules/omelasticsearch.html) and [so is syslog-ng](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.21/administration-guide/32#TOPIC-1197819). For example, if you use rsyslog on Ubuntu, you'll install the Elasticsearch output module first:
```
sudo apt-get install rsyslog-elasticsearch
```
Then, in the configuration file, you need two elements:

1.  **A template that formats your syslog messages as JSON**, for Elasticsearch to consume
```
template(name="LogseneFormat" type="list" option.json="on") {
 constant(value="{")
 constant(value="\\"@timestamp\\":\\"")
 property(name="timereported" dateFormat="rfc3339")
 constant(value="\\",\\"message\\":\\"")
 property(name="msg")
 constant(value="\\",\\"host\\":\\"")
 property(name="hostname")
 constant(value="\\",\\"severity\\":\\"")
 property(name="syslogseverity-text")
 constant(value="\\",\\"facility\\":\\"")
 property(name="syslogfacility-text")
 constant(value="\\",\\"syslog-tag\\":\\"")
 property(name="syslogtag")
 constant(value="\\",\\"source\\":\\"")
 property(name="programname")
 constant(value="\\"}")
}
```
2.  **An action that forwards data to Elasticsearch**, using the template specified above
```
module(load="omelasticsearch")
action(type="omelasticsearch"
 template="LogseneFormat" # the template that you defined earlier
 searchIndex="LOGSENE_APP_TOKEN_GOES_HERE"
 server="logsene-receiver.sematext.com"
 serverport="443"
 usehttps="on"
 bulkmode="on"
 queue.dequeuebatchsize="100" # how many messages to send at once
 action.resumeretrycount="-1") # buffer messages if connection fails
```
The above example shows how to send messages to [Sematext Cloud's Elasticsearch API](https://sematext.com/docs/logs/index-events-via-elasticsearch-api/), but you can adjust the action element to point it to your local Elasticsearch:

-   `searchIndex` would be your own [rolling index alias](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/indices-rollover-index.html)
-   `server` would be the hostname of an Elasticsearch node
-   `serverport` can be 9200 or a custom port Elasticsearch listens to
-   `usehttps="off"` would send data over plain HTTP

Whether you use a syslog protocol, the Elasticsearch API or something else, it's better to **forward syslog directly** from the syslog daemon than to **tail individual files from** /var/log using a [different log shipper](https://sematext.com/blog/logstash-alternatives/). Tailing files will add overhead and miss some of the metadata, such as facility or severity. Which is not to say that files in /var/log are useless. You'll need them in two scenarios:

-   Logs of applications that write directly to /var/log. For example, HTTP logs, FTP logs, mysql logs and so on. You can tail such files with a log shipper. We have tutorials on [parsing apache logs with rsyslog](https://sematext.com/blog/recipe-apache-logs-rsyslog-parsing-elasticsearch/) and [with Logstash](https://sematext.com/blog/getting-started-with-logstash/).
-   **Process system logs with UNIX text tools like grep**. Here, different log files contain different kinds of data. We'll look at the typical configuration in the next section.

## What Are the Most Important Log Files You Should Monitor

By default, some distributions write system logs to syslog (either directly or from the journal). The syslog daemon writes these logs to files under `/var/log`. Typically that syslog daemon is rsyslog, though syslog-ng works in a similar fashion. In this section, we'll look at the important log files and:

-   what kind of information you'll find in them
-   how is rsyslog configured to write there (in case you want to change the configuration)
-   how to view the same information with `journalctl`, in case it doesn't forward to syslog

### /var/log/syslog or /var/log/messages

This is the “catch-all” of syslog. For example:
```
# logger "this is a test"
# tail -1 /var/log/syslog
May 7 15:33:11 ubuntu-bionic test-user: this is a test
```
Typically, you'll find all messages here (error logs, informational messages, and every other [severity](https://en.wikipedia.org/wiki/Syslog#Severity_level)), as this line from `/etc/rsyslog.conf` suggests:
```
*.* /var/log/syslog
```
The only exception is the [stop action](https://www.rsyslog.com/doc/v8-stable/configuration/actions.html#discard-stop). For example, you may find something like this:
```
:msg,contains,"[UFW " /var/log/ufw.log
& stop
```
In plain English, this block says:

-   If the [msg property](https://www.rsyslog.com/doc/v8-stable/configuration/properties.html#message-properties) of this message contains "\[UFW "
-   Then write to /var/log/ufw.log (the [file output module](https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfile.html) is implied)
-   If the action succeeds (&), then don't process this message further (stop)

So if the `/var/log/syslog` action comes later, it won't write UFW messages there. If there's nothing in `/var/log/syslog` or `/var/log/messages`, you probably have journald set up not to forward to syslog. The same data (and more) can be viewed via `journalctl` with no parameters. By default, `journalctl` pages data through `less`, but if you want to filter through `grep` you'll need to disable paging:
```
# journalctl --no-pager | grep "this is a test"
May 07 15:33:11 ubuntu-bionic test-user[7526]: this is a test
```
### /var/log/kern.log or /var/log/dmesg

This is where kernel messages go by default:
```
Apr 17 16:47:28 ubuntu-bionic kernel: [ 0.004000] console [tty1] enabled
```
It's really down to filtering syslog messages by the `kern` facility:
```
kern.* /var/log/kern.log
```
If you don't have syslog (or the file is missing) and you have journald, you can show kernel messages in `journalctl`:
```
# journalctl -k
...
Apr 17 16:47:28 ubuntu-bionic kernel: console [tty1] enabled
...
```
### /var/log/auth.log or /var/log/secure

This is where you find authentication messages, generated by services like sshd:
```
May 7 15:03:09 ubuntu-bionic sshd[1202]: pam_unix(sshd:session): session closed for user vagrant
```
This is another filter by facility, this time by two values (`auth` and `authpriv`):
```
auth,authpriv.* /var/log/auth.log
```
You can do such filters in `journalctl` as well, except that you have to provide [numeric facility levels](https://en.wikipedia.org/wiki/Syslog#Facility):
```
# journalctl SYSLOG_FACILITY=4 SYSLOG_FACILITY=10
...
May 7 15:03:09 ubuntu-bionic sshd[1202]: pam_unix(sshd:session): session closed for user vagrant
...
```
### /var/log/cron.log

This is where your [cron](http://man7.org/linux/man-pages/man8/cron.8.html) messages go (i.e., jobs that run regularly):
```
May 06 08:19:01 localhost.localdomain anacron[1142]: Job `cron.daily' started
```
Yet another facility filter:
```
cron.* /var/log/cron
```
With journalctl, you'd do:
```
# journalctl SYSLOG_FACILITY=9
```
### /var/log/mail.log or /var/log/maillog

Email daemons such as Postfix typically log to syslog in the `mail` facility, just like `cron` logs to the `cron` facility. Then, rsyslog puts these logs in a different file:
```
mail.* /var/log/mail.log
```
If you're using journald, you can still view mail logs with:
```
# journalctl SYSLOG_FACILITY=2
```
Because journald exposes the [syslog API](https://linux.die.net/man/3/syslog), everything that normally goes to syslog ends up in the journal.

## TL;DR Takeaways

Let's summarize the actionables here:

-   The location and format of your Linux system logs **depends on how your distro is configured**.
-   **Most distros have systemd**. It means all your system **logs live in the journal**. To view and search it, **use journalctl**. Use the [complete guide to journald](https://sematext.com/blog/journald-logging-tutorial) for reference.
-   **Some distros get system logs to syslog**. Either directly or through the journal. In this case you likely have logs written to various files in `/var/log`. Have a look at the section above for details on each important file.
-   Either way, if you manage multiple servers, you'll want to centralize system logs with a [log management software](https://sematext.com/blog/best-log-management-tools/) such as [Sematext Cloud](https://sematext.com/cloud). Sematext makes this very easy, as it has both [journald integration](https://sematext.com/docs/integration/journald-integration/) and [syslog integration](https://sematext.com/docs/logs/syslog/). Though you can use your own [ELK stack](https://sematext.com/guides/elk-stack/) if you prefer [build vs buy](https://sematext.com/elastic-stack-alternative/).
-   If you need help with your own ELK stack, please reach out, as we provide [ELK stack consulting](https://sematext.com/consulting/logging/), [Elasticsearch production support](https://sematext.com/support/elasticsearch-production-support/) and [Elasticsearch and ELK stack training classes](https://sematext.com/training/elasticsearch/)