Tutorial: Logging with journald
Tags: journald, journalctl, syslog, elk
Published: 2020-06-09 07:37:39

---
title: Tutorial: Logging with journald
published: true
description: Logging with journald: why use it, how to use it, how to centralize journald logs; comparison and interaction with syslog
tags: journald, journalctl, syslog, elk
canonical_url: https://sematext.com/blog/journald-logging-tutorial
cover_image: https://sematext.com/wp-content/uploads/2020/04/Logging-wiht-journalid.jpg
---

If you're using Linux, I'm sure you bumped into [journald](https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html): it's what most distros use by default for system logging. Most applications running as a service will also log to the journal. So how do you make use of these logs to:

-   find the error or debug message that you're looking for?
-   make sure logs don't fill your disk?
-   centralize journals so you don't have to ssh to each box?

In this post, we'll answer all the above and more. We will dive into the following topics:

-   **what is journald**, how it came to be and what are its benefits
-   **main configuration options**, like when to remove old logs so you don't run out of disk
-   **journald and containers**: can/should containers log to the journal?
-   **journald vs syslog**: advantages and disadvantages of both, how they integrate
-   **ways to centralize journals**. Advantages and disadvantages of each method, and which is [the best](https://sematext.com/product-updates/%23/2020/we-have-a-new-logs-integration-for-journald). Spoiler alert: you can [configure journald to send logs directly to Sematext Cloud](https://sematext.com/docs/integration/journald-integration/); or you can [use the open-source Logagent as a journald aggregator](https://sematext.com/docs/logagent/input-plugin-journald-upload/). Either way, you'll have one place to search and analyze your journal events:

![](https://sematext.com/wp-content/uploads/2020/04/jornald-logging-post-image2.png)

There are lots of other options to centralize journal entries, and lots of tools to help. We'll explore them in detail, but before that, let's zoom in to journald itself.

## What is journald?

**journald** is the part of [systemd](https://systemd.io/) that deals with logging. **systemd**, at its core, is in charge of managing services: it starts them up and keeps them alive.

All services and systemd itself need to log: “ssh started” or “user root logged in”, they might say. That's where journald comes in: to capture these logs, record them, make them easy to find, and remove them when they pass a certain age.

## Why use journald?

In short, because syslog sucks :) Jokes aside, the [paper announcing journald](https://docs.google.com/document/pub?id%3D1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUEHIX3MSTs%26pli%3D1) explained that systemd needed functionality that was hard to get through [existing syslog implementations](https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/). Examples include structured logging, indexing logs for fast search, access control and signed messages.

As you might expect, [not everyone agrees with these statements](https://rainer.gerhards.net/2013/05/rsyslog-vs-systemd-journal.html) or the general approach systemd took with journald. But by now, systemd is adopted by most Linux distributions, and it includes journald as well. journald happily coexists with syslog daemons, as:

-   some syslog daemons can both read from and write to the journal
-   journald exposes the [syslog API](https://linux.die.net/man/3/syslog)

### journald benefits

Think of journald as your mini-command-line-[ELK](https://sematext.com/guides/elk-stack/) that lives on virtually every Linux box. It provides lots of features, most importantly:

-   **Indexing**. journald uses a binary storage for logs, where data is indexed. Lookups are much faster than with plain text files
-   **Structured logging**. Though [it's possible with syslog, too](https://sematext.com/blog/structured-logging-with-rsyslog-and-elasticsearch/), it's enforced here. Combined with indexing, it means you can easily filter specific logs (e.g. with a set priority, in a set timeframe)
-   **Access control**. By default, storage files are split by user, with different permissions to each. As a regular user, you won't see everything root sees, but you'll see your own logs
-   **Automatic log rotation**. You can configure journald (see below) to keep logs only up to a space limit, or based on free space

## Configuring journald

To tweak how journald behaves, you'll edit `/etc/systemd/journald.conf` and then reload the journal service like:

    systemctl reload systemd-journald.service

Though [earlier versions of journald need to be restarted](https://github.com/systemd/systemd/issues/2236):

    systemctl restart systemd-journald.service

Most important settings will be around storage: whether the journal should be kept in memory or on disk, when to remove old logs and how much to rate limit. We'll focus on some of those next, but you can see all the configuration options in [journald.conf's man page](https://www.freedesktop.org/software/systemd/man/journald.conf.html%23).

### journald storage

The **`Storage`** option controls whether the journal is stored in memory (under `/run/log/journal`) or on disk (under `/var/log/journal`). Setting **`Storage=volatile`** will store the journal in memory, while **`Storage=persistent`** will store it on disk. Most distributions have it set to auto, which means it will store the journal on disk if `/var/log/journal` exists, otherwise it will be stored in memory.

Once you've decided where to store the journal, you may want to set some limits. For example, **`SystemMaxUse=4G`** will limit `/var/log/journal` to about 4GB. Similarly, **`SystemKeepFree=10G`** will try to keep 10GB of disk space free. If you choose to keep the journal in memory, the equivalent options are **`RuntimeMaxUse`** and **`RuntimeKeepFree`**.

You can check the current disk usage of the journal with [journalctl](https://www.freedesktop.org/software/systemd/man/journalctl.html) via **`journalctl --disk-usage`**. If you need to, you can clean it up on demand via **`journalctl --vacuum-size=4GB`** (i.e. to reduce it to 4GB).

Compression is enabled by default on log entries larger than 512 bytes. If you want to change this threshold to, say 1KB, you'd add **`Compress=1K`**.

Also by default, journald will drop all log messages from a service if it passes certain limits. These limits can be configured via **`RateLimitBurst`** and **`RateLimitIntervalSec`**, which default to **`10000`** and **`30s`** respectively. Actual values will depend on the available free space. For example, if you have more than 64GB of free disk space, the multiplier will be 6. Meaning it will drop logs from a service after 60K messages sent in 30 seconds.

The rate limit defaults are sensible, unless you have a specific service that's generating lots of logs (e.g. a web server). In that case, it might be better to **`LogRateLimitBurst`** and **`LogRateLimitIntervalSec`** in that application's [service definition](https://www.freedesktop.org/software/systemd/man/systemd.exec.html).

## journald commands via journalctl

[journalctl](https://www.freedesktop.org/software/systemd/man/journalctl.html) is your main tool for interacting with the journal. If you just run it, you'll see:

-   all entries, from oldest to newest
-   paged by [less](https://linux.die.net/man/1/less)
-   lines go past the edge of your screen if they have to (use left and right arrow keys to navigate)
-   format is similar to the syslog output, as it is configured in most Linux distributions: **syslog timestamp + hostname + program and its PID + message**

Here's an example snippet:

    Apr 09 10:22:49 localhost.localdomain su[866]: pam_unix(su-l:session): session opened for user solr by (uid=0)<
    Apr 09 10:22:49 localhost.localdomain systemd[1]: Started Session c1 of user solr.<
    Apr 09 10:22:49 localhost.localdomain systemd[1]: Created slice User Slice of solr.<
    Apr 09 10:22:49 localhost.localdomain su[866]: (to solr) root on none

This is rarely what you want. More common scenarios are:

-   **last N lines** (equivalent of tail -n 20 - if N=20): `journalctl -n 20`
-   **follow** (tail -f equivalent): `journalctl -f`
-   page **from newest to oldest**: `journalctl --reverse`
-   **skip paging and just grep** for something (e.g. “solr”): `journalctl --no-pager | grep solr`

If you often find yourself using `--no-pager`, you can change the default pager through the `SYSTEMD_PAGER` variable. `export SYSTEMD_PAGER=cat` **will disable paging**. That said, you might want to look into journalctl's own options for displaying and filtering - described below - before using text processing tools.

### journalctl display settings

The main option here is `--output`, which can [take many values](https://www.freedesktop.org/software/systemd/man/journalctl.html). As an [ELK consultant](https://sematext.com/consulting/logging/), I want my timestamps [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601), and **`--output=short-iso`** will do just that. Now this is more like it:

    2020-04-09T10:23:01+0000 localhost.localdomain solr[860]: Started Solr server on port 8983 (pid=999). Happy searching!
    2020-04-09T10:23:01+0000 localhost.localdomain su[866]: pam_unix(su-l:session): session closed for user solr

journald keeps more information than what the **short/short-iso** output shows. Adding **`--output=json-pretty`** (or just **json** if you want it compact) can look like this for a single event:

    {
     "__CURSOR" : "s=83694dffb084461ea30a168e6cef1e6c;i=103f;b=f0bbba1703cb43229559a8fcb4cb08b9;m=c2c9508c;t=5a2d9c22f07ed;x=c5fe854a514cef39",
     "__REALTIME_TIMESTAMP" : "1586431033018349",
     "__MONOTONIC_TIMESTAMP" : "3267973260",
     "_BOOT_ID" : "f0bbba1703cb43229559a8fcb4cb08b9",
     "PRIORITY" : "6",
     "_UID" : "0",
     "_GID" : "0",
     "_MACHINE_ID" : "13e3a06d01d54447a683822d7e0b4dc9",
     "_HOSTNAME" : "localhost.localdomain",
     "SYSLOG_FACILITY" : "3",
     "SYSLOG_IDENTIFIER" : "systemd",
     "_TRANSPORT" : "journal",
     "_PID" : "1",
     "_COMM" : "systemd",
     "_EXE" : "/usr/lib/systemd/systemd",
     "_CAP_EFFECTIVE" : "1fffffffff",
     "_SYSTEMD_CGROUP" : "/",
     "CODE_FILE" : "src/core/job.c",
     "CODE_FUNCTION" : "job_log_status_message",
     "RESULT" : "done",
     "MESSAGE_ID" : "9d1aaa27d60140bd96365438aad20286",
     "_SELINUX_CONTEXT" : "system_u:system_r:init_t:s0",
     "UNIT" : "user-0.slice",
     "MESSAGE" : "Removed slice User Slice of root.",
     "CODE_LINE" : "781",
     "_CMDLINE" : "/usr/lib/systemd/systemd --switched-root --system --deserialize 22",
     "_SOURCE_REALTIME_TIMESTAMP" : "1586431033018103"
    }

This is where you can use structured logging to filter events. Next up, we'll look closer at the most important options for filtering.

### journald log filtering

You can filter by any field (see the JSON output above) by specifying **_key=value arguments_**, like:

    journalctl _SYSTEMD_UNIT=sshd.service

There are shortcuts, for example the **`_SYSTEMD_UNIT`** above can be expressed as **`-u`**. The above command is the equivalent of of:

    journalctl -u sshd.service

Other useful shortcuts:

-   **severity** (here called **priority**). **`journalctl -p warning`** will show logs with at least a severity of **`warning`**
-   show only kernel messages: **`journalctl --dmesg`**

You can also filter by time, of course. Here, you have multiple options:

-   **`--since/--until`** as a **full timestamp**. For example: **`journalctl --since="2020-04-09 11:30:00"`**
-   **date only** (00:00:00 is assumed as the time): **`journalctl --since=2020-04-09`**
-   **`abbreviations: journalctl --since=yesterday --until=now`**

In general, you have to specify the exact value you're looking for. With the exception of \_SYSTEMD\_UNIT. Here, patterns also work:

    journalctl -u sshd*

Newer versions of systemd also allow a **`--grep`** flag, which allows you to filter the `MESSAGE` field by regex. But you can always pipe the journalctl output through grep itself.

### journald and boots

Besides messages logged by applications, journald remembers significant events, such as system reboots. Here's an example:

    # journalctl MESSAGE="Server listening on 0.0.0.0 port 22."
    **-- Logs begin at Wed 2020-04-08 11:53:18 UTC, end at Thu 2020-04-09 12:01:01 UTC. --**
    Apr 08 11:53:23 localhost.localdomain sshd[822]: Server listening on 0.0.0.0 port 22.
    Apr 08 13:23:42 localhost.localdomain sshd[7425]: Server listening on 0.0.0.0 port 22.
    **-- Reboot --**
    Apr 09 10:22:49 localhost.localdomain sshd[857]: Server listening on 0.0.0.0 port 22.

You can suppress these special messages via **-q**. Use **-b** to show only messages after a certain boot. For example, to show messages since the last boot:

    # journalctl MESSAGE="Server listening on 0.0.0.0 port 22." -b
    -- Logs begin at Wed 2020-04-08 11:53:18 UTC, end at Thu 2020-04-09 12:01:01 UTC. --
    Apr 09 10:22:49 localhost.localdomain sshd[857]: Server listening on 0.0.0.0 port 22.

You can specify a boot as an offset to the current one (e.g. **-b -1** is the boot before the last). You can also specify a boot ID, but to do this you need to know what are the available boot IDs:

    # journalctl --list-boots
    -1 d26652f008ef4020b15a3d510bbcb381 Wed 2020-04-08 11:53:18 UTC—Wed 2020-04-08 14:31:16 UTC
     0 f0bbba1703cb43229559a8fcb4cb08b9 Thu 2020-04-09 10:22:43 UTC—Thu 2020-04-09 12:01:01 UTC

And then:

    # journalctl MESSAGE="Server listening on 0.0.0.0 port 22." -b d26652f008ef4020b15a3d510bbcb381
    -- Logs begin at Wed 2020-04-08 11:53:18 UTC, end at Thu 2020-04-09 12:01:01 UTC. --
    Apr 08 11:53:23 localhost.localdomain sshd[822]: Server listening on 0.0.0.0 port 22.
    Apr 08 13:23:42 localhost.localdomain sshd[7425]: Server listening on 0.0.0.0 port 22.

This is all useful if you configure journald for persistent storage (see the configuration section above).

## journald centralized logging

As you probably noticed, journald is quite host-centric. In practice, you'll want to access these logs in a central location, without having to SSH into each machine.

There are multiple ways of centralizing journald logs, and we'll detail each below:

-   **[systemd-journal-upload](https://www.freedesktop.org/software/systemd/man/systemd-journal-upload.html) uploads journal entries**. Either [directly to Sematext Cloud](https://sematext.com/docs/integration/journald-integration/) or to a log shipper that can read its output, such as the [open-source Logagent](https://sematext.com/docs/logagent/input-plugin-journald-upload/)
-   **[systemd-journal-remote](https://www.freedesktop.org/software/systemd/man/systemd-journal-remote.service.html%23) as a “centralizer”**. The idea is to have all journals on one host, so you can use journalctl to search (see above). This can work in “pull” or “push” mode
-   **a [syslog daemon](https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/) or [another log shipper](https://sematext.com/blog/logstash-alternatives/) reads from the local journal**. Then, it forwards logs to a central store like [ELK](https://sematext.com/guides/elk-stack/) or [Sematext Cloud](https://sematext.com/docs/logs/syslog/)
-   **journald forwards entries to a local syslog socket**. Then, a log shipper (typically a syslog daemon) picks messages up and forwards them to the central store

### systemd-journal-upload to ELK or Sematext Cloud

[systemd-journal-upload](https://www.freedesktop.org/software/systemd/man/systemd-journal-upload.html) is a service that pushes new journal entries over HTTP/HTTPS. That destination can be the [Sematext Cloud Journald Receiver](https://sematext.com/docs/integration/journald-integration/) \- the easiest way to centralize journald logs. And probably the best, as we'll discuss below.

Although it's part of journald/systemd, `systemd-journal-upload` isn't installed by default on most distros. So you have to add it via something like:

    apt-get install systemd-journal-remote

Then, uploading journal entries is as easy as:

    systemd-journal-upload --url=http://logsene-journald-receiver.sematext.com:80/YOUR_LOGS_TOKEN

Though most likely you'll want to configure it as a service:

    $ cat /etc/systemd/journal-upload.conf
    [Upload]
    URL=http://logsene-journald-receiver.sematext.com:80/YOUR_LOGS_TOKEN

If you need more control, or if you want to send journal entries to your local Elasticsearch, you can use the [open-source](https://github.com/sematext/logagent-js)[Logagent](https://github.com/sematext/logagent-js) with its [journald input plugin](https://sematext.com/docs/logagent/input-plugin-journald-upload/) as a journald centralizer: ![](https://sematext.com/wp-content/uploads/2020/04/jornald-logging-post-image1.png) Here's the relevant part of `logagent.conf`:

    input:
      journal-upload:
        module: input-journald-upload
        port: 9090
        worker: 0
        systemdUnitFilter:
          include: !!js/regexp /.*/i

**Using Logagent and Elasticsearch or Sematext Cloud** (i.e. we host Logagent and Elasticsearch for you) is probably **the best option to centralize journald logs**. That's because you get all journald's structured data over a reliable protocol (HTTP/HTTPS) with minimal overhead. The catch? Initial import is tricky, because it can generate a massive HTTP payload. For this, you might want to do the initial import by streaming journalctl output through [Logagent](https://github.com/sematext/logagent-js), like:

    journalctl --output=json --no-page | logagent --index SEMATEXT-LOGS-TOKEN

### systemd-journal-remote

Journald comes with its own “log centralizer”: [systemd-journal-remote](https://www.freedesktop.org/software/systemd/man/systemd-journal-remote.service.html%23). You don't get anywhere near the flexibility of ELK/Sematext Cloud, but it's already there and it might be enough for small environments.

`systemd-journal-remote` can either pull journals from remote systems or listen for journal entries on HTTP/HTTPS. The push model - where `systemd-journal-upload` is in charge of pushing logs - is typically better because:

-   it can continuously tail the journal and remembers where it left off (i.e. maintains a cursor)
-   you don't need to open access to the journal of every system

`systemd-journal-remote` typically comes in the same package as `systemd-journal-upload`. Once it's installed, you can make it listen to HTTP/HTTPS traffic:

    host2# systemd-journal-remote --listen-http=0.0.0.0:19352 --output=/var/log/journal/remote

Now you can push the journal of a remote host like this:

    host1# systemd-journal-upload --url=http://host2:19352

### systemd-journal-remote and systemd-journal-gatewayd

[s](https://www.freedesktop.org/software/systemd/man/systemd-journal-remote.service.html%23)[ystemd-journal-remote](https://www.freedesktop.org/software/systemd/man/systemd-journal-remote.service.html%23) can also pull journal entries from remote hosts. These hosts would normally serve their journal via [systemd-journal-gatewayd](https://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html%23) (which is often provided by the same package). Once you have systemd-journal-gatewayd, you can start it via:

    host1# systemctl start systemd-journal-gatewayd.socket

You can verify if it works like this:

    curl host1:19531/entries

Then, from the “central” host, you can use systemd-journal-remote to fetch journal entries:

    host2# systemd-journal-remote --url [http://](http://host1:19531)[host1](http://host1:19531)[:19531](http://host1:19531)

By default, systemd-journal-remote will write the imported journal to `/var/log/journal/remote/` (you might have to create it first!), so you can search them via `journalctl`:

    journalctl -D /var/log/journal/remote/

### Tools that read directly from the journal

Another approach for centralizing journald logs is to **have a [log shipper](https://sematext.com/blog/logstash-alternatives/) read from the journal**, much like journalctl does. Then, it can process logs and send them to destinations like Elasticsearch or Sematext Cloud (which exposes the [Elasticsearch API](https://sematext.com/docs/logs/index-events-via-elasticsearch-api/)).

For this approach, there's a PoC [journald input plugin for Logstash](https://github.com/logstash-plugins/logstash-input-journald). As you probably know, [Logstash is easy to use](https://sematext.com/blog/getting-started-with-logstash/), so reading from the journal is as easy as:

    input {
      journald {
      # you may add other options here, but of course the defaults are sensible :)
      }
    }

[Journalbeat](https://www.elastic.co/guide/en/beats/journalbeat/master/journalbeat-overview.html) is also available. It's as easy to install and use as [Filebeat](https://sematext.com/blog/using-filebeat-to-send-elasticsearch-logs-to-logsene/), except that it reads from the journal. But it's marked as experimental.

Why PoC and experimental? Because of potential journal corruption which might lead to nasty results. Check the comments in [rsyslog's journal input documentation](https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html) for details.

[Syslog daemons](https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/) are also log shippers. Some of them can also read from the journal, or even write to it. There's a lot to say about syslog and the journal, so we'll dissect the topic in a section of its own.

## journald vs syslog

**Journald provides a good out-of-the-box logging experience** for systemd. The trade-off is, journald is **a bit of a monolith**, having everything from log storage and rotation, to log transport and search. Some would argue that **syslog is more UNIX-y**: more lenient, easier to integrate with other tools. Which was its main criticism to begin with.

Flame wars aside, there's good integration between the two. Journald provides a [syslog API](https://manpages.debian.org/jessie/manpages-dev/syslog.3.en.html) and can forward to syslog (see below). On the other hand, syslog daemons have journal integrations. For example, [rsyslog](https://www.rsyslog.com/) provides plugins to both [read from journald](https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html) and [write to journald](https://rsyslog.readthedocs.io/en/latest/configuration/modules/omjournal.html). In fact, they recommend two architectures:

-   A small setup (e.g. N embedded devices and one server) could work by centralizing journald logs (see above). If embedded devices don't have systemd/journald but have syslog, they can centralize via syslog to the server and finally write to the server's journal. This journal will act like a mini-ELK
-   A larger setup can work by aggregating journal entries through a syslog daemon. We'll concentrate on this scenario in the rest of this section

There are two ways of centralizing journal entries via syslog:

1.  **syslog daemon acts as a journald client** (like journalctl or Logstash or Journalbeat)
2.  **journald forwards messages to syslog** (via socket)

Option 1) is slower - reading from the journal is slower than reading from the socket - but captures all the fields from the journal. Option 2) is safer (e.g. no issues with journal corruption), but the journal will only forward traditional syslog fields (like severity, hostname, message..). Typically, you'd go for 2) unless you need the structured info. Here's an example configuration for implementing 1) with rsyslog, and writing all messages to Elasticsearch or [Sematext Cloud](https://sematext.com/cloud/):

    # module that reads from journal
    module(load="imjournal"
     StateFile="/var/run/journal.state" # we write here where we left off
     PersistStateInterval="100" # update the state file every 100 messages
    )
    # journal entries are read as JSON, we'll need this to parse them
    module(load="mmjsonparse")
    # Elasticsearch or Sematext Cloud HTTP output
    module(load="omelasticsearch")
    
    # this is done on every message (i.e. parses the JSON)
    action(type="mmjsonparse")
    
    # output template that simply writes the parsed JSON
    template(name="all-json" type="list"){
     property(name="$!all-json")
    }
    
    action(type="omelasticsearch"
     template="all-json" # use the template defined earlier
     searchIndex="SEMATEXT-LOGS-APP-TOKEN-GOES-HERE"
     server="logsene-receiver.sematext.com"
     serverport="80"
     bulkmode="on" # use the bulk API
     action.resumeretrycount="-1" # retry indefinitely if Logsene/Elasticsearch is unreachable
    )

For option 2), we'll need to configure journald to forward to a socket. It's as easy as adding this to /etc/systemd/journald.conf:

    ForwardToSyslog=yes

And it will write messages, in syslog format, to /run/systemd/journal/syslog. On the rsyslog side, you'll have to configure its [socket input module](https://rsyslog-doc.readthedocs.io/en/latest/configuration/modules/imuxsock.html) to listen to that socket. Here's a similar example of sending logs to Elasticsearch or Sematext Cloud:

    module(load="imuxsock"
     SysSock.Name="/run/systemd/journal/syslog")
    
    # template to write traditional syslog fields as JSON
    template(name="plain-syslog"
     type="list") {
     constant(value="{")
     constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
     constant(value="\",\"host\":\"") property(name="hostname")
     constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
     constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
     constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
     constant(value="\",\"message\":\"") property(name="msg" format="json")
     constant(value="\"}")
    }
    
    action(type="omelasticsearch"
     template="plain-syslog" # use the template defined earlier
     searchIndex="SEMATEXT-LOGS-APP-TOKEN-GOES-HERE"
     server="logsene-receiver.sematext.com"
     serverport="80"
     bulkmode="on" # use the bulk API
     action.resumeretrycount="-1" # retry indefinitely if Logsene/Elasticsearch is unreachable
    )

Whether you read the journal through syslog, systemd-journal-upload or through a log shipper, all the above methods assume that you're dealing with Linux running on bare metal or VMs. But what if you're using containers? Let's explore your options in the next section.

## journald and containers

In this context, I think it's worth making a distinction between Docker containers and systemd containers. Let's take them one at a time.

### journald and Docker

Typically, a Docker container won't have systemd, because it would make it too “heavy”. As a consequence, it won't have journald, either. That said, you probably have journald on the host, if the host is running Linux. This means you can use the [journald logging driver](https://docs.docker.com/config/containers/logging/journald/) to send all the logs of a host's containers to that host's journal. It's as easy as:

    docker run my_container --log-driver=journald

And that container's logs will be in the journal:

    # journalctl CONTAINER_NAME=my_container --all
    Apr 09 13:03:28 localhost.localdomain dockerd-current[25558]: hello journal

If you want to use journald by default, you can make the change in daemon.json and restart Docker:

    # cat /etc/docker/daemon.json
    {
     "log-driver": "journald"
    }
    systemctl restart docker

If you have more than one host, you're back to the centralizing problem that we explored in the previous section: getting all journals in one place. This makes journald an intermediate step that may not be necessary.

A better approach is to [centralize container logs](https://sematext.com/docs/logs/sending-docker-logs/) via Logagent, which can run as a container. Here, Logagent picks up logs and forwards them to a central place, like Elasticsearch or Sematext Cloud. But it's not the only way. In fact, we explore different approaches, with their pros and cons, in our [Complete Guide to Docker logging](https://sematext.com/guides/docker-logs/).

### journald and systemd containers

systemd provides containers as well (called [machines](https://www.freedesktop.org/software/systemd/man/systemd-machined.service.html%23)) via [systemd-nspawn](https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html%23). Unlike Docker containers, systemd-nspawn machines can log to the journal directly. You can read the logs of a specific machine like this:

    journalctl --machine $MACHINE_NAME

Where `$MACHINE_NAME` is one of the running machines. You'd use **machinectl list** to see all of them.

As with Docker's journald logging driver, this setup might be challenging when you have multiple hosts. You'll either want to centralize your journals - as described in the previous section. Or, you can send logs from your systemd containers directly to the central location - either via a [log shipper or a logging library](https://sematext.com/blog/logging-libraries-vs-log-shippers/).

## Conclusions

Did you read all the way to the end? You're a hero! And you probably figured that journald is good for structured logging, quick local searches, and tight integration with systemd. Its design shows its weaknesses when it comes to centralizing log events. Here we have many options, but none is perfect. That said, [Logagent's journald input](https://sematext.com/docs/logagent/input-plugin-journald-upload/) and [Sematext Cloud's journald receiver](https://sematext.com/docs/integration/journald-integration/) (the hosted equivalent) come pretty close.