Linux: processes core dumps, systemd-coredump and Debian
Tags: linux, tutorial, todayilearned
Published: 2020-03-14 11:04:16

---
title: Linux: processes core dumps, systemd-coredump and Debian
published: true
date: 2020-03-14 10:52:43 UTC
tags: Linux,tutorial,todayilearned
canonical_url: https://rtfm.co.ua/en/linux-processes-core-dumps-systemd-coredump-and-debian/
---

![](https://rtfm.co.ua/wp-content/uploads/2011/08/linux-logo.png)Need to get a dump from fro ma PHP process on Debian 9.

In this post will take a Linux kernel mechanism to create and manage processes dumps.

Kernel’s dumps are created in another way, check [Kdump](https://wiki.archlinux.org/index.php/Kdump) на Arch Wiki.

### Linux Core Dump

The kernel will create a process dump if it performed an invalid operation and has to be stopped.

To do so, the kernel will send a special signal to such a process so the process can handle it itself or using standard mechanisms and will cause the kernel to apply the mechanism to create a dump of the memory of this process.

The complete list can be found in the kernel source in the [`SIG_KERNEL_COREDUMP_MASK`](https://elixir.bootlin.com/linux/v5.5.8/source/include/linux/signal.h#L413) macros:

```
...
#define SIG_KERNEL_COREDUMP_MASK (\
        rt_sigmask(SIGQUIT)   |  rt_sigmask(SIGILL)    | \
  rt_sigmask(SIGTRAP)   |  rt_sigmask(SIGABRT)   | \
        rt_sigmask(SIGFPE)    |  rt_sigmask(SIGSEGV)   | \
  rt_sigmask(SIGBUS)    |  rt_sigmask(SIGSYS)    | \
        rt_sigmask(SIGXCPU)   |  rt_sigmask(SIGXFSZ)   | \
  SIGEMT_MASK  
...
```

Which is used by the [`sig_kernel_coredump`](https://elixir.bootlin.com/linux/v5.5.8/source/include/linux/signal.h#L433) macros:

```
...
#define sig_kernel_coredump(sig)  siginmask(sig, SIG_KERNEL_COREDUMP_MASK)
...
```

Which is called in case of Fatal-errors and in its turn will call the [`do_coredump()`](https://elixir.bootlin.com/linux/v5.5.8/source/kernel/signal.c#L2716) function:

```
...
  fatal:
                ...
    if (sig_kernel_coredump(signr)) {
      if (print_fatal_signals)
        print_fatal_signal(ksig->info.si_signo);
      proc_coredump_connector(current);
                        ...
      do_coredump(&ksig->info);
    }
...
```

And [`do_coredump()`](https://elixir.bootlin.com/linux/v5.5.8/source/fs/coredump.c#L565) will create a memory dump and will save it on a hard disk.

#### Signals and dump creation

Let’s check how it’s working.

Take a simple code in C:

```
#include <stdio.h>
#include <unistd.h>
int main() {
    while(1) {
        pid_t pid = getpid();
        printf ("Working with PID %lu\n", pid);
        sleep(5);
    }
}
```

Build and run:

```
$ gcc make_dump.c -o make_dump
./make_dump
Working with PID 2714790
```

In another terminal – send a signal, for example [`SIGSEGV`](https://ru.wikipedia.org/wiki/SIGSEGV) (_Segmentation violation_), code 11:

```
$ kill -s SIGSEGV 2714790
```

In a terminal with the app running check its output:

```
$ ./make_dump
Working with PID 2714790
...
Working with PID 2714790
Segmentation fault (core dumped)
```

Check a dump file:

```
$ ls -l /tmp/ | grep 2714790
-rw------- 1 setevoy setevoy  380928 Mar 10 11:24 coredump-make_dump.2714790
```

In the same way, you can create a dump for any already running process, for instances – run `sleep`:

```
$ sleep 100 &
[1] 2761144

$ kill -s SIGSEGV 2761144
[1]+  Segmentation fault      (core dumped) sleep 100

$ file /tmp/coredump-sleep.2761144
/tmp/coredump-sleep.2761144: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'sleep 100', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid: 1000, execfn: '/usr/bin/sleep', platform: 'x86_64'
```

#### GDB – create a core dump

Besides of a signal sending you can use `gcore` from the [`gdb`](https://rtfm.co.ua/c-otladka-s-gdb-primery/):

```
$ sleep 100 &
[1] 2762961

$ sudo gcore 2762961
...
Saved corefile core.2762961

$ file core.2762961
core.2762961: ELF 64-bit LSB core file
```

#### GDB – read a core dump

To check a dump’s content you can use `gdb` and pass an executable as a first argument and a path to a dump file as a second argument:

```
$ gdb make_dump coredump-make_dump.2714790
...
[New LWP 2714790]
Core was generated by `./make_dump'.
Program terminated with signal SIGSEGV, Segmentation fault.
0  0x00007f50d566e27e in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6
(gdb)
```

#### `kernel.core_pattern`

During dump creation kernel will check the `kernel.core_pattern` parameter which determines how the dump will be handled.

Here you can specify a path and a filename with specificators or pass it external dumps handlers like [`systemd-coredump`](https://www.freedesktop.org/software/systemd/man/systemd-coredump.html) (рассмотрим ниже).

Check the _Naming of core dump files_ documentation [here>>>](http://man7.org/linux/man-pages/man5/core.5.html).

Most used options here are:

- `%e` executable filename (without path prefix)
- `%p` PID of dumped process, as seen in the PID namespace in which the process resides
- `%t` time of dump, expressed as seconds since the Epoch, 1970-01-01 00:00:00 +0000 (UTC)

Our file _/tmp/coredump-make_dump.2714790_, which we observed in the GDB above consists of the `kernel.core_pattern = /tmp/coredump-%e.%p`:

1. `/tmp` catalog
2. `%e` – file name started with the _coredump_ + an executable name файла _make_dump_
3. `%p` – and a killed process PID – _2714790_

Also, instead of using the direct path and file name you can pass dump data via a pipe `|` to the `/dev/null` or to a handler like `systemd-coredump`.

#### `limits.conf`

Before creating a dump the kernel also will check _soft_ and _hard_ limits for the `core` in the `/etc/security/limits.conf`:

```
$ cat /etc/security/limits.conf | grep core
- core - limits the core file size (KB)
*               soft    core            0
```

Or check directly in your current environment:

```
$ ulimit -c
unlimited
```

A running process limits can be found in its limits from the `/proc`:

```
$ ./make_dump
Working with PID 2753034
```

And limits:

```
$ cat /proc/2753034/limits
Limit                     Soft Limit           Hard Limit           Units
...
Max core file size        unlimited            unlimited            bytes
...
```

#### `fs.suid_dumpable`

Sometimes a dump can be not created if a process made a suid-operation for example by using the [`setuid()`](http://man7.org/linux/man-pages/man2/seteuid.2.html) syscall.

In this case, the behavior is controlled by the `fs.suid_dumpable`:

```
$ sysctl fs.suid_dumpable
fs.suid_dumpable = 2
```

It can accept 0, 1 or 2, see the [man proc](http://man7.org/linux/man-pages/man5/proc.5.html):

- 0: (_default_) This provides the traditional (pre-Linux 2.6.13) behaviour. A core dump will NOT be produced for a process which has changed credentials (by calling seteuid or similar) or whose binary does not have read permission enabled.
- 1: (“_debug_“) All processes dump core when possible.
- 2: (“_suidsafe_“) Any binary which normally would not be dumped (see “0” above) is dumped readable by root only.

### `systemd-coredump`

The [`systemd`](https://www.freedesktop.org/software/systemd/man/systemd-coredump.html) of course, has its own dumps handler – already mentioned [`systemd-coredump`](https://www.freedesktop.org/software/systemd/man/systemd-coredump.html).

On Arch Linux is used by default, for Debian 9 can be installed with `apt`:

```
$ sudo apt -y install systemd-coredump
```

Config file – `/etc/systemd/coredump.conf`.

After installation configure the kernel – send dumps via a pipe to the `systemd-coredump`:

```
root@bttrm-production-app-1:/home/admin#  echo '|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %e' > /proc/sys/kernel/core_pattern
```

Check it:

```
root@bttrm-production-app-1:/home/admin# cat /proc/sys/kernel/core_pattern
|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %e
```

Create some dump:

```
root@bttrm-production-app-1:/home/admin# sleep 10 &
[1] 27117

root@bttrm-production-app-1:/home/admin# kill -s 11 27117
[1]+  Segmentation fault      (core dumped) sleep 10
```

#### `coredumpctl`

To work with dumps managed with `systemd-coredump` the `coredumpctl` utility can be used:

```
root@bttrm-production-app-1:/home/admin# coredumpctl
TIME                            PID   UID   GID SIG COREFILE EXE
Mon 2020-03-09 20:10:16 EET   20882     0     0  11 present  /home/admin/dump_test
...
Tue 2020-03-10 09:04:14 EET   16786  1003  1003  11 present  /usr/sbin/php-fpm7.4
Tue 2020-03-10 12:23:43 EET   27117     0     0  11 present  /bin/sleep
```

In the _SIG_ column, we can see a signal sent to the process killed, in those cases it were _11_ == `SIGSEGV`.

Dump files are located at the `/var/lib/systemd/coredump` directory:

```
root@bttrm-production-app-1:/home/admin# ll /var/lib/systemd/coredump/
total 125376
-rw-r-----  1 root root    25208 Mar  9 20:10 core.dump_test.0.6bb23c691d354e9dbf4382d109a5c1d4.20882.1583777416000000000000.lz4
...
-rw-r-----  1 root root    33962 Mar 10 12:23 core.sleep.0.6bb23c691d354e9dbf4382d109a5c1d4.27117.1583835823000000000000.lz4
```

To observe a dump’s content use info with some MATCH condition, for example, a PID:

```
root@bttrm-production-app-1:/home/admin# coredumpctl info 27117
PID: 27117 (sleep)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Tue 2020-03-10 12:23:43 EET (3min 3s ago)
Command Line: sleep 10
Executable: /bin/sleep
...
Hostname: bttrm-production-app-1
Storage: /var/lib/systemd/coredump/core.sleep.0.6bb23c691d354e9dbf4382d109a5c1d4.27117.1583835823000000000000.lz4
Message: Process 27117 (sleep) of user 0 dumped core.
Stack trace of thread 27117:
0  0x00007fc1a8053270 __nanosleep (libc.so.6)
1  0x000056159fa6f91f n/a (sleep)
2  0x000056159fa6f700 n/a (sleep)
3  0x000056159fa6c9a4 n/a (sleep)
4  0x00007fc1a7fbb2e1 __libc_start_main (libc.so.6)
5  0x000056159fa6ca7a n/a (sleep)
```

### Debian – core dump is not created

Everything described above is working perfectly on two laptops with Arch Linux, but on my project’s servers with Debian 9 dumps are created in a usual way, without  `systemd-coredump`.

The `core_pattern` config:

```
root@bttrm-production-app-1:/home/admin# cat /proc/sys/kernel/core_pattern
/tmp/core-%e.%p
```

Run a process, kill it with 11 signal, check `/tmp`  – no dumps here.

Run an app again:

```
root@bttrm-production-app-1:/home/admin# ./dump_test
Working with PID 27954
```

And check its limits:

```
root@bttrm-production-app-1:/home/admin# cat /proc/27954/limits
Limit                     Soft Limit           Hard Limit           Units
...
Max core file size        0                    unlimited            bytes
...
```

The _Soft_ limit is set to the zero value, e.g. dumps are disabled at all as kernel can’t use more than 0 bytes for a dump file.

Set it to the _unlimited_:

```
root@bttrm-production-app-1:/home/admin# ulimit -S -c unlimited
```

Run an app again, check limits now:

```
root@bttrm-production-app-1:/home/admin# ./dump_test 1>/dev/null &
[3] 28399

root@bttrm-production-app-1:/home/admin# cat /proc/$!/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        unlimited            unlimited            bytes
...
```

Create a dump:

```
root@bttrm-production-app-1:/home/admin# sleep 10 &
[4] 28613

root@bttrm-production-app-1:/home/admin# kill -s 11 $!
```

Check it:

```
root@bttrm-production-app-1:/home/admin# ls -l /tmp/coredump-sleep.28613
-rw------- 1 root root 380928 Mar 10 12:47 /tmp/coredump-sleep.28613
```

Done.

### Useful links

- [Understand and configure core dumps on Linux](https://linux-audit.com/understand-and-configure-core-dumps-work-on-linux/)
- [Hunting the Core](https://www.fromdual.com/hunting-the-core)
- [Core dump](https://wiki.archlinux.org/index.php/Core_dump) (Arch Wiki)
- [Generate PHP core dumps on segfaults in PHP-FPM](https://ma.ttias.be/generate-php-core-dumps-segfaults-php-fpm/)
- [Generating a gdb backtrace](https://bugs.php.net/bugs-generating-backtrace.php)
- [Проблема с eaccelerator и генерация coredump для php-fpm](https://blog.amet13.name/2015/05/eaccelerator-coredump-php-fpm.html)
- [How to get a core dump for a segfault on Linux](https://jvns.ca/blog/2018/04/28/debugging-a-segfault-on-linux/)
- [kill: Creating a core dump](https://bencane.com/2011/09/22/kill-creating-a-core-dump/)

### Similar posts

- <small>01/23/2020</small> [Debian: php7.3-curl : Depends: libcurl3 (>= 7.44.0) but it is not installable](https://rtfm.co.ua/en/debian-php7-3-curl-depends-libcurl3-7-44-0-but-it-is-not-installable-2/) 
- <small>01/27/2017</small> [Debian: apt – GPG error: The following 
- <small>02/24/2020</small> [Linux: PHP-FPM, Docker, STDOUT and STDERR – no an application’s error logs](https://rtfm.co.ua/en/linux-php-fpm-docker-stdout-and-stderr-no-an-applications-error-logs/)