Dockercontainer won’t start - Getting the final child’s pid from pipe caused “EOF”
Tags: linux, docker, strato, beancounter
Published: 2022-07-18 12:00:35

---
title: Dockercontainer won’t start - Getting the final child’s pid from pipe caused “EOF”
published: true
date: 2022-07-03 20:00:00 UTC
tags: linux,docker,strato,beancounter
canonical_url: https://www.zufallsheld.de/2022/07/03/docker-container-wont-start-getting-the-final-child-pid-from-pipe-caused-eof/
---

Randomly some Docker-containers on a clients Linux machine wouldn’t start - they’d fail with the error:

<!-- PELICAN_END_SUMMARY -->
```
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused \"EOF\"": unknown.

```

It didn’t matter what container it was, nor where or who started it. Sometimes it happened with Gitlab-CI jobs, sometimes with manually run containers. It also didn’t happen everytime. Sometimes it just worked, other times it didn’t work for hours and I’d ignore it.

The last time it happened I googled the error again and again found [this huge open issue](https://github.com/moby/moby/issues/40835) on Docker’s Github issue-tracker. This time I read every response in the thread and came upon this answer:

> Okay, just found another interesting post on other forum. If you are running an vps which is virtualized with Virtuozzo you hosting provider maybe locked your tasks…
> 
> Im using strato and it seems to be that they have limited my server. Under /proc/user\_beancounters you can find those settings. The numprocs is set to 700 and my actual held is 661. Starting an bigger docker stock seems to be impossible…
> 
> You can find more in this post https://serverfault.com/questions/1017994/docker-compose-oci-runtime-create-failed-pthread-create-failed/1018402
> 
> It seems to be there is no bug…

This sounded like my kind of problem as my client is using a Strato server, too!

Now my native language isn’t English, however I remembered the term “beancounter” from the [BOFH](https://www.theregister.com/offbeat/bofh/) and I really do hope that this feature originates somewhat from there (though I know that the term is older than the BOFH).

Looking at the servers beancounter gives me the following output:

```
root@h2939459:~# cat /proc/user_beancounters 
Version: 2.5
    uid resource held maxheld barrier limit failcnt
2939459: kmemsize 353345536 1649811456 9223372036854775807 9223372036854775807 0
            lockedpages 0 32 9223372036854775807 9223372036854775807 0
            privvmpages 5861268 6833651 9223372036854775807 9223372036854775807 0
            shmpages 2222328 2255102 9223372036854775807 9223372036854775807 0
            dummy 0 0 9223372036854775807 9223372036854775807 0
            numproc 541 541 1100 1100 0
            physpages 1461605 8319910 8388608 8388608 0
            vmguarpages 0 0 9223372036854775807 9223372036854775807 0
            oomguarpages 1525697 8388608 0 0 0
            numtcpsock 0 0 9223372036854775807 9223372036854775807 0
            numflock 0 0 9223372036854775807 9223372036854775807 0
            numpty 1 2 9223372036854775807 9223372036854775807 0
            numsiginfo 16 147 9223372036854775807 9223372036854775807 0
            tcpsndbuf 0 0 9223372036854775807 9223372036854775807 0
            tcprcvbuf 0 0 9223372036854775807 9223372036854775807 0
            othersockbuf 0 0 9223372036854775807 9223372036854775807 0
            dgramrcvbuf 0 0 9223372036854775807 9223372036854775807 0
            numothersock 0 0 9223372036854775807 9223372036854775807 0
            dcachesize 264458240 1497632768 9223372036854775807 9223372036854775807 0
            numfile 4284 7930 9223372036854775807 9223372036854775807 0
            dummy 0 0 9223372036854775807 9223372036854775807 0
            dummy 0 0 9223372036854775807 9223372036854775807 0
            dummy 0 0 9223372036854775807 9223372036854775807 0
            numiptent 1997 2000 2000 2000 74

```

Take a look at the last line:

```
uid resource held maxheld barrier limit failcnt
     numiptent 1997 2000 2000 2000 74

```

This shows that the resource “numiptent” is currently using 1997 “units”, where 2000 can be held totally. “failcnt” shows the count of refused allocations. When I run another container, this count increased so it must be the problem!

A quick search revealed that “numiptent” are the number of NETFILTER (IP packet filtering) entries, or in simpler terms: the number of iptables rules.

I immediatly knew the reason for this: the client is using [fail2ban](https://www.fail2ban.org/wiki/index.php/Main_Page) which blocks IP-addresses that try to bruteforce ssh-logins on the server. Looking at the fail2ban-overview I noticed that 1900 IP-addresses where blocked, conspicuously close to the beancounter-limit of 2000!

Restarting fail2ban threw out all these IP-addresses which decreased the beancounter on “numiptent”.

Reason found! Now why Strato decided to limit the number of iptables-entries and what to do against this (apart from disabling fail2ban) - I don’t know yet.