Repository: nextdns/nextdns

Issue #196: Randomly hang once a day
Created: 2020-05-04T22:08:47Z
The daemon seems to randomly hang about once a day. The status command shows it as running and the logs show no errors before it fails. Restarting fixes it, but the daemon goes down causing a full internet outage each time. Is there a way to gather additional logs from the daemon?
Context

Version: 1.5.4
Platform EdgeOS on ER-4

Duplicate issue #226: CLI not failing over when there is an issue with upstream server
Created: 2020-05-19T08:31:49Z
There were service disruptions today on 2 nextdns edge routers that I was able to witness today.
The following 2 ip's were failing multiple times between 7:00 and 9:00 UTC:
https://zepto-ams-1.edge.nextdns.io#78.142.193.36
https://dns1.nextdns.io#45.90.28.0,2a07:a8c0::
https://zepto-ams-2.edge.nextdns.io#93.189.61.195

It looks like Nextdns client does not notice the failure from upstream within a sensible amount of time.
some logs:
Tue May 19 07:15:37 2020 daemon.notice nextdns[1741]: Setting up router
Tue May 19 07:15:37 2020 daemon.notice nextdns[1741]: Activating
Tue May 19 07:21:19 2020 daemon.notice nextdns[1741]: Connected 78.142.193.36:443 (con=4ms tls=21ms, TLS13)
Tue May 19 07:25:23 2020 daemon.notice nextdns[1741]: Received signal: terminated
Tue May 19 07:25:23 2020 daemon.notice nextdns[1741]: Stopping NextDNS 1.5.8/linux
Tue May 19 07:25:23 2020 daemon.notice nextdns[1741]: Restore router settings
Tue May 19 07:25:26 2020 daemon.notice nextdns[1741]: Deactivating
Tue May 19 07:25:26 2020 daemon.notice nextdns[1741]: NextDNS 1.5.8/linux stopped
Tue May 19 07:25:31 2020 daemon.notice nextdns[3100]: Starting NextDNS 1.5.8/linux on :53
Tue May 19 07:25:31 2020 daemon.notice nextdns[3100]: Listening on TCP/:53
Tue May 19 07:25:31 2020 daemon.notice nextdns[3100]: Starting discovery resolver
Tue May 19 07:25:31 2020 daemon.notice nextdns[3100]: Listening on UDP/:53
Tue May 19 07:25:33 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=6ms tls=110ms, TLS13)
Tue May 19 07:25:33 2020 daemon.warn nextdns[3100]: Endpoint failed: https://zepto-ams-1.edge.nextdns.io#78.142.193.36: roundtrip: EOF
Tue May 19 07:25:33 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=6ms tls=24ms, TLS13)
Tue May 19 07:25:33 2020 daemon.notice nextdns[3100]: Switching endpoint: https://dns1.nextdns.io#45.90.28.0,2a07:a8c0::
Tue May 19 07:25:36 2020 daemon.notice nextdns[3100]: Setting up router
Tue May 19 07:25:36 2020 daemon.notice nextdns[3100]: Activating
Tue May 19 07:29:54 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=0ms tls=0ms, )
Tue May 19 07:31:14 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=0ms tls=24ms, TLS13)
Tue May 19 07:38:26 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=0ms tls=0ms, )
Tue May 19 07:39:14 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=0ms tls=0ms, )
Tue May 19 07:53:23 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=0ms tls=0ms, )
Tue May 19 07:54:16 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=6ms tls=24ms, TLS13)
Tue May 19 07:55:52 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=6ms tls=23ms, TLS13)
Tue May 19 08:00:11 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=0ms tls=0ms, )
Tue May 19 08:01:14 2020 daemon.notice nextdns[3100]: Connected 45.90.28.0:443 (con=7ms tls=25ms, TLS13)

This is what trying to lookup a dns looks like at the moment of an "outage".
root@OpenWrt:~# date
Tue May 19 08:20:56 UTC 2020
root@OpenWrt:~# nslookup nu.nl
;; connection timed out; no servers could be reached

root@OpenWrt:~# nslookup nu.nl
;; connection timed out; no servers could be reached

root@OpenWrt:~# nslookup nu.nl
;; connection timed out; no servers could be reached

root@OpenWrt:~# date
Tue May 19 08:23:08 UTC 2020
root@OpenWrt:~# nextdns status
running
root@OpenWrt:~# curl -v https://45.90.28.0
* ssl_handshake returned - mbedTLS: (-0x7280) SSL - The connection indicated an EOF
curl: (35) ssl_handshake returned - mbedTLS: (-0x7280) SSL - The connection indicated an EOF

Context

Version [e.g. 1.5.2]: 1.5.8
Platform [e.g. macOS, ASUS-Merlin]: OpenWRT
Config:

root@OpenWrt:~# nextdns config
cache-max-age 0s
detect-captive-portals false
hardened-privacy false
bogus-priv true
use-hosts true
auto-activate true
config xxxxxx
log-queries false
cache-size 10MB
timeout 5s
listen 127.0.0.1:5342
max-ttl 0s
report-client-info true
setup-router true

Edit 1: adding https://zepto-ams-2.edge.nextdns.io#93.189.61.195 to the services that went down causing the same behavior described above.

Closing comment (2020-05-22T01:18:51Z): Certainly a dup of #196