Repository: thanos-io/thanos

Issue #2486: Web route prefix is not applied on debug endpoints.
Created: 2020-04-21T03:04:49Z
In Prometheus, if the web external path is set, then all Prometheus routes will be updated. There is a good example here https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
You should be aware that with this external URL, the /prometheus/ path prefix will be required for all HTTP access to Prometheus. 

The /metrics will be on http://localhost:9090/prometheus/metrics for example.

However, in thanos, if web route prefix is set to /thanos, only /api/v1 endpoints will be updated to /thanos/api/v1. But other endpoints like /metrics, still serve on /metrics, we should add web route prefix to these endpoints as well.
AC:

Endpoints like /metrics or /-/ready can work with web route prefix just like Prometheus.

Duplicate issue #2477: Thanos rule metrics missing after upgrade to 0.12.0.
Created: 2020-04-20T15:48:17Z
Thanos, Prometheus and Golang version used:
# thanos --version            
thanos, version 0.12.0 (branch: HEAD, revision: 69f9c4e604d20b2e640f790957414cf3d19ecafa)
  build user:       root@aab79ed2e8d5
  build date:       20200415-09:34:28
  go version:       go1.13.6


Object Storage Provider:
irrelevant
What happened:
Thanos rule metrics endpoint is not reachable after Thanos upgrade to 0.12.0
# ps aux | grep "thanos rule"
prometh+   1149  0.0  2.4 220684 94420 ?        Ssl  Apr15   3:26 /usr/bin/thanos rule --data-dir=/var/lib/thanos/rule --http-address=0.0.0.0:19902 --grpc-address=0.0.0.0:19901 --rule-file=/etc/prometheus/thanos_alerts.yml --query=localhost:19192 --alertmanagers.url=https://alerts.example.com --alert.query-url=https://metrics.example.com/all --web.external-prefix=https://metrics.example.com/thanos/query

# netstat -tulpn | grep thanos
tcp6       0      0 :::19192                :::*                    LISTEN      1151/thanos         
tcp6       0      0 :::19901                :::*                    LISTEN      1149/thanos         
tcp6       0      0 :::19902                :::*                    LISTEN      1149/thanos         
tcp6       0      0 :::19092                :::*                    LISTEN      1151/thanos    

# curl localhost:19902/       
<a href="https:/metrics.example.com/thanos/query/alerts">Found</a>.

# curl --max-time 10 localhost:19902/metrics
curl: (28) Operation timed out after 10001 milliseconds with 0 out of -1 bytes received

What you expected to happen:
Metrics to be returned
How to reproduce it (as minimally and precisely as possible):
See above
Full logs to relevant components:
Nothing in logs (with info log level), I can try debug logging if necessary.

Anything else we need to know:
Previously system was running Thanos 0.11.0 and metrics were working fine. The same system has Thanos query running and metrics for that service are returned just fine.
Environment:
# cat /etc/*release
CentOS Linux release 7.7.1908 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.7.1908 (Core)
CentOS Linux release 7.7.1908 (Core)
# uname -a
Linux hostname 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Closing comment (2020-04-21T06:34:19Z): Thanks for report and details!
Looks like a duplicate of #2486