Repository: thanos-io/thanos

Issue #844: compactor: File name is too long error
Created: 2019-02-14T19:49:26Z
Thanos, Prometheus and Golang version used
-> prometheus instances run in kubernetes with thanos 0.3 side car and image tags used 2.7.1.
-> % thanos --version
type: S3
thanos, version 0.3.0 (branch: HEAD, revision: 837e967)
build user:       root@986454de7a63
build date:       20190208-15:24:36
go version:       go1.11.5

What happened
Running compact command fails against a known bucket containing our metrics data. The data itself is currently not huge, may be around 3GB so far. Thanos sidecar keep pushing the data in to our local s3 compatible storage and we were hoping to keep this data at lesser granularity for several years. Here is the command I used:

% thanos compact --log.level=info --data-dir=/tmp/ --objstore.config-file=thanos-s3-bucket.conf --retention.resolution-raw=0d --retention.resolution-5m=180d --retention.resolution-1h=1000d --wait

What you expected to happen
I was hoping the compact command will process the data :)
How to reproduce it (as minimally and precisely as possible):
Our data is collected by prometheus from a bunch of netdata agents but with limited metrics we have identified. This gets scraped every 15s at the moment and all targets are CentOS 6/7. I feel like that may not be as relevant, as this looks like something else while creating a directory locally.
Full logs to relevant components
level=info ts=2019-02-14T19:39:03.453025Z caller=factory.go:39 msg="loading bucket configuration"
level=info ts=2019-02-14T19:39:03.453635Z caller=compact.go:196 msg="retention policy of 5 min aggregated samples is enabled" duration=4320h0m0s
level=info ts=2019-02-14T19:39:03.453659Z caller=compact.go:199 msg="retention policy of 1 hour aggregated samples is enabled" duration=24000h0m0s
level=info ts=2019-02-14T19:39:03.453759Z caller=compact.go:281 msg="starting compact node"
level=info ts=2019-02-14T19:39:03.453862Z caller=compact.go:821 msg="start sync of metas"
level=info ts=2019-02-14T19:39:03.453839Z caller=main.go:308 msg="Listening for metrics" address=0.0.0.0:10902
level=info ts=2019-02-14T19:39:03.658623Z caller=compact.go:827 msg="start of GC"
level=error ts=2019-02-14T19:39:03.808235Z caller=compact.go:265 msg="retriable error" err="compaction failed: compaction: download block 01D3MZ4NWJ8Q5YGSCB5F3SCDFT: create dir: mkdir /tmp/compact/0@{prometheus="kube-monitoring/operator-prometheus-lsf",prometheus_replica="prometheus-operator-prometheus-lsf-0",region="sc",role="lsf-metrics"}/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT/01D3MZ4NWJ8Q5YGSCB5F3SCDFT: file name too long"
^Clevel=info ts=2019-02-14T19:39:08.503792Z caller=main.go:192 msg="caught signal. Exiting." signal=interrupt
level=info ts=2019-02-14T19:39:08.503963Z caller=main.go:184 msg=exiting

Anything else we need to know
Nope and thanks for looking in to this, I hope I didn't miss anything else to help you understand this issue.

Duplicate issue #1491: compactor: failing to create directory, with error "file name too long"
Created: 2019-09-04T10:29:01Z
Thanos: v0.6.0
Prometheus: v2.11.2

thanos:v0.6.0
What happened
We started the thanos-compactor, after a while it crashes and ends up in a crashloop
What you expected to happen
Thanos-compactor running and compacting data
How to reproduce it (as minimally and precisely as possible):
Don't know.
Full logs to relevant components
Unfortunately, I can't egress the full logs, but the failure log message looks like:
level=error ts=2019-09-04T10:11:14.436861213Z caller=main.go:199 msg="running command failed" err="error executing compaction: compaction failed: compaction failed for group 0@{prometheus="REDACTED/prometheus-REDACTED",prometheus_replica="prometheus-prometheus-REDACTED-0"}: download block 01DCQ2K02CKMAD9N4R6ZA5VZW9: create dir: mkdir /local/state/compact/0@{prometheus="datadev/prometheus-REDACTED",prometheus_replica="prometheus-prometheus-REDACTED-0"}/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9/01DCQ2K02CKMAD9N4R6ZA5VZW9: file name too long"
Anything else we need to know

Closing comment (2019-09-04T11:24:37Z): Hi, Thanks for reporting. Looks like some bug, but users run 0.6.0 compactor just fine, so it must be specific to your environment.
How you configure Thanos compact? What object storage do you use? This looks like a duplicate of #844 So can you provide more data to that ticket instead?