Repository: prometheus-operator/kube-prometheus

Issue #503: code_verb:apiserver_request_total:increase30d is failing to evaluate
Created: 2020-04-18T10:49:57Z
What happened?
I upgraded from an older version from a few months ago to the latest 0.5 release with jb and Prometheus is failing rule evaluations starting to fire immediately. I checked Prometheus and https://github.com/coreos/kube-prometheus/blob/dcc46c8aa8c242b845024188a66171b5f08b8513/manifests/prometheus-rules.yaml#L393 is in ERR state: query processing would load too many samples into memory in query execution
Did you expect to see some different?
The rule included shouldn't be failing
How to reproduce it (as minimally and precisely as possible):
Environment
GKE


Prometheus Operator version:
v0.38.1


Kubernetes version information:


Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-26T06:16:15Z", GoVersion:"go1.14", Compiler:"gc", Platform:"darwin/amd64"} 
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-gke.5", GitCommit:"a5bf731ea129336a3cf32c3375317b3a626919d7", GitTreeState:"clean", BuildDate:"2020-03-31T02:49:49Z", GoVersion:"go1.12.17b4", Compiler:"gc", Platform:"linux/amd64"}


Kubernetes cluster kind:
GKE


level=warn ts=2020-04-18T11:01:09.892Z caller=manager.go:525 component="rule manager" group=kube-apiserver.rules msg="Evaluating rule failed" rule="record: code_verb:apiserver_request_total:increase30d\nexpr: sum by(code, verb) (increase(apiserver_request_total{job=\"apiserver\"}[30d]))\n" err="query processing would load too many samples into memory in query execution"

Duplicate issue #747: Does prometheus have a thread leak other CPU usage bug?
Created: 2020-10-29T22:22:12Z
What happened?
Installed Prometheus Operator via a helm chart v 9.3.1 . after a few days we get CPUThrottling alerts.
looking at the Kubernetes / Compute Resources / Pod dashboard in grafana over 7 days I can see a gradual rise in CPU until I start throttling see image attached

Did you expect to see some different?
expected no alerts. we are not adding more work to the system or more exports and servicemonitors
How to reproduce it (as minimally and precisely as possible):
deploy prometheus-operator to an environment and watch it for a few days
Environment
Production


Prometheus Operator version:
quay.io/coreos/prometheus-operator:v0.38.1
  


kubectl -n monitoring describe deployment prometheus-prometheus-oper-operator
Name:                   prometheus-prometheus-oper-operator
Namespace:              monitoring
CreationTimestamp:      Tue, 29 Sep 2020 08:49:40 +1000
Labels:                 app=prometheus-operator-operator
                        chart=prometheus-operator-9.3.1
                        heritage=Tiller
                        release=prometheus
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=prometheus-operator-operator,release=prometheus
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=prometheus-operator-operator
                    chart=prometheus-operator-9.3.1
                    heritage=Tiller
                    release=prometheus
  Service Account:  prometheus-prometheus-oper-operator
  Containers:
   prometheus-operator:
    Image:      quay.io/coreos/prometheus-operator:v0.38.1
    Port:       8080/TCP
    Host Port:  0/TCP
    Args:
      --manage-crds=true
      --kubelet-service=kube-system/prometheus-prometheus-oper-kubelet
      --logtostderr=true
      --localhost=127.0.0.1
      --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.38.1
      --config-reloader-image=docker.io/jimmidyson/configmap-reload:v0.3.0
      --config-reloader-cpu=0
      --config-reloader-memory=0
    Environment:  <none>
    Mounts:       <none>
   tls-proxy:
    Image:      squareup/ghostunnel:v1.5.2
    Port:       8443/TCP
    Host Port:  0/TCP
    Args:
      server
      --listen=:8443
      --target=127.0.0.1:8080
      --key=cert/key
      --cert=cert/cert
      --disable-authentication
    Environment:  <none>
    Mounts:
      /cert from tls-proxy-secret (ro)
  Volumes:
   tls-proxy-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-prometheus-oper-admission
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   prometheus-prometheus-oper-operator-6dd4fd7466 (1/1 replicas created)
Events:          <none>




Kubernetes version information:
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-20T14:12:21Z", GoVersion:"go1.15.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T18:46:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  


Kubernetes cluster kind:
AWS EKS v 1.17
insert how you created your cluster: kops, bootkube, tectonic-installer, etc.
terraform using this module terraform-aws-eks


Manifests:


insert manifests relevant to the issue


Prometheus Operator Logs:

Insert Prometheus Operator logs relevant to the issue here


Prometheus Logs:

Insert Prometheus logs relevant to the issue here

Anything else we need to know?:
I have not added any rules other than the defaults that come with prometheus-operator helm chart. I see the same behaviour in an brand new EKS kubernetes environment with nothing deployed except propmetheus-operator and nginx-ingress and external-dns

Closing comment (2020-12-07T16:51:03Z): Looking quickly at the CPU profile, it looks like most of the time is spent in rules evaluation. From the collected metrics, the group that takes the longest time to evaluate is kube-apiserver.rules. I suspect you're hitting something similar to #503.