Repository: kubernetes/kube-state-metrics

Issue #2100: Custom resource stopped working with 2.9.2
Created: 2023-06-22T17:15:25Z
What happened:
Updated from 2.8.2 to 2.9.2, kube-state-metrics does not start. Same configuration.

I0622 17:08:31.102616       1 server.go:198] "Used resources" resources=[daemonsets persistentvolumes volumeattachments virtualmachineinstances endpoints secrets horizontalpodautoscalers ingresses limitranges poddisruptionbudgets certificatesigningrequests namespaces nodes statefulsets configmaps pods replicationcontrollers storageclasses cronjobs replicasets resourcequotas services deployments mutatingwebhookconfigurations persistentvolumeclaims validatingwebhookconfigurations jobs networkpolicies]
E0622 17:08:31.102649       1 wrapper.go:40] "Failed to run kube-state-metrics" err="failed to set up resources: resource virtualmachineinstances does not exist. Available resources: replicasets,secrets,storageclasses,networkpolicies,ingresses,jobs,mutatingwebhookconfigurations,pods,services,volumeattachments,endpointslices,endpoints,limitranges,poddisruptionbudgets,serviceaccounts,clusterroles,nodes,validatingwebhookconfigurations,namespaces,horizontalpodautoscalers,leases,resourcequotas,rolebindings,deployments,persistentvolumes,roles,statefulsets,persistentvolumeclaims,cronjobs,replicationcontrollers,configmaps,clusterrolebindings,daemonsets,ingressclasses,certificatesigningrequests"

What you expected to happen:
No configuration change required between 2.8.2 and 2.9.2, since the changelog doesn't mention anything. Alternatively, a useful error message, not "resource does not exist" when it is listed in "Used resources".
How to reproduce it (as minimally and precisely as possible):
Command line (running in-cluster):
--port=8080
--resources=certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments,virtualmachineinstances
--custom-resource-state-config-file=/config/custom-resource-config.yaml
--telemetry-port=8081

/config/custom-resource-config.yaml:
spec:
  resources:
    - groupVersionKind:
        group: kubevirt.io
        version: v1
        kind: VirtualMachineInstance
      labelsFromPath:
        namespace: [metadata, namespace]
        name: [metadata, name]
      metrics:
        - name: vmi_migration_method
          help: "Migration method: LiveMigration if only the instance's memory needs to be copied, or BlockMigration if some of the disks require copying"
          each:
            type: StateSet
            stateSet:
              labelName: migration_method
              path: [status, migrationMethod]
              list: [LiveMigration, BlockMigration]
        - name: vmi_conditions
          help: "VMI conditions"
          each:
            type: Gauge
            gauge:
              path: [status, conditions]
              labelsFromPath:
                type: ["type"]
              valueFrom: ["status"]
        - name: vmi_last_migration
          help: "End timestamp of last migration"
          each:
            type: Gauge
            gauge:
              path: [status, migrationState, endTimestamp]
        - name: vmi_phase
          help: "VMI phase"
          each:
            type: StateSet
            stateSet:
              labelName: phase
              path: [status, phase]
              list: [Pending, Scheduling, Scheduled, Running, Succeeded, Failed, Unknown]
        - name: vmi_info
          help: "VirtualMachineInstance information"
          each:
            type: Info
            info:
              labelsFromPath:
                node_name: [status, nodeName]
Anything else we need to know?:
Environment:

kube-state-metrics version: 2.9.2
Kubernetes version (use kubectl version): 1.25.10
Cloud provider or hardware configuration: kubeadm Linux amd64

Duplicate issue #2098: kube-state-metrics 2.9.2 is not generating custom metrics with namespace role
Created: 2023-06-16T22:53:25Z
What happened:
When using kube-state-metrics with custom config with namespace role. It worked well until 2.8.2 It started failing in 2.9.2 version. I see metrics not getting generated with an error
1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.5/tools/cache/reflector.go:169: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:namespace:role-name" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope

What you expected to happen:
Metrics should be displayed with namespace role. This CRDs are part of namespace. So, it should list those CRDs in the metrics with custom-config file.
How to reproduce it (as minimally and precisely as possible):
Deploy kube-state-metrics with namespace role. Pass any custom config to kube-state-metrics error can be seen in logs.
Anything else we need to know?:
N/A
Environment:

kube-state-metrics version:  2.9.2
Kubernetes version (use kubectl version): Can be reproduced on any version.
Cloud provider or hardware configuration: N/A
Other info: N/A

Closing comment (2023-06-27T16:43:33Z): This is a duplicate of #2100, custom metrics now requires rbac permissions on the CRD resource. We will update the documentation soon, sorry for the inconvenience.