Repository: kubernetes
Issue: expose some metrics from HPA controller
Labels: sig/autoscaling,kind/feature,sig/instrumentation,triage/accepted

Part of https://github.com/kubernetes/kubernetes/issues/115467
Ref: https://github.com/kubernetes/enhancements/issues/1610

### What would you like to be added?

Currently, HPA controller doesn't have any metrics in it. 

Ideas:

- `scaling_attempt_duration_seconds`: How long the HPA controller takes to complete reconcile one HPA object.
  - Probably equals the execution time of [`reconcileAutoscaler`](https://github.com/kubernetes/kubernetes/blob/9b09d0600a69a2eb36b0d136465ccc3c179dacdb/pkg/controller/podautoscaler/horizontal.go#L677). 
  - Has these labels:
    - `error`: should be `true` or `false`.
  - Usecase: the performance measurement.
- `metrics_computation_duration_seconds`: How long each metric type takes to compute the ideal replica num. 
  - Probably equals the execution time around [here](https://github.com/kubernetes/kubernetes/blob/9b09d0600a69a2eb36b0d136465ccc3c179dacdb/pkg/controller/podautoscaler/horizontal.go#L400-L434).
  - Has these labels:
    -  `metric_type`: should be one of `Resource`, `Pods`, `Object`, `ContainerResource`, and `External`.
    - `error`: should be `true` or `false`. 
  - Usecase: the performance measurement. e.g., If the external metrics take too much time compared to other metric types, then users may want to look into the reason to improve.
- `metrics_computation_attempts_total`: The number of attempts to compute the desired num of replicas for that metric.
  - Has these labels:
    -  `metric_type`: should be one of `Resource`, `Pods`, `Object`, `ContainerResource`, and `External`.
    - `error`: should be `true` or `false`. 
  - Usecase: check if metric type is being used in their cluster when some new metric type is introduced like https://github.com/kubernetes/enhancements/issues/1610. 
- `horizontal_autoscalers`: The total number of HPA objects in the cluster.
  - Usecase: The more HPA objects, the longer time HPA controller takes to recalculate. The the cluster admin may want to monitor the num of HPA objects on the cluster. (Actually, GKE has the suggested upper limit on HPA object. [ref](https://cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler#scalability))
- `horizontal_autoscaler_metrics`: The total number of each metrics in all HPAs in the cluster. (= How many HPAs are using specific metric type)
  - Has these labels:
    -  `metric_type`: should be one of `Resource`, `Pods`, `Object`, `ContainerResource`, and `External`.
  - Usecase: When the cluster is using the custom metrics or the external metrics, the num of HPA objects which refer to them is the point to decide the capacity of the related component. (e.g., Prometheus Adapter)
  
Please comment if any other ideas.

### Why is this needed?

In the PRR for any new features, we need to provide a way to enable admins to know if a feature is being used.
That's the initial motivation for this. 
https://github.com/kubernetes/enhancements/pull/3834#discussion_r1099581726

But, of course, we can discuss the metrics from other perspective as well; HPA is the critical component for the service reliability, and end-users want to make sure if something wrong/unexpected happens carefully.