Repository: kubernetes
Issue: Scrape etcd_db_total_size_in_use_in_bytes metric from API server to accurately track etcd database usage
Labels: kind/feature,needs-triage,sig/etcd

### What would you like to be added?

The Kubernetes API server currently scrapes the `etcd_db_total_size_in_bytes` metric, which has been recently renamed to the `apiserver_storage_size_bytes` metric. According to [Kubernetes documentation](https://kubernetes.io/docs/reference/instrumentation/metrics/), this metric provides the size of the etcd database file, as it is defined as `Size of the storage database file physically allocated in bytes`.

This metric is generally used to check for etcd database size contention, where the database is growing too large, allowing cluster admins to delete excess objects. The current scraping logic for this metric is [here](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go#L461-L493).
EKS - https://aws.github.io/aws-eks-best-practices/reliability/docs/controlplane/#etcd
Azure - https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/create-upgrade-delete/troubleshoot-apiserver-etcd?tabs=resource-specific#cause-2-an-offending-client-leaks-etcd-objects-and-results-in-a-slowdown-of-etcd

However, the above metric only tracks the file size of the etcd database, and not the actual space used by etcd. This metric does not change when etcd performs compaction, which [frees up space for etcd to use](https://etcd.io/docs/v3.4/op-guide/maintenance/#:~:text=Compacting%20old%20revisions%20internally%20fragments,reclaim%20the%20space%20on%20disk.), but does not reflect within the underlying host filesystem as the storage space is fragmented. Hence, the `apiserver_storage_size_bytes` metric could potentially mislead admins who believe that their etcd database is running out of space, but there is still free fragmented capacity that is usable. In addition, when admins delete excess objects, this would not be reflected in the `apiserver_storage_size_bytes` until an etcd defrag is performed later.

I'm proposing the addition of a new metric to the API server (could be called `apiserver_storage_size_in_use_bytes`), which would scrape the `etcd_db_total_size_in_use_in_bytes` metric from etcd. This metric tracks the amount of storage space that is actually in-use by etcd, and would change as admins delete excess objects and when compaction is performed.

Adding etcd metric definitions below for reference, fetched from [etcd page](https://etcd.io/docs/v3.6/metrics/etcd-metrics-latest.txt) 

| Metric | Definition | Notes |
| --- | --- |
| etcd_mvcc_db_total_size_in_bytes | Total size of the underlying database physically allocated in bytes.| Currently tracked as `apiserver_storage_size_bytes` in API server metrics |
| etcd_mvcc_db_total_size_in_use_in_bytes | Total size of the underlying database logically in use in bytes. | Proposing this metric to also be tracked within API server |

Happy to create a PR for the required changes as well

### Why is this needed?

This would allow cluster admins to effectively detect when their clusters are encountering storage contention issues at the etcd layer and take corrective actions, such as deleting Kubernetes resources that are no longer required. It would also allow admins to track storage consumption in real-time after deleting the objects, allowing them to verify that their changes are effective.