Repository: kubernetes
Issue: CRI stats are overwritten by cAdvisor ones for Summary API
Labels: kind/bug,sig/node,sig/instrumentation,priority/important-longterm,triage/accepted

### What happened?

I was going over the memory metrics being returned by Summary API based on the table that @haircommander built [here](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2371-cri-pod-container-stats#current-fulfiller-of-metrics-endpoints--future-proposal). Notice that the provider for `container_memory_working_set_bytes` is listed as CRI. This appears to match what is being done in the code, as it's only `Memory.WorkingSetBytes` being picked up from the CRI stats for memory as seen here https://github.com/kubernetes/kubernetes/blob/0153febd9f0098d4b8d0d484927710eaf899ef40/pkg/kubelet/stats/cri_stats_provider.go#L627-L635

So the outcome should be that `Memory.WorkingSetBytes` is provided by CRI and all the other memory metrics come from cAdvisor.

Let's take the case of a Kubernetes cluster running on containerd (e.g. like AKS does) and also assume that the `PodAndContainerStatsFromCRI` feature gate introduced in version 1.23 is not enabled. In turn the CRI provider will be created accordingly here: https://github.com/kubernetes/kubernetes/blob/0153febd9f0098d4b8d0d484927710eaf899ef40/pkg/kubelet/kubelet.go#L717-L727
Next let's consider the method here: https://github.com/kubernetes/kubernetes/blob/0153febd9f0098d4b8d0d484927710eaf899ef40/pkg/kubelet/stats/cri_stats_provider.go#L270
Since we agreed that the `PodAndContainerStatsFromCRI` feature gate is not enabled, then the `if` block below doesn't run: https://github.com/kubernetes/kubernetes/blob/0153febd9f0098d4b8d0d484927710eaf899ef40/pkg/kubelet/stats/cri_stats_provider.go#L279-L305
We move forward and we retrieve the metrics from the containers via CRI here: https://github.com/kubernetes/kubernetes/blob/0153febd9f0098d4b8d0d484927710eaf899ef40/pkg/kubelet/stats/cri_stats_provider.go#L307
Then we get the list of cAdvisor metrics a few lines further down: https://github.com/kubernetes/kubernetes/blob/0153febd9f0098d4b8d0d484927710eaf899ef40/pkg/kubelet/stats/cri_stats_provider.go#L312
Then we start iterating over the metrics returned by the CRI, and we first process the CRI metrics. Next pod stats are updated via this method: https://github.com/kubernetes/kubernetes/blob/0153febd9f0098d4b8d0d484927710eaf899ef40/pkg/kubelet/stats/cri_stats_provider.go#L528
Now this is where it gets interesting: since cAdvisor most likely will return data for every single container - the following line runs: 
https://github.com/kubernetes/kubernetes/blob/0153febd9f0098d4b8d0d484927710eaf899ef40/pkg/kubelet/stats/cri_stats_provider.go#L349
Inside `addCadvisorContainerCPUAndMemoryStats` the container data that was previously returned from CRI (`cs.Memory`) appears to be wiped out by the cAdvisor data here https://github.com/kubernetes/kubernetes/blob/0153febd9f0098d4b8d0d484927710eaf899ef40/pkg/kubelet/stats/cri_stats_provider.go#L894-L909

But since going through the code is error-prone, I followed the advice from @dashpole and used a unit test. Running the `TestCRIListPodCPUAndMemoryStats` in [cri_stats_provider_test.go](https://github.com/kubernetes/kubernetes/blob/38ec822b46853acfb3c3dc7e386a8d4636c67e03/pkg/kubelet/stats/cri_stats_provider_test.go#L435) gives the following outcome for one of the containers:

![image](https://user-images.githubusercontent.com/31319583/147010219-814f6298-db77-4e47-a81d-9e5511b91ffe.png)

`containerStats0` contains the initial CRI mock memory metric data, `infos` is the map the contains the initial cAdvisor mock data, while c0 is one of the results of running the `ListPodCPUAndMemoryStats` method. Notice that the initial value for `WorkingSetBytes` is no longer present in the output.

### What did you expect to happen?

The `WorkingSetBytes` should have been picked up based on the CRI stats, not cAdvisor ones.

### How can we reproduce it (as minimally and precisely as possible)?

Run the unit test indicated in the original description.

### Anything else we need to know?

Building and testing were done on a Windows 10 box (21H1) running JetBrains GoLand 2021.2.4 configured to use an Ubuntu 20.04 instance running under WSL2 as its compile and test target.

### Kubernetes version

I've tested on the current `master` branch, but the same outcome is seen with tag `v1.21.1`.

### Cloud provider

Not relevant

### OS version

_No response_

### Install tools

_No response_

### Container runtime (CRI) and and version (if applicable)

_No response_

### Related plugins (CNI, CSI, ...) and versions (if applicable)

_No response_