Repository: pytorch
Issue: Timer benchmark stores only one time value, and therefore has broken mean/median/etc metrics
Labels: triaged,module: benchmark

### 🐛 Describe the bug

Here is a simple code to reproduce the error:

```
import torch
import torchvision
import torch.utils.benchmark as benchmark

if __name__ == '__main__':
    net = torchvision.models.resnet18().cuda()
    data = torch.rand((1, 3, 224, 224), device='cuda')

    net.eval()
    with torch.inference_mode():
        timer = benchmark.Timer(stmt='net(data)', num_threads=1, globals={'net': net, 'data': data})
        measurement = timer.timeit(number=10)
    print(measurement.raw_times, measurement.median, measurement.mean, measurement.iqr)
```

This should display a list of 10 values, different median and mean and non-zero iqr.
However, I get this:

```
[0.05537769995862618] 0.005537769995862618 0.005537769995862618 0.0
```

This means that only one value is stored, which not only prevents me doing what I wanted (i.e. get all the raw values and plot them on a graph), but also explains the broken behavior of the metrics (same mean and median, null iqr...).

I looked up the source code a bit, in torch/utils/benchmark/utils/timer.py, and found that:

```
    def _timeit(self, number: int) -> float:
        # Even calling a timer in C++ takes ~50 ns, so no real operation should
        # take less than 1 ns. (And this prevents divide by zero errors.)
        return max(self._timer.timeit(number), 1e-9)

    def timeit(self, number: int = 1000000) -> common.Measurement:
        """Mirrors the semantics of timeit.Timer.timeit().

        Execute the main statement (`stmt`) `number` times.
        https://docs.python.org/3/library/timeit.html#timeit.Timer.timeit
        """
        with common.set_torch_threads(self._task_spec.num_threads):
            # Warmup
            self._timeit(number=max(int(number // 100), 2))

            return common.Measurement(
                number_per_run=number,
                raw_times=[self._timeit(number=number)],
                task_spec=self._task_spec
            )
```

And, if I understand well, it means that, by definition, only one value will ever be stored.
I don't think this is the intended behavior at all, and therefore suggest either to fix the source code, or to update the documentation to make it clearer how it actually behaves.

### Versions

Collecting environment information...
PyTorch version: 2.1.0.dev20230726+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Famille
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 12.2.91
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1050
Nvidia driver version: 536.25
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:










Revision=

Versions of relevant libraries:
[pip3] numpy==1.25.1
[pip3] torch==2.1.0.dev20230726+cu121
[pip3] torchaudio==2.1.0.dev20230727+cu121
[pip3] torchinfo==1.8.0
[pip3] torchvision==0.16.0.dev20230727+cu121
[conda] Could not collect