Repository: pytorch/benchmark
Issue: torch.cuda.OutOfMemoryError: CUDA out of memory.
Root cause: Documentation Deficiency

[readme_jp.txt](https://github.com/pytorch/benchmark/files/11311750/readme_jp.txt)


We only test our models on Nvidia A10G or higher GPUs in our CI. If you met this issue, it means your GPU doesn't have enough memory to run all the models in this suite.

Please consider limit your benchmark to run a subset of models, or tune the model batch sizes to avoid this issue.

all right, but how can I understand that GPU is slower?
``` 
(myenv_TESTPYTORCH) $ python test.py -k "test_mobilenet_v3_large_train_cpu"

Ran 1 test in 3.819s

OK
(myenv_TESTPYTORCH) $ python test.py -k "test_mobilenet_v3_large_train_cuda"

Ran 1 test in 5.541s

OK
```
[output_test_subset_0c.txt](https://github.com/pytorch/benchmark/files/11313508/output_test_subset_0c.txt)


> all right, but how can I understand that GPU is slower?
> 
> ```
> (myenv_TESTPYTORCH) $ python test.py -k "test_mobilenet_v3_large_train_cpu"
> 
> Ran 1 test in 3.819s
> 
> OK
> (myenv_TESTPYTORCH) $ python test.py -k "test_mobilenet_v3_large_train_cuda"
> 
> Ran 1 test in 5.541s
> 
> OK
> ```
> 
> [output_test_subset_0c.txt](https://github.com/pytorch/benchmark/files/11313508/output_test_subset_0c.txt)

Because `test.py` is our unit test script, not benchmarking script.
To speedup our unit test, we set `batch_size=1` whenever possible: https://github.com/pytorch/benchmark/blob/main/test.py#L84

When running with `bs=1`, it is common that GPU is slower because the GPU usually shows speedup only for large batch sizes.

To run a quick benchmark, use `run.py` instead.

```
$ python run.py mobilenet_v3_large -d cuda -t train
Running train method from mobilenet_v3_large on cuda in eager mode with input batch size 32 and precision fp32.
GPU Time:             24.026 milliseconds
CPU Total Wall Time:  24.061 milliseconds
GPU 0 Peak Memory:              2.6381 GB
CPU Peak Memory:                2.2002 GB

$ python run.py mobilenet_v3_large -d cpu -t train
Running train method from mobilenet_v3_large on cpu in eager mode with input batch size 32 and precision fp32.
CPU Total Wall Time: 804.349 milliseconds
CPU Peak Memory:                2.0547 GB
```

On batch size 32, GPU is ~40x faster than CPU.

```
$ python run.py mobilenet_v3_large -d cpu -t train --bs 1
Running train method from mobilenet_v3_large on cpu in eager mode with input batch size 1 and precision fp32.
CPU Total Wall Time:  67.230 milliseconds
CPU Peak Memory:                0.4287 GB
$ python run.py mobilenet_v3_large -d cuda -t train --bs 1
Running train method from mobilenet_v3_large on cuda in eager mode with input batch size 1 and precision fp32.
GPU Time:             17.833 milliseconds
CPU Total Wall Time:  17.882 milliseconds
GPU 0 Peak Memory:              1.3334 GB
CPU Peak Memory:                2.1953 GB
```

On batch size 1, GPU is only ~3x faster than CPU.

thank you

Am Mo., 24. Apr. 2023 um 20:51 Uhr schrieb Xu Zhao ***@***.***
>:

> When I use the run.py script to benchmark:
>
> $ python run.py mobilenet_v3_large -d cuda -t train
> Running train method from mobilenet_v3_large on cuda in eager mode with input batch size 32 and precision fp32.
> GPU Time:             24.026 milliseconds
> CPU Total Wall Time:  24.061 milliseconds
> GPU 0 Peak Memory:              2.6381 GB
> CPU Peak Memory:                2.2002 GB
>
> $ python run.py mobilenet_v3_large -d cpu -t train
> Running train method from mobilenet_v3_large on cpu in eager mode with input batch size 32 and precision fp32.
> CPU Total Wall Time: 804.349 milliseconds
> CPU Peak Memory:                2.0547 GB
>
> On the same batch size, GPU is ~40x faster than CPU.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pytorch/benchmark/issues/1572#issuecomment-1520662241>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AIQ5H6SANSFLPQLYWYPJYU3XC3DUPANCNFSM6AAAAAAXJW3WAI>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


-- 
Regards,
Joseph Pareti - Artificial Intelligence consultant
Joseph Pareti's AI Consulting Services
https://www.joepareti54-ai.com/
https://joeparetiai.org/ - under construction -
cell +49 1520 1600 209
cell +39 339 797 0644