Repository: pytorch/benchmark
Issue: How to execute with docker?
Root cause: Documentation Deficiency

I'm using ARG BASE_IMAGE=ghcr.io/pytorch/torchbench:latest 
but I am having problems with this container.
or should use ghcr.io/pytorch:pytorch-nightly or [ghcr.io/pytorch:pytorch-nightly](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch)

`ghcr.io/pytorch/torchbench:latest` is the container we use. Can I have the concrete problems with this container?

> `ghcr.io/pytorch/torchbench:latest` is the container we use. Can I have the concrete problems with this container?

so pulling from this image, can I directly put this in my personal computer?
pytest test_bench.py --benchmark-autosave --ignore_machine_config

You can try this command.
Another option is to try `python run.py resnet50 -d cpu` to test individual model run.

> You can try this command. Another option is to try `python run.py resnet50 -d cpu` to test individual model run.

it's for review gpu's, for example I have 3090, 4090, and rtx a6000 ada so i run all tests with statistics

You need to run the docker with `--gpus all`, and try 
```
python run.py resnet50 -d cuda
```

Note that since we don't test on these hardware (we only test on NVidia A10G and A100 GPU platforms), you may encounter issues such as CUDA OOM.


> You need to run the docker with `--gpus all`, and try
> 
> ```
> python run.py resnet50 -d cuda
> ```
> 
> Note that since we don't test on these hardware (we only test on NVidia A10G and A100 GPU platforms), you may encounter issues such as CUDA OOM.

What if I wanted to run all the models and get a csv, json or something similar to get all the values both training and evaluation? And high memory(batch_size) use like rtx ada a6000 48gb?
Is it running like
```bash
python3 run_benchmark.py ? -d cuda
```


Can you please try the following command:
```
python run_benchmark.py torch-nightly -d cuda -t train,eval
```

The output will be a json file as `.userbenchmark/torch-nightly/metrics-YYmmddHHMMSS.json`

> Can you please try the following command:
> 
> ```
> python run_benchmark.py torch-nightly -d cuda -t train,eval
> ```
> 
> The output will be a json file as `.userbenchmark/torch-nightly/metrics-YYmmddHHMMSS.json`

Is the batch size calculated automatically?
For example, take advantage of the 48gb of the rtxa6000 and see if there are differences with the 4090 for example.

> Is the batch size calculated automatically? For example, take advantage of the 48gb of the rtxa6000 and see if there are differences with the 4090 for example.

It uses the default batch size setup in the model code and metadata.yaml. The determination of the default batch size is a complex process, see: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/model.py#L186

Some models don't allow customizing batch size, for those, we don't allow user changing the batch size.
Otherwise, we will use the default batch size from either upstream model repo or research paper for train.
For inference, we use the largest batch size possible on A100 by default.

> .userbenchmark/torch-nightly/metrics-YYmmddHHMMSS.json

Running TorchBenchModelConfig(name='hf_T5_base', test='train', device='cuda', jit=False, batch_size=None, extra_args=[], extra_env=None) ...components._impl.workers.subprocess_rpc.ChildTraceException: Traceback (most recent call last):
  File "/workspace/benchmark/components/_impl/workers/subprocess_rpc.py", line 482, in _run_block
    exec(  # noqa: P204
  File "<subprocess-worker>", line 2, in <module>
  File "/workspace/benchmark/torchbenchmark/util/model.py", line 269, in invoke
    self.train()
  File "/workspace/benchmark/torchbenchmark/util/framework/huggingface/model_factory.py", line 143, in train
    outputs = self.model(**self.example_inputs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1704, in forward
    decoder_outputs = self.decoder(
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1074, in forward
    layer_outputs = layer_module(
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 693, in forward
    self_attention_outputs = self.layer[0](
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 600, in forward
    attention_output = self.SelfAttention(
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 563, in forward
    attn_weights = nn.functional.dropout(
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/nn/functional.py", line 1270, in dropout
    return _VF.dropout_(input, p, training) if inplace else _VF.dropout(input, p, training)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 384.00 MiB. GPU 0 has a total capacty of 47.50 GiB of which 201.44 MiB is free. Process 56330 has 1000.00 MiB memory in use. Process 63646 has 44.40 GiB memory in use. Of the allocated memory 43.78 GiB is allocated by PyTorch, and 129.50 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

    working_dir: /tmp/tmpxnmu1g0i
    stdout:
        [2023-06-06] 15:36:25.193302: TIMER_SUBPROCESS_BEGIN_EXEC
        [2023-06-06] 15:36:25.549047: TIMER_SUBPROCESS_FAILED
        [2023-06-06] 15:36:25.549076: TIMER_SUBPROCESS_FINISHED
        [2023-06-06] 15:36:25.549083: TIMER_SUBPROCESS_BEGIN_READ

    stderr:


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/benchmark/run_benchmark.py", line 36, in <module>
    run()
  File "/workspace/benchmark/run_benchmark.py", line 30, in run
    benchmark.run(bm_args)
  File "/workspace/benchmark/userbenchmark/torch-nightly/run.py", line 214, in run
    metrics = run_config(config, dryrun=args.dryrun)
  File "/workspace/benchmark/userbenchmark/torch-nightly/run.py", line 165, in run_config
    result: TorchBenchModelMetrics = get_model_test_metrics(model, metrics=metrics)
  File "/workspace/benchmark/torchbenchmark/util/experiment/metrics.py", line 99, in get_model_test_metrics
    latencies = get_latencies(model.invoke, device, nwarmup=nwarmup, num_iter=num_iter)
  File "/workspace/benchmark/torchbenchmark/util/experiment/metrics.py", line 27, in get_latencies
    func()
  File "/workspace/benchmark/torchbenchmark/__init__.py", line 401, in invoke
    self.worker.run("""
  File "/workspace/benchmark/components/_impl/workers/subprocess_worker.py", line 155, in run
    self._run(snippet)
  File "/workspace/benchmark/components/_impl/workers/subprocess_worker.py", line 320, in _run
    subprocess_rpc.SerializedException.raise_from(
  File "/workspace/benchmark/components/_impl/workers/subprocess_rpc.py", line 458, in raise_from
    raise e from ChildTraceException(traceback_str)
components._impl.workers.subprocess_rpc.UnserializableException: ("<class 'torch.cuda.OutOfMemoryError'>", "('CUDA out of memory. Tried to allocate 384.00 MiB. GPU 0 has a total capacty of 47.50 GiB of which 201.44 MiB is free. Process 56330 has 1000.00 MiB memory in use. Process 63646 has 44.40 GiB memory in use. Of the allocated memory 43.78 GiB is allocated by PyTorch, and 129.50 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF',)")
--2023-06-06 17:36:26--  https://raw.githubusercontent.com/lambdal/deeplearning-benchmark/master/pytorch/setup.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK


This is the CUDA OOM error. The direct cause is small GPU memory size. It could be the model is too large or the model input size is too large. In the latter case, you can try use a smaller batch size.

> This is the CUDA OOM error. The direct cause is small GPU memory size. It could be the model is too large or the model input size is too large. In the latter case, you can try use a smaller batch size.

Where do you change this?

> > This is the CUDA OOM error. The direct cause is small GPU memory size. It could be the model is too large or the model input size is too large. In the latter case, you can try use a smaller batch size.
> 
> Where do you change this?

Different models have different default batch sizes. For `hf_T5_base` train test, you can change it at https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/hf_T5_base/__init__.py#L8

> > > This is the CUDA OOM error. The direct cause is small GPU memory size. It could be the model is too large or the model input size is too large. In the latter case, you can try use a smaller batch size.
> > 
> > 
> > Where do you change this?
> 
> Different models have different default batch sizes. For `hf_T5_base` train test, you can change it at https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/hf_T5_base/__init__.py#L8

new error:
Running TorchBenchModelConfig(name='torchrec_dlrm', test='train', device='cuda', jit=False, batch_size=None, extra_args=[], extra_env=None) ...components._impl.workers.subprocess_rpc.ChildTraceException: Traceback (most recent call last):
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/_ops.py", line 709, in __getattr__
    op, overload_names = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator fbgemm::jagged_2d_to_dense

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/benchmark/components/_impl/workers/subprocess_rpc.py", line 482, in _run_block
    exec(  # noqa: P204
  File "<subprocess-worker>", line 44, in <module>
  File "<subprocess-worker>", line 14, in _run_in_worker_f
  File "/opt/conda/envs/torchbench/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/workspace/benchmark/torchbenchmark/models/torchrec_dlrm/__init__.py", line 7, in <module>
    from .data.dlrm_dataloader import get_dataloader
  File "/workspace/benchmark/torchbenchmark/models/torchrec_dlrm/data/dlrm_dataloader.py", line 13, in <module>
    from torchrec.datasets.criteo import (
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torchrec/__init__.py", line 8, in <module>
    import torchrec.distributed  # noqa
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torchrec/distributed/__init__.py", line 36, in <module>
    from torchrec.distributed.model_parallel import DistributedModelParallel  # noqa
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torchrec/distributed/model_parallel.py", line 20, in <module>
    from torchrec.distributed.planner import (
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torchrec/distributed/planner/__init__.py", line 22, in <module>
    from torchrec.distributed.planner.planners import EmbeddingShardingPlanner  # noqa
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torchrec/distributed/planner/planners.py", line 19, in <module>
    from torchrec.distributed.planner.constants import BATCH_SIZE, MAX_SIZE
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torchrec/distributed/planner/constants.py", line 10, in <module>
    from torchrec.distributed.embedding_types import EmbeddingComputeKernel
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torchrec/distributed/embedding_types.py", line 14, in <module>
    from fbgemm_gpu.split_table_batched_embeddings_ops_training import EmbeddingLocation
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/fbgemm_gpu/__init__.py", line 21, in <module>
    from . import _fbgemm_gpu_docs  # noqa: F401, E402
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/fbgemm_gpu/_fbgemm_gpu_docs.py", line 19, in <module>
    torch.ops.fbgemm.jagged_2d_to_dense,
  File "/opt/conda/envs/torchbench/lib/python3.10/site-packages/torch/_ops.py", line 713, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' 'fbgemm' object has no attribute 'jagged_2d_to_dense'


torchrec_dlrm requires only works on torch nightly release, we are consider moving it to the canary_models. Maybe just try skip this specific model.

> torchrec_dlrm requires only works on torch nightly release, we are consider moving it to the canary_models. Maybe just try skip this specific model.

I’m using torch nightly with cuda 12.1. Well I will try to skip. Thanks
```
python run_benchmark.py torch-nightly -d cuda -t train,eval
```
https://github.com/johnnynunez/pytorch_benchmark/blob/main/docker/Dockerfile

> > torchrec_dlrm requires only works on torch nightly release, we are consider moving it to the canary_models. Maybe just try skip this specific model.
> 
> I’m using torch nightly with cuda 12.1. Well I will try to skip. Thanks
> 
> ```
> python run_benchmark.py torch-nightly -d cuda -t train,eval
> ```
> 
> https://github.com/johnnynunez/pytorch_benchmark/blob/main/docker/Dockerfile

The error means the fbgemm_gpu_nightly package version you are using is not compatible to the torch nightly version.

If so, you can try installing the fbgemm_gpu_nightly package, or to recompile the package against the pytorch version you are using. If it still doesn't work, try skip it.
https://github.com/pytorch/fbgemm

> > > torchrec_dlrm requires only works on torch nightly release, we are consider moving it to the canary_models. Maybe just try skip this specific model.
> > 
> > 
> > I’m using torch nightly with cuda 12.1. Well I will try to skip. Thanks
> > ```
> > python run_benchmark.py torch-nightly -d cuda -t train,eval
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > https://github.com/johnnynunez/pytorch_benchmark/blob/main/docker/Dockerfile
> 
> The error means the fbgemm_gpu_nightly package version you are using is not compatible to the torch nightly version.
> 
> If so, you can try installing the fbgemm_gpu_nightly package, or to recompile the package against the pytorch version you are using. If it still doesn't work, try skip it. https://github.com/pytorch/fbgemm

In case of skip, which file is it?

> > > torchrec_dlrm requires only works on torch nightly release, we are consider moving it to the canary_models. Maybe just try skip this specific model.
> > 
> > 
> > I’m using torch nightly with cuda 12.1. Well I will try to skip. Thanks
> > ```
> > python run_benchmark.py torch-nightly -d cuda -t train,eval
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > https://github.com/johnnynunez/pytorch_benchmark/blob/main/docker/Dockerfile
> 
> The error means the fbgemm_gpu_nightly package version you are using is not compatible to the torch nightly version.
> 
> If so, you can try installing the fbgemm_gpu_nightly package, or to recompile the package against the pytorch version you are using. If it still doesn't work, try skip it. https://github.com/pytorch/fbgemm
<img width="755" alt="Screenshot 2023-06-07 at 10 32 38" src="https://github.com/pytorch/benchmark/assets/22727137/d3cad870-8936-484e-b279-9f49894c59ef">

Not save the time for each model?
How to interpret the results? Do we need to compute_score?


@johnnynunez 
The `*_latency` metric is the median per-iteration latency of the test in ms, `*_cmem` is the peak CPU memory, and `*_gmem` is the peak GPU memory, both in GB.

> @johnnynunez
> The `*_latency` metric is the median per-iteration latency of the test in ms, `*_cmem` is the peak CPU memory, and `*_gmem` is the peak GPU memory, both in GB.

How Could I generate an score? I see score.py but, it’s not updated

@johnnynunez 

`score.py` is deprecated. Can you try running `python run_benchmark.py torch-nightly -c v3-cuda-tests.yaml`? You might need to delete the torchrec_dlrm model from the config: https://github.com/pytorch/benchmark/blob/main/userbenchmark/torch-nightly/v3-cuda-tests.yaml

If it works, you will see the "v3_score" metrics in metrics-YYmmddHHMMSS.json, like the artifact in https://github.com/pytorch/benchmark/actions/runs/5213333440.

> python run_benchmark.py torch-nightly -c v3-cuda-tests.yaml

Nice, I will try to help you, and add the recent networks and improvement in the future, sorry with spam, but I'm interested because is the best repository with the recent networks, and I have a lot of hardware to review. I will investigate for example calculate the batch automatically...

@johnnynunez Please consider writing a userbenchmark for calculating the bach sizes automatically, thanks!