Repository: mlcommons/inference
Issue: RNNT reference implement fails with pytorch 2.0.1
Root cause: External Dependency Breakage

Running rnnt reference implementation with latest pytorch gives the below error.

```
python3 -m pip install cmind
cm pull repo mlcommons@ck
cm run script --tags=run,mlperf,inference,generate-run-cmds,_performance-only \
--submitter="cTuning" --model=rnnt --backend=pytorch --device=cpu --scenario=Offline \
--test_query_count=5 --precision=fp32 --target_qps=5 --adr.librosa.version_max=0.9.1  -v --quiet
```

```
Traceback (most recent call last):
Dataset loaded with 4.36 hours. Filtered 1.02 hours. Number of samples: 2513
  File "/home/runner/CM/repos/local/cache/c8fc099562594c52/inference/speech_recognition/rnnt/run.py", line 91, in <module>
    main()
  File "/home/runner/CM/repos/local/cache/c8fc099562594c52/inference/speech_recognition/rnnt/run.py", line 56, in main
    sut = PytorchSUT(args.pytorch_config_toml, args.pytorch_checkpoint,
  File "/home/runner/CM/repos/local/cache/c8fc099562594c52/inference/speech_recognition/rnnt/pytorch_SUT.py", line 61, in __init__
    self.audio_preprocessor = torch.jit.script(self.audio_preprocessor)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/torch/jit/_script.py", line 1284, in script
    return torch.jit._recursive.create_script_module(
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/torch/jit/_recursive.py", line 480, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/torch/jit/_recursive.py", line 542, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/torch/jit/_script.py", line 614, in _construct
    init_fn(script_module)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/torch/jit/_recursive.py", line 520, in init_fn
    scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/torch/jit/_recursive.py", line 546, in create_script_module_impl
    create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/torch/jit/_recursive.py", line 397, in create_methods_and_properties_from_stubs
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
RuntimeError: 
undefined value torch:
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 154
        # dither
        if self.dither > 0 and not self.use_deterministic_dithering:
            x += self.dither * torch.randn_like(x)
                               ~~~~~ <--- HERE
    
        # do preemphasis
```

Reverting to torch==1.13.0 makes it work. 

```
python3 -m pip install cmind
cm pull repo mlcommons@ck
cm run script --tags=run,mlperf,inference,generate-run-cmds,_performance-only \
--submitter="cTuning" --model=rnnt --backend=pytorch --device=cpu --scenario=Offline \
--test_query_count=5 --precision=fp32 --target_qps=5 --adr.librosa.version_max=0.9.1  \
--adr.ml-engine-pytorch.version=1.13.0 --adr.ml-engine-torchvision.version=0.14.1 \
-v --quiet
```

Bumping this issue, would like to try run RNNT on Pytorch >= 2.0.0

Same issue here.
Torch version=2.2.0.dev20231010+rocm5.6

I run into the same issue using any torch version >= 2.0.1 with ROCm support.
Downgrading to torch 1.13.1 makes it work but it is not the ideal solution.

> I run into the same issue using any torch version >= 2.0.1 with ROCm support. Downgrading to torch 1.13.1 makes it work but it is not the ideal solution.

Hi @atsetogl , I saw previous version page on PyTorch official website (link is below)

https://pytorch.org/get-started/previous-versions/

previous version command is "torch==1.13.1+rocm5.2"

do you use this command? or "torch==1.13.1+rocm5.6" command is available? or I need build from PyTorch source code?

thanks for reply.

Hi @GasonC ,

You can install torch==1.13.1+rocm5.6 but not from the index of PyTorch.
To do so you need to run this command:
```
pip3 install torch==1.13.1 torchvision==0.14.1 -f https://repo.radeon.com/rocm/manylinux/rocm-rel-5.6/
```

It comes directly by Radeon and you can find more info [here](https://repo.radeon.com/rocm/manylinux/rocm-rel-5.6/README.html).

[This PR](https://github.com/mlcommons/inference/pull/1506) should fix the issue with pytorch 2.0. We'll soon add out-of-the-box run of mlperf models on Radeon GPUs. 

Hi @atsetogl , I downgrade my torch to version "1.13+rocm5.6" then rnn-t is worked.
but I monitor the gpu from "rocm-smi" command, it's seems my gpu is idle.
Is related because I downgrade python3.8 too?
(My python is 3.9 before, but torch-1.13/torchvision-0.14.1 needs python3.8)

Hi @GasonC , for GPU execution you need to use the out-of-the-box execution where you can use the argument `--device` and set `gpu` or `cuda` if I am not mistaken. However as Arjun mentioned earlier this is not available yet for AMD GPUs. The other solution would be to modify the code on your own for GPU execution. Thus the issue is not related to the Python version.

Thanks for reply, I'll check the code.

We now have support to run the reference implementation on GPUs. 

### On Nvidia RTX 4090

```
cm run script --tags=generate-run-cmds,inference --model=rnnt  --device=cuda \
--backend=pytorch --test_query_count=1000 --rerun \
--adr.mlperf-implementation.tags=_repo.https://github.com/GATEOverflow/inference,_branch.test \
--adr.mlperf-implementation.version=custom
```

```
Dataset loaded with 4.36 hours. Filtered 1.02 hours. Number of samples: 2513
Running Loadgen test...
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 31.101
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
```

This command can be tried on AMD GPU. It should automatically install the required pytorch variant.
```
cm run script --tags=generate-run-cmds,inference --model=rnnt  --device=rocm \
--backend=pytorch --test_query_count=100 --rerun \
--adr.mlperf-implementation.tags=_repo.https://github.com/GATEOverflow/inference,_branch.test \
--adr.mlperf-implementation.version=custom
```

Hi @arjunsuresh, thanks for the info. This works great!