Repository: pytorch
Issue: RuntimeError: dims.value().size() == self->getMaybeRFactorDomain().size()
Labels: oncall: jit

### 🐛 Describe the bug

When I was doing the inference with a model export by `torch.jit.script()` (the pytorch version used to export the model is before v2.0, I don't know what is it exactly). And I met this errors:

```
python tools/recognize.py --world-s
ize 1 --manifest-in data/manifests/librilight_chunk_cuts_small.jsonl.gz --manifest-out librilight_cuts_test2.jsonl.gz --nn-model-filename exp/exp/jit_script.pt
 --tokens exp/data/lang_bpe_500/tokens.txt                                                                                                                     
2023-08-31 15:27:31,194 INFO [recognize.py:323] Decoding started                                                                                               
2023-08-31 15:27:31,197 INFO [recognize.py:336] {'subsampling_factor': 4, 'frame_shift_ms': 10, 'beam_size': 4, 'world_size': 1, 'master_port': 12354, 'manifes
t_in': PosixPath('data/manifests/librilight_chunk_cuts_small.jsonl.gz'), 'manifest_out': PosixPath('librilight_cuts_test2.jsonl.gz'), 'log_dir': PosixPath('log
s'), 'nn_model_filename': 'exp/exp/jit_script.pt', 'tokens': 'exp/data/lang_bpe_500/tokens.txt', 'decoding_method': 'greedy_search', 'max_duration': 600.0, 're
turn_cuts': True, 'num_mel_bins': 80, 'num_workers': 8, 'manifest_out_dir': PosixPath('.'), 'suffix': '.jsonl.gz', 'cuts_filename': 'librilight_cuts_test2', 'b
lank_id': 0, 'unk_id': 2, 'vocab_size': 500}                                                                                                                   
2023-08-31 15:27:31,268 INFO [recognize.py:341] device: cuda:0                                                                                                 
2023-08-31 15:27:31,269 INFO [recognize.py:343] Loading jit model                                                                                              
2023-08-31 15:27:44,860 INFO [recognize.py:299] cuts processed until now is 20                                                                                 
/star-kw/kangwei/dev_tools/anaconda/envs/textsearch/lib/python3.10/site-packages/torch/nn/modules/module.py:1501: UserWarning: operator() profile_node %626 : int = prim::profile_ivalue(%624)
 does not have profile information (Triggered internally at ../third_party/nvfuser/csrc/graph_fuser.cpp:104.)
  return forward_call(*args, **kwargs)
/star-kw/kangwei/dev_tools/anaconda/envs/textsearch/lib/python3.10/site-packages/torch/nn/modules/module.py:1501: UserWarning: FALLBACK path has been taken in$ide: compileCudaFusionGroup. This is an indication that codegen Failed for some reason.
To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback`
To report the issue, try enable logging via setting the envvariable ` export PYTORCH_JIT_LOG_LEVEL=manager.cpp`
 (Triggered internally at ../third_party/nvfuser/csrc/manager.cpp:243.)
  return forward_call(*args, **kwargs)
/star-kw/kangwei/dev_tools/anaconda/envs/textsearch/lib/python3.10/site-packages/torch/nn/modules/module.py:1501: UserWarning: FALLBACK path has been taken inside: runCudaFusionGroup. This is an indication that codegen Failed for some reason.
To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback`
 (Triggered internally at ../third_party/nvfuser/csrc/manager.cpp:335.)
  return forward_call(*args, **kwargs)
Traceback (most recent call last):
  File "/star-kw/kangwei/code/text_search/examples/libriheavy/tools/recognize.py", line 433, in <module>
    main()
  File "/star-kw/kangwei/code/text_search/examples/libriheavy/tools/recognize.py", line 426, in main
    run(rank=0, world_size=world_size, args=args, in_cuts=in_cuts)
  File "/star-kw/kangwei/dev_tools/anaconda/envs/textsearch/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)                                                
  File "/star-kw/kangwei/code/text_search/examples/libriheavy/tools/recognize.py", line 365, in run
    decode_dataset(                     
  File "/star-kw/kangwei/code/text_search/examples/libriheavy/tools/recognize.py", line 287, in decode_dataset
    hyps, timestamps, scores = decode_one_batch(                                  File "/star-kw/kangwei/code/text_search/examples/libriheavy/tools/recognize.py", line 184, in decode_one_batch
    encoder_out, encoder_out_lens = model.encoder(                              
  File "/star-kw/kangwei/dev_tools/anaconda/envs/textsearch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):                               
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):                               
RuntimeError: shape '[1, 0, 2]' is invalid for input of size 7659520
```

When I set the `export PYTORCH_NVFUSER_DISABLE=fallback` the logs became:

```
python tools/recognize.py --world-size 1 --manifest-in data/manifests/librilight_chunk_cuts_small.jsonl.gz --manifest-out librilight_cuts_test2.jsonl.gz --nn-model-filename exp/exp/jit_script.pt --tokens exp/data/lang_bpe_500/tokens.txt 
2023-08-31 15:33:15,659 INFO [recognize.py:323] Decoding started
2023-08-31 15:33:15,663 INFO [recognize.py:336] {'subsampling_factor': 4, 'frame_shift_ms': 10, 'beam_size': 4, 'world_size': 1, 'master_port': 12354, 'manifest_in': PosixPath('data/manifests/librilight_chunk_cuts_small.jsonl.gz'), 'manifest_out': PosixPath('librilight_cuts_test2.jsonl.gz'), 'log_dir': PosixPath('logs'), 'nn_model_filename': 'exp/exp/jit_script.pt', 'tokens': 'exp/data/lang_bpe_500/tokens.txt', 'decoding_method': 'greedy_search', 'max_duration': 600.0, 'return_cuts': True, 'num_mel_bins': 80, 'num_workers': 8, 'manifest_out_dir': PosixPath('.'), 'suffix': '.jsonl.gz', 'cuts_filename': 'librilight_cuts_test2', 'blank_id': 0, 'unk_id': 2, 'vocab_size': 500}
2023-08-31 15:33:15,737 INFO [recognize.py:341] device: cuda:0
2023-08-31 15:33:15,737 INFO [recognize.py:343] Loading jit model
2023-08-31 15:33:29,322 INFO [recognize.py:299] cuts processed until now is 20
/star-kw/kangwei/dev_tools/anaconda/envs/textsearch/lib/python3.10/site-packages/torch/nn/modules/module.py:1501: UserWarning: operator() profile_node %626 : int = prim::profile_ivalue(%624)
 does not have profile information (Triggered internally at ../third_party/nvfuser/csrc/graph_fuser.cpp:104.)
  return forward_call(*args, **kwargs)
Traceback (most recent call last):
  File "/star-kw/kangwei/code/text_search/examples/libriheavy/tools/recognize.py", line 433, in <module>
    main()
  File "/star-kw/kangwei/code/text_search/examples/libriheavy/tools/recognize.py", line 426, in main
    run(rank=0, world_size=world_size, args=args, in_cuts=in_cuts)
  File "/star-kw/kangwei/dev_tools/anaconda/envs/textsearch/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/star-kw/kangwei/code/text_search/examples/libriheavy/tools/recognize.py", line 365, in run
    decode_dataset(
  File "/star-kw/kangwei/code/text_search/examples/libriheavy/tools/recognize.py", line 287, in decode_dataset
    hyps, timestamps, scores = decode_one_batch(
  File "/star-kw/kangwei/code/text_search/examples/libriheavy/tools/recognize.py", line 184, in decode_one_batch
    encoder_out, encoder_out_lens = model.encoder(
  File "/star-kw/kangwei/dev_tools/anaconda/envs/textsearch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
RuntimeError: dims.value().size() == self->getMaybeRFactorDomain().size() INTERNAL ASSERT FAILED at "../third_party/nvfuser/csrc/parser.cpp":3399, please report a bug to PyTorch. 
```

This is the issue in my repo https://github.com/k2-fsa/text_search/issues/53, if you need to reproduce the issue, pls let me know, I can upload some testdata here.

### Versions

Collecting environment information...                                                                                                                          
PyTorch version: 2.0.1+cu117                                                                                                                                   
Is debug build: False                                                                                                                                          
CUDA used to build PyTorch: 11.7                                                                                                                               
ROCM used to build PyTorch: N/A                                                                                                                                
                                                                                                                                                               
OS: Ubuntu 18.04.5 LTS (x86_64)                                                                                                                                
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0                                                                                                               
Clang version: Could not collect                                                                                                                               
CMake version: version 3.27.2                                                                                                                                  
Libc version: glibc-2.27                                                                                                                                       
                                                                                                                                                               
Python version: 3.10.12 (main, Jul  5 2023, 18:54:27) [GCC 11.2.0] (64-bit runtime)                                                                            
Python platform: Linux-5.4.54-1.0.0.std7c.el7.2.x86_64-x86_64-with-glibc2.27                                                                                   
Is CUDA available: True                                                                                                                                        
CUDA runtime version: 10.1.243                                                                                                                                 
CUDA_MODULE_LOADING set to: LAZY                                                                                                                               
GPU models and configuration:                                                                                                                                  
GPU 0: Tesla V100-PCIE-32GB                                                                                                                                    
GPU 1: Tesla V100-PCIE-32GB                                                                                                                                    
GPU 2: Tesla V100-PCIE-32GB                                                                                                                                    
GPU 3: Tesla V100-PCIE-32GB                                                                                                                                    
GPU 4: Tesla V100-PCIE-32GB                                                                                                                                    
GPU 5: Tesla V100-PCIE-32GB                                                                                                                                    
GPU 6: Tesla V100-PCIE-32GB                                                                                                                                    
GPU 7: Tesla V100-PCIE-32GB                                                                                                                                    
                                                                                                                                                               
Nvidia driver version: 455.32.00 
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.2                      
HIP runtime version: N/A                                                        
MIOpen runtime version: N/A                                                     
Is XNNPACK available: True                                                      

CPU:                                                                            
Architecture:        x86_64                                                     
CPU op-mode(s):      32-bit, 64-bit                                             
Byte Order:          Little Endian                                              
CPU(s):              72                                                         
On-line CPU(s) list: 0-71
Thread(s) per core:  2
Core(s) per socket:  18
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
Stepping:            7
CPU MHz:             3299.999
BogoMIPS:            5200.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            25344K
NUMA node0 CPU(s):   0-17,36-53
NUMA node1 CPU(s):   18-35,54-71
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb
 rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg f
ma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_
l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 er
ms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_oc
cup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp_epp pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.25.2
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] triton==2.0.0
[conda] numpy                     1.25.2                   pypi_0    pypi
[conda] torch                     2.0.1                    pypi_0    pypi
[conda] torchaudio                2.0.2                    pypi_0    pypi
[conda] triton                    2.0.0                    pypi_0    pypi

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel