Repository: pytorch
Issue: Pytorch 2.0: Detection models from torchvision don't work with onnx and tensorrt backends
Labels: module: onnx,triaged,oncall: pt2

### 🐛 Describe the bug

Detection models  (ssd, retinanet, RCNN-s) from torchvision don't work with onnx and tensorrt backends.

```
model = maskrcnn_resnet50_fpn_v2()
device = torch.device('cuda')

model.to(device).eval()
model = torch.compile(model, backend='onnxrt')

input_data = torch.randn(3, 224, 224)
with torch.no_grad():
    result = model([input_data.to(device)])
```

Error:
```
/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/jit/_check.py:172: UserWarning: The TorchScript type system doesn't support instance-level annotations on empty non-base types in `__init__`. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in `torch.jit.Attribute`.
  warnings.warn("The TorchScript type system doesn't support "
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 670, in call_user_compiler
    compiled_fn = compiler_fn(gm, self.fake_example_inputs())
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1055, in debug_wrapper
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 101, in wrapper
    return fn(model, inputs, **kwargs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/backends/onnxrt.py", line 51, in onnxrt
    return onnxrt(gm, example_inputs, filename=tmp.name)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 101, in wrapper
    return fn(model, inputs, **kwargs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/backends/onnxrt.py", line 66, in onnxrt
    torch.jit.script(gm),
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/jit/_script.py", line 1286, in script
    return torch.jit._recursive.create_script_module(
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/jit/_recursive.py", line 480, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/jit/_recursive.py", line 546, in create_script_module_impl
    create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/jit/_recursive.py", line 397, in create_methods_and_properties_from_stubs
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
RuntimeError: 
Arguments for call are not valid.
The following variants are available:
  
  aten::device(str a) -> Device:
  Argument a not provided.
  
  device(str type) -> Device:
  Keyword argument index unknown.

The original call is:
  File "<eval_with_key>.1", line 5
def forward(self, image : torch.Tensor):
    as_tensor = torch.as_tensor([0.485, 0.456, 0.406], dtype = torch.float32, device = device(type='cuda', index=0))
                                                                                       ~~~~~~ <--- HERE
    as_tensor_1 = torch.as_tensor([0.229, 0.224, 0.225], dtype = torch.float32, device = device(type='cuda', index=0))
    getitem = as_tensor[(slice(None, None, None), None, None)];  as_tensor = None


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/PycharmProjects/pytorch2test/main.py", line 55, in <module>
    benchmark(model, dtype='fp32', input_shape=(3, 224, 224), nruns=1000)
  File "/home/user/PycharmProjects/pytorch2test/main.py", line 25, in benchmark
    features = model(input_data)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
    return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
    return fn(*args, **kwargs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 83, in forward
    images, targets = self.transform(images, targets)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torchvision/models/detection/transform.py", line 129, in forward
    image = self.normalize(image)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 330, in catch_errors
    return callback(frame, cache_size, hooks)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
    result = inner_convert(frame, cache_size, hooks)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
    return fn(*args, **kwargs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
    return _compile(
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
    out_code = transform_code_object(code, transform)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 361, in transform_code_object
    transformations(instructions, code_options)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
    tracer.run()
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1715, in run
    super().run()
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 564, in run
    and self.step()
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 527, in step
    getattr(self, inst.opname)(inst)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1781, in RETURN_VALUE
    self.output.compile_subgraph(self)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 517, in compile_subgraph
    self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 588, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/user/anaconda3/envs/pytorch2test/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 675, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: onnxrt raised RuntimeError: 
Arguments for call are not valid.
The following variants are available:
  
  aten::device(str a) -> Device:
  Argument a not provided.
  
  device(str type) -> Device:
  Keyword argument index unknown.

The original call is:
  File "<eval_with_key>.1", line 5
def forward(self, image : torch.Tensor):
    as_tensor = torch.as_tensor([0.485, 0.456, 0.406], dtype = torch.float32, device = device(type='cuda', index=0))
                                                                                       ~~~~~~ <--- HERE
    as_tensor_1 = torch.as_tensor([0.229, 0.224, 0.225], dtype = torch.float32, device = device(type='cuda', index=0))
    getitem = as_tensor[(slice(None, None, None), None, None)];  as_tensor = None


Set torch._dynamo.config.verbose=True for more information


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True

```

### Versions

PyTorch version: 2.0.0.dev20230204+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.25.2
Libc version: glibc-2.31

Python version: 3.10.8 (main, Nov 24 2022, 14:13:03) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.4
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-triton==2.0.0+0d7e753227
[pip3] torch==2.0.0.dev20230204+cu117
[pip3] torch2trt==0.4.0
[pip3] torchtriton==2.0.0+0d7e753227
[pip3] torchvision==0.15.0.dev20230204+cu117
[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] pytorch-triton            2.0.0+0d7e753227          pypi_0    pypi
[conda] torch                     2.0.0.dev20230204+cu117          pypi_0    pypi
[conda] torch2trt                 0.4.0                    pypi_0    pypi
[conda] torchtriton               2.0.0+0d7e753227          pypi_0    pypi
[conda] torchvision               0.15.0.dev20230204+cu117          pypi_0    pypi


cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh