Repository: pytorch
Issue: scripted fasterRCNN model cannot be loaded with libtorch c++ API
Labels: oncall: jit,module: cpp

### 🐛 Describe the bug

I have a trained fasterRCNN model that I want to load via the libtorch C++ API for deployment. The model can be scripted using the following python code snippet successfully.
```python
# Sample code to reproduce the bug
import torch
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection import fasterrcnn_resnet50_fpn, FasterRCNN_ResNet50_FPN_Weights

weights = FasterRCNN_ResNet50_FPN_Weights.DEFAULT
model = fasterrcnn_resnet50_fpn(weights=weights)
in_features = model.roi_heads.box_predictor.cls_score.in_features
# replace the pre-trained head with a new one
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, 3)
# model.load_state_dict(torch.load('weights/tuned_weights.pth'))
model.eval()
script_module = torch.jit.script(model)
script_module.save("frcnn_torchscript_n3.pt")
```

Then I use the following C++ code to load it
```Cpp
#include <iostream>
#include <memory>
#include <torch/script.h>

int main(int argc, const char* argv[])
{
torch::jit::script::Module module;
    try {
        module = torch::jit::load(argv[1]);
        std::cout<<"SUCCESS"<<std::endl;
    }
    catch (const c10::Error& e) {
        std::cerr << "Error loading the model!\n";
        std::exit(EXIT_FAILURE);
    }
}
```
I get the following error
```bash
terminate called after throwing an instance of 'c10::Error'
  what():  isTuple()INTERNAL ASSERT FAILED at "../aten/src/ATen/core/ivalue_inl.h":1916, please report a bug to PyTorch. Expected Tuple but got String
Exception raised from toTupleRef at ../aten/src/ATen/core/ivalue_inl.h:1916 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f30a36070eb in /home/ash/Ash/libtorch-1.11.0+cu113/libtorch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f30a3602abe in /home/ash/Ash/libtorch-1.11.0+cu113/libtorch/lib/libc10.so)
frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x4e (0x7f30a360485e in /home/ash/Ash/libtorch-1.11.0+cu113/libtorch/lib/libc10.so)
frame #3: <unknown function> + 0x360de17 (0x7f3025757e17 in /home/ash/Ash/libtorch-1.11.0+cu113/libtorch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x360e075 (0x7f3025758075 in /home/ash/Ash/libtorch-1.11.0+cu113/libtorch/lib/libtorch_cpu.so)
frame #5: torch::jit::SourceRange::highlight(std::ostream&) const + 0x3d (0x7f3023399b6d in /home/ash/Ash/libtorch-1.11.0+cu113/libtorch/lib/libtorch_cpu.so)
frame #6: torch::jit::ErrorReport::what() const + 0x351 (0x7f302337cd51 in /home/ash/Ash/libtorch-1.11.0+cu113/libtorch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x41f58 (0x55a325061f58 in /home/ash/Ash/cmake_build/load_torch_model)
frame #8: <unknown function> + 0x1c222 (0x55a32503c222 in /home/ash/Ash/cmake_build/load_torch_model)
frame #9: <unknown function> + 0x1cbd3 (0x55a32503cbd3 in /home/ash/Ash/cmake_build/load_torch_model)
frame #10: <unknown function> + 0x1d46d (0x55a32503d46d in /home/ash/Ash/cmake_build/load_torch_model)
frame #11: <unknown function> + 0x29ac5 (0x55a325049ac5 in /home/ash/Ash/cmake_build/load_torch_model)
frame #12: <unknown function> + 0x4a279 (0x55a32506a279 in /home/ash/Ash/cmake_build/load_torch_model)
frame #13: <unknown function> + 0x42e63 (0x55a325062e63 in /home/ash/Ash/cmake_build/load_torch_model)
frame #14: <unknown function> + 0x282e4 (0x55a3250482e4 in /home/ash/Ash/cmake_build/load_torch_model)
frame #15: <unknown function> + 0xc3b6 (0x55a32502c3b6 in /home/ash/Ash/cmake_build/load_torch_model)
frame #16: <unknown function> + 0xba6f (0x55a32502ba6f in /home/ash/Ash/cmake_build/load_torch_model)
frame #17: __libc_start_main + 0xe7 (0x7f2fe8325c87 in /lib/x86_64-linux-gnu/libc.so.6)
frame #18: <unknown function> + 0xb4da (0x55a32502b4da in /home/ash/Ash/cmake_build/load_torch_model)
```

Then I tried loading my model with the latest [libtorch](https://pytorch.org/) (1.12.1, cuda 11.3) and I get the following error

```
C++ exception with description "
Unknown builtin op: torchvision::nms.
Could not find any similar ops to torchvision::nms. This op may not exist or may not be currently supported in TorchScript.
:
  File "code/__torch__/torchvision/ops/boxes.py", line 138
  _59 = __torch__.torchvision.extension._assert_has_ops
  _60 = _59()
  _61 = ops.torchvision.nms(boxes, scores, iou_threshold)
        ~~~~~~~~~~~~~~~~~~~ <--- HERE
  return _61
'nms' is being compiled since it was called from '_batched_nms_vanilla'
  File "/home/ash/anaconda3/envs/torch_latest/lib/python3.10/site-packages/torchvision/ops/boxes.py", line 109
    for class_id in torch.unique(idxs):
        curr_indices = torch.where(idxs == class_id)[0]
        curr_keep_indices = nms(boxes[curr_indices], scores[curr_indices], iou_threshold)
                            ~~~ <--- HERE
        keep_mask[curr_indices[curr_keep_indices]] = True
    keep_indices = torch.where(keep_mask)[0]
Serialized   File "code/__torch__/torchvision/ops/boxes.py", line 83
    _31 = torch.index(boxes, _30)
    _32 = annotate(List[Optional[Tensor]], [curr_indices])
    curr_keep_indices = __torch__.torchvision.ops.boxes.nms(_31, torch.index(scores, _32), iou_threshold, )
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    _33 = annotate(List[Optional[Tensor]], [curr_keep_indices])
    _34 = torch.index(curr_indices, _33)
'_batched_nms_vanilla' is being compiled since it was called from 'batched_nms'
Serialized   File "code/__torch__/torchvision/ops/boxes.py", line 35
    idxs: Tensor,
    iou_threshold: float) -> Tensor:
  _9 = __torch__.torchvision.ops.boxes._batched_nms_vanilla
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
  _10 = __torch__.torchvision.ops.boxes._batched_nms_coordinate_trick
  _11 = torch.numel(boxes)
'batched_nms' is being compiled since it was called from 'RegionProposalNetwork.filter_proposals'
Serialized   File "code/__torch__/torchvision/models/detection/rpn.py", line 72
    _11 = __torch__.torchvision.ops.boxes.clip_boxes_to_image
    _12 = __torch__.torchvision.ops.boxes.remove_small_boxes
    _13 = __torch__.torchvision.ops.boxes.batched_nms
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    num_images = (torch.size(proposals))[0]
    device = ops.prim.device(proposals)
'RegionProposalNetwork.filter_proposals' is being compiled since it was called from 'RegionProposalNetwork.forward'
  File "/home/ash/anaconda3/envs/torch_latest/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 376
        proposals = self.box_coder.decode(pred_bbox_deltas.detach(), anchors)
        proposals = proposals.view(num_images, -1, 4)
        boxes, scores = self.filter_proposals(proposals, objectness, images.image_sizes, num_anchors_per_level)
                        ~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    
        losses = {}
Serialized   File "code/__torch__/torchvision/models/detection/rpn.py", line 43
    proposals0 = torch.view(proposals, [num_images, -1, 4])
    image_sizes = images.image_sizes
    _8 = (self).filter_proposals(proposals0, objectness0, image_sizes, num_anchors_per_level, )
                                                                       ~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    boxes, scores, = _8
    losses = annotate(Dict[str, Tensor], {})
" thrown in the test body.

Process finished with exit code 1

```

### Versions

downloaded the libtorch-1.11.0+cu113 and also tried with 1.12.1 + cuda 11.3


PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.24.1
Libc version: glibc-2.27

Python version: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 9.1.85
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070 with Max-Q Design
Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.5.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] ros_numpy==0.0.3
[pip3] torch==1.12.1
[pip3] torchaudio==0.12.1
[pip3] torchvision==0.13.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0           py310h7f8727e_0  
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0  
[conda] mkl_random                1.2.2           py310h00e6091_0  
[conda] numpy                     1.23.1          py310h1794996_0  
[conda] numpy-base                1.23.1          py310hcba007f_0  
[conda] pytorch                   1.12.1          py3.10_cuda11.3_cudnn8.3.2_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.1              py310_cu113    pytorch
[conda] torchvision               0.13.1              py310_cu113    pytorch


cc @jbschlosser