Repository: pytorch
Issue: pack_sequence() always fail after set_default_tensor_type to CUDA
Labels: module: rnn,triaged

### 🐛 Describe the bug

It seems that after `torch.set_default_tensor_type('torch.cuda.FloatTensor')`, certain functions does not work, for example, `torch.nn.utils.rnn.pack_sequence`.
```
>>>pack_sequence([torch.tensor([1,2,3]), torch.tensor([4,5])])
Out[1]: PackedSequence(data=tensor([1, 4, 2, 5, 3]), batch_sizes=tensor([2, 2, 1]), sorted_indices=None, unsorted_indices=None)
pack_sequence([torch.tensor([1,2,3]).cuda(), torch.tensor([4,5]).cuda()])
Out[2]: PackedSequence(data=tensor([1, 4, 2, 5, 3], device='cuda:0'), batch_sizes=tensor([2, 2, 1]), sorted_indices=None, unsorted_indices=None)
pack_sequence([torch.tensor([1,2,3]).cuda(0), torch.tensor([4,5]).cuda(0)])
Out[3]: PackedSequence(data=tensor([1, 4, 2, 5, 3], device='cuda:0'), batch_sizes=tensor([2, 2, 1]), sorted_indices=None, unsorted_indices=None)
torch.set_default_tensor_type('torch.cuda.FloatTensor')
pack_sequence([torch.tensor([1,2,3]), torch.tensor([4,5])])
Traceback (most recent call last):
  File "/home/xuancong/anaconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-2a6bbd446608>", line 1, in <module>
    pack_sequence([torch.tensor([1,2,3]), torch.tensor([4,5])])
  File "/home/xuancong/anaconda3/lib/python3.8/site-packages/torch/nn/utils/rnn.py", line 484, in pack_sequence
    return pack_padded_sequence(pad_sequence(sequences), lengths, enforce_sorted=enforce_sorted)
  File "/home/xuancong/anaconda3/lib/python3.8/site-packages/torch/nn/utils/rnn.py", line 262, in pack_padded_sequence
    _VF._pack_padded_sequence(input, lengths, batch_first)
RuntimeError: 'lengths' argument should be a 1D CPU int64 tensor, but got 1D cuda:0 Long tensor
pack_sequence([torch.tensor([1,2,3],device='cpu'), torch.tensor([4,5],device='cpu')])
Traceback (most recent call last):
  File "/home/xuancong/anaconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-4b903e3514ab>", line 1, in <module>
    pack_sequence([torch.tensor([1,2,3],device='cpu'), torch.tensor([4,5],device='cpu')])
  File "/home/xuancong/anaconda3/lib/python3.8/site-packages/torch/nn/utils/rnn.py", line 484, in pack_sequence
    return pack_padded_sequence(pad_sequence(sequences), lengths, enforce_sorted=enforce_sorted)
  File "/home/xuancong/anaconda3/lib/python3.8/site-packages/torch/nn/utils/rnn.py", line 262, in pack_padded_sequence
    _VF._pack_padded_sequence(input, lengths, batch_first)
RuntimeError: 'lengths' argument should be a 1D CPU int64 tensor, but got 1D cuda:0 Long tensor
```

From the error messages above, it seems that each `torch.tensor` has a field called `lengths` which is usually stored in CPU. `torch.set_default_tensor_type('torch.cuda.FloatTensor')` will cause this field to be stored into GPU and then pack_sequence() will always fail even if I put `device="cpu"` in the tensor creation function.

### Versions

PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 11 (bullseye) (x86_64)
GCC version: (Ubuntu 10.3.0-1ubuntu1~20.04) 10.3.0
Clang version: Could not collect
CMake version: version 3.20.4
Libc version: glibc-2.31

Python version: 3.8.2 (default, Mar 26 2020, 15:53:00)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.15.40-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1050 Ti with Max-Q Design
Nvidia driver version: 520.56.06
cuDNN version: Probably one of the following:
/usr/local/cuda-11.4/targets/x86_64-linux/lib/libcudnn.so.8.2.2
/usr/local/cuda-11.4/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.2.2
/usr/local/cuda-11.4/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.2.2
/usr/local/cuda-11.4/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.2.2
/usr/local/cuda-11.4/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.2.2
/usr/local/cuda-11.4/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.2.2
/usr/local/cuda-11.4/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.2.2
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.19.5
[pip3] numpydoc==1.1.0
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0
[pip3] torchvision==0.14.0
[conda] blas                      1.0                         mkl  
[conda] mkl                       2020.4                intel_304    intel
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1  
[conda] torch                     1.13.0                   pypi_0    pypi
[conda] torchaudio                0.13.0                   pypi_0    pypi
[conda] torchvision               0.14.0                   pypi_0    pypi


cc @zou3519