Repository: pytorch
Issue: Unexpected behavior of "to" method inside a torch.jit.script decorated function
Labels: oncall: jit,days,TSRootCause:DefaultTypes,TSUsability

## 🐛 Bug

Hi, I was doing some work with `@torch.jit.script` decorator and came upon this strange bug which changed the value (by changing the data type of the tensor) of the resulting values I obtained with the decorated function.

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Here is a minimal working example:

```py
import torch

print("Torch Version: ", torch.__version__)

var_on_cuda = torch.Tensor([0.0]).cuda()
b = torch.Tensor([0.5])


def success_case(var_on_cuda, b):
    print('\nSuccess case:')
    print("\tb before: ", b)
    print("\tget_device: ", var_on_cuda.get_device())
    b = b.to(var_on_cuda.get_device())
    print("\tb after: ", b)
    # do some calculations with b


@torch.jit.script
def fail_case(var_on_cuda, b):
    print('\nFail case:')
    print("\tb before: ", b)
    print("\tget_device: ", var_on_cuda.get_device())
    b = b.to(var_on_cuda.get_device())
    print("\tb after: ", b)
    # do some calculations with b


success_case(var_on_cuda, b)
fail_case(var_on_cuda, b)
```

And this is the output I get:

```bash
Torch Version:  1.6.0

Success case:
        b before:  tensor([0.5000])
        get_device:  0
        b after:  tensor([0.5000], device='cuda:0')

Fail case:
        b before:   0.5000
[ CPUFloatType{1} ]
        get_device:  0
        b after:   0
[ CPUByteType{1} ]
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

`b` getting transferred to the same GPU on which the `var_on_cuda` is, not getting converted to a `ByteTensor`.

<!-- A clear and concise description of what you expected to happen. -->

## Environment

```bash
PyTorch version: 1.6.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080
Nvidia driver version: 450.51.05
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.6.0
[pip3] torchvision==0.7.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] mkl                       2020.1                      217  
[conda] mkl-service               2.3.0            py37he904b0f_0  
[conda] mkl_fft                   1.1.0            py37h23d657b_0  
[conda] mkl_random                1.1.1            py37h0da4684_0    conda-forge
[conda] numpy                     1.19.1           py37hbc911f0_0  
[conda] numpy-base                1.19.1           py37hfa32c7d_0  
[conda] pytorch                   1.6.0           py3.7_cuda10.2.89_cudnn7.6.5_0    pytorch
[conda] torchvision               0.7.0                py37_cu102    pytorch
```


cc @ezyang @gchanan @zou3519 @suo @gmagogsfm