Repository: pytorch
Issue: `torch.compile` mode and eager mode are different in generating tensor API behavior
Labels: triaged,oncall: pt2,module: dynamo

### 🐛 Describe the bug

The `torch.compile` mode and the eager mode differ in generating tensor API behavior, which usually results in the eager mode running without problems, while the `torch.compile` mode does not run

## BUG Case 1 
Generating a list of tensors into a single tensor using an API like `torch.tensor` or `np.array` under `torch.compile` raises ` ValueError:only one element tensors can be converted to Python scalars` error, but not eager mode

### BUG code
```python
import torch
import numpy as np

def _get_rot(self, rot):
    return torch.Tensor(
        [
            [np.cos(rot), np.sin(rot)],
            [-np.sin(rot), np.cos(rot)],
        ]
    )

def make(*args, **kwargs):
    @torch.compile(backend='inductor')
    def f():
        a = _get_rot(*args, **kwargs)
    return f

rot = (-0.3925, 0.3925)
make(rot)()
```
No error will be reported if the following code is modified
```python
def _get_rot(rot):
    return torch.Tensor(
        np.stack([
            np.stack([np.cos(rot), np.sin(rot)]),
            np.stack([-np.sin(rot), np.cos(rot)]),
        ])
    )
```

## BUG Case 2
You can only use `torch.tensor` to generate tensors. When use `FloatTensor` etc., will raise error.

### BUG code
```python
import torch
import numpy as np

def forward_each_level(anchors, stride):
    stride_h = stride
    stride_w = stride
    scaled_anchors = [
        (a_w / stride_w, a_h / stride_h) for a_w, a_h in anchors
    ]
    
    anchor_w = torch.FloatTensor(scaled_anchors)

def make(*args, **kwargs):
    @torch.compile(backend='inductor')
    def f():
        forward_each_level(*args, **kwargs)
    return f

anchors = [(116, 90), (156, 198), (373, 326)]
stride = 32
make(anchors, stride)()
```
For example, `torch.cuda.LongTensor`, `torch.LongTensor`, `torch.cuda.FloatTensor` etc. also has problems

## BUG Case 3
`torch.Tensor.new_tensor` can be used, but the `device` information cannot be copied in `torch.compile` mode

### BUG code
```python
import torch
from torch import Tensor, nn
from typing import Dict, List, Tuple
import numpy as np


def point_sampling(
    reference_points: Tensor,
    pc_range: List[float],
    img_metas: Dict,
    im_shape: Tuple[int],
) -> Tuple[Tensor, Tensor]:
    """Sample img points."""

    reference_points = reference_points.to(torch.float32)
    ego2img = img_metas["ego2img"]

    ## BUG here, change to `ego2img = torch.tensor(ego2img, device=reference_points.device) ` will OK
    ego2img = np.asarray(ego2img)
    ego2img = reference_points.new_tensor(ego2img)  # (B, N, 4, 4)
    
    reference_points = reference_points.clone()

    reference_points[..., 0:1] = (
        reference_points[..., 0:1] * (pc_range[3] - pc_range[0])
        + pc_range[0]
    )
    reference_points[..., 1:2] = (
        reference_points[..., 1:2] * (pc_range[4] - pc_range[1])
        + pc_range[1]
    )
    reference_points[..., 2:3] = (
        reference_points[..., 2:3] * (pc_range[5] - pc_range[2])
        + pc_range[2]
    )

    reference_points = torch.cat(
        (reference_points, torch.ones_like(reference_points[..., :1])), -1
    )

    reference_points = reference_points.permute(1, 0, 2, 3)
    D, B, num_query = reference_points.size()[:3]
    num_cam = ego2img.size(1)

    reference_points = (
        reference_points.view(D, B, 1, num_query, 4)
        .repeat(1, 1, num_cam, 1, 1)
        .unsqueeze(-1)
    )

    ego2img = ego2img.view(1, B, num_cam, 1, 4, 4).repeat(
        D, 1, 1, num_query, 1, 1
    )

    ## Error here, when in `torch.compile` mode ego2img in `cpu`, but eager mode is `cuda`
    reference_points_cam = torch.matmul(
        ego2img.to(torch.float32), reference_points.to(torch.float32)
    ).squeeze(-1)

    eps = 1e-5

    bev_mask = reference_points_cam[..., 2:3] > eps
    reference_points_cam = reference_points_cam[..., 0:2] / torch.maximum(
        reference_points_cam[..., 2:3],
        torch.ones_like(reference_points_cam[..., 2:3]) * eps,
    )

    reference_points_cam[..., 0] /= im_shape[1]
    reference_points_cam[..., 1] /= im_shape[0]

    bev_mask = (
        bev_mask
        & (reference_points_cam[..., 1:2] > 0.0)
        & (reference_points_cam[..., 1:2] < 1.0)
        & (reference_points_cam[..., 0:1] < 1.0)
        & (reference_points_cam[..., 0:1] > 0.0)
    )

    bev_mask = torch.nan_to_num(bev_mask)

    reference_points_cam = torch.clamp(
        reference_points_cam, min=-2.1, max=2.1
    )
    reference_points_cam = reference_points_cam.permute(2, 1, 3, 0, 4)
    bev_mask = bev_mask.permute(2, 1, 3, 0, 4).squeeze(-1)

    return reference_points_cam, bev_mask

def make(*args, **kwargs):
    @torch.compile(backend='inductor')
    def f():
        a = point_sampling(*args, **kwargs)
    return f

for i in range(16):
    reference_points = torch.randn(2, 2, 4, 3).cuda()
    pc_range = [-51.2, -51.2, -5.0, 51.2, 51.2, 3.0]
    img_metas = {"ego2img": torch.randn(2, 2, 4, 4).tolist()}
    im_shape = (256, 256)
    make(reference_points, pc_range, img_metas, im_shape)()
```

### Versions

# Torch.compile Code Case Summary

## Environment
| Env                     | Version                |
| -----------             | -----------            |
| Python                  | 3.8.18                 |
| CUDA                    | 12.2                   |
| Torch                   | 2.1.0+cu118            |
| Torchvision             | 0.16.0+cu118           |
| GPU                     | 2 * RTX4090            |
| CPU                     | Intel-i7-12700KF       |
| Ubuntu                  | 22.04                  |

cc @ezyang @anijain2305 @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng