Repository: pytorch
Issue: A backward bug of dtensor seems to be caused by new_empty_strided
Labels: high priority,oncall: distributed,triaged,module: dtensor

### 🐛 Describe the bug

```python
device_mesh = DeviceMesh("cpu", [1,2,3,4])
local_tensor = torch.randn(8,8, requires_grad=True, device="cpu")
my_dtensor = distribute_tensor(local_tensor, device_mesh, [Shard(0)])
# bug happen when backward()
my_dtensor.to_local().sum().backward()
```

```
Traceback (most recent call last):
  File "/home/hzh/mytest/test_dist.py", line 101, in main_worker
    my_dtensor.to_local().sum().backward()
  File "/root/miniconda3/envs/hzh/lib/python3.9/site-packages/torch/_tensor.py", line 491, in backward
    torch.autograd.backward(
  File "/root/miniconda3/envs/hzh/lib/python3.9/site-packages/torch/autograd/__init__.py", line 250, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/root/miniconda3/envs/hzh/lib/python3.9/site-packages/torch/distributed/_tensor/api.py", line 258, in __torch_dispatch__
    res = op_dispatch.operator_dispatch(
  File "/root/miniconda3/envs/hzh/lib/python3.9/site-packages/torch/distributed/_tensor/dispatch.py", line 114, in operator_dispatch
    out, _, _ = _operator_dispatch(op_call, args, kwargs, sharding_propagator)
  File "/root/miniconda3/envs/hzh/lib/python3.9/site-packages/torch/distributed/_tensor/dispatch.py", line 254, in _operator_dispatch
    local_results = op_call(*local_tensor_args, **local_tensor_kwargs)
  File "/root/miniconda3/envs/hzh/lib/python3.9/site-packages/torch/_ops.py", line 435, in __call__
    return self._op(*args, **kwargs or {})
RuntimeError: The size of tensor a (8) must match the size of tensor b (2) at non-singleton dimension 0
```

this bug is directly caused here by the size check in operator `copy_` during the process of backward, and can be traced back to the previous calling of operator `new_empty_strided`.
https://github.com/pytorch/pytorch/blob/f13101640f548f8fa139c03dfa6711677278c391/torch/csrc/autograd/utils/grad_layout_contract.h#L65-L70
https://github.com/pytorch/pytorch/blob/a506d0ad8f21dd4090594c0aaca62518a5438081/aten/src/ATen/native/native_functions.yaml#L2317
When calling `new_empty_strided` with `self` to be a dtensor, it will be dispatched into `Dtensor.__torch_dispatch__`. Here, the paramater `size` and `stride` will be directly used to create a local tensor which is not i want. 
https://github.com/pytorch/pytorch/blob/a506d0ad8f21dd4090594c0aaca62518a5438081/torch/distributed/_tensor/dispatch.py#L128-L146
For the example above, `self` is a dtensor with `placements = [Shard(0)]`, `shape = [8, 8]` and `_local_tensor.shape = [2, 8]`. When i call `new_empty_strided(self, [8, 8], [8, 1])`, i want a dtensor with `shape = [8, 8]` and `_local_tensor.shape = [2, 8]` just like `self`. But, in fact, it return a local tensor with `shape = [8, 8]` after `op_call`, and finally return a **self-contradictory** dtensor with `placements = [Shard(0)]`, `shape = [8, 8]` and `_local_tensor.shape = [8, 8]` which is generated by simply calling the method `Dtensor.__new__()`.
https://github.com/pytorch/pytorch/blob/a506d0ad8f21dd4090594c0aaca62518a5438081/torch/distributed/_tensor/dispatch.py#L265

### Versions

```
PyTorch version: 2.1.0a0+git849fbc6
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 7.5.0
Clang version: Could not collect
CMake version: version 3.26.4
Libc version: glibc-2.17

Python version: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03)  [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-3.10.0-x86_64-with-glibc2.17
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.25.2
[pip3] torch==2.1.0a0+git849fbc6
```

cc @ezyang @gchanan @zou3519 @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu