Repository: pytorch
Issue: `torch.scatter` doesn't fail correctly on CUDA (memory overlap)
Labels: module: cuda,module: tests,triaged,module: scatter & gather ops,module: structured kernels

## 🐛 Bug

While working on https://github.com/pytorch/pytorch/pull/63312 with @ysiraichi, we realized that `torch.scatter` doesn't fail for CUDA devices when passed with the same tensor as that of the index, but of a size where resizing is necessary.

An error like:

```
RuntimeError: unsupported operation: some elements of the input tensor and the written-to tensor refer to a single memory location. Please clone() the tensor before performing the operation.
```

is expected, but is not raised with CUDA tensors on `scatter` op. This is most probably the case with other ops as well, which were ported to structured kernels recently.

**Note**: This also points that the testing for these ops should be improved, as this should have been caught by the tests as it got caught for `gather`. Looks like there aren't enough tests for `scatter` and equivalent ops which have this bug.

## To Reproduce

Doesn't fail on CUDA:

```python
>>> output = torch.scatter(index.clone(), 0, index[1:], index.clone()[1:], out=index[:1])
<stdin>:1: UserWarning: An output with one or more elements was resized since it had shape [1], which does not match the required output shape [4].This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at  ../aten/src/ATen/native/Resize.cpp:16.)
>>> output
tensor([0, 1, 2, 3], device='cuda:0')
```

Though it fails correctly on CPU:

```python
>>> index = torch.tensor([0, 1, 3, 2], device='cpu')
>>> output = torch.scatter(index.clone(), 0, index[1:], index.clone()[1:], out=index[:1])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: unsupported operation: some elements of the input tensor and the written-to tensor refer to a single memory location. Please clone() the tensor before performing the operation.
```

In case the `out` tensor is of the same size as `index` tensor, it does fail correctly with CUDA tensors.

```python
>>> index = torch.tensor([0, 1, 3, 2], device=device)
>>> output = torch.scatter(index.clone(), 0, index[1:], index.clone()[1:], out=index[1:])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: unsupported operation: some elements of the input tensor and the written-to tensor refer to a single memory location. Please clone() the tensor before performing the operation
```

## Expected behavior

The behavior should be consistent on both CPU and CUDA devices. IMO, the error should be raised but I may be wrong.

## Environment

This error will be reproducible on the master branch. 

## Additional context

From what we discussed offline, here is what I and @ysiraichi think could be the reason:

* We do memory overlap tests before resizing output tensor: https://github.com/pytorch/pytorch/blob/f5d585391d13287250e85cbd55a17c5e0b8ac2a8/aten/src/ATen/native/TensorAdvancedIndexing.cpp#L99-L104
* Resizing is done here after the memory overlap tests are done: https://github.com/pytorch/pytorch/blob/f5d585391d13287250e85cbd55a17c5e0b8ac2a8/aten/src/ATen/native/TensorAdvancedIndexing.cpp#L107.

**Why does it fail with CPU tensors?**

* By default, `check_mem_overlap_` is set to `true` in `TensorIteratorConfig`: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/TensorIterator.h#L669.
* When called inside the CPU kernel for `scatter` (common for `scatter` and `gather`), memory overlap checks are done again (which will fail now since resizing has been done, and there is a memory overlap): https://github.com/pytorch/pytorch/blob/f5d585391d13287250e85cbd55a17c5e0b8ac2a8/aten/src/ATen/native/cpu/ScatterGatherKernel.cpp#L123

**For CUDA**, this (through `set_check_mem_overlap`) is set to `false`: https://github.com/pytorch/pytorch/blob/f5d585391d13287250e85cbd55a17c5e0b8ac2a8/aten/src/ATen/native/cuda/ScatterGatherKernel.cu#L170-L177.

**One possible solution** is to resize the output (if defined) in the meta function itself, then do the memory overlap checks, something like this: https://github.com/pytorch/pytorch/pull/63312/files#diff-c4e35e76279419d9edda979a995412ee494fad3a97d3fb59486a1f0e326d48beR91-R97.

cc: @ezyang @ysiraichi 

cc @ngimel @mruberry @VitalyFedyunin @walterddr