Repository: pytorch
Issue: Indexing assignment can have no effect on CUDA with deterministic algorithms
Labels: high priority,triaged,module: advanced indexing,module: determinism

### 🐛 Describe the bug

On CUDA with `use_deterministic_algorithms(True)`, advanced indexing assignment has no effect on target tensors with more than one effective dimension when the source tensor has one dimension.


## To reproduce

```python
import torch

torch.use_deterministic_algorithms(True)
device = 'cuda'

# Snippet shows the case for shape (4, 2)
# Can also be (n, 1, 2), (n, 1, 1, 3), etc.
# While (n, 1), (n, 1, 1), etc. work as intended
n = 4  # First dim size
d = 2  # Number of dims
m = 2  # Last dim size

a = torch.zeros((n, *(1, )*(d-2), m), device=device)  # Target
b = torch.ones(n, device=device)                      # Source
print('Full shape:          ', a.shape)

# Assign to all rows at subsequent 0-indices (e.g. first column)
idx_tuple = (torch.arange(n), *(torch.tensor(0), )*(d-1))
print('Assigning to indices:', idx_tuple)

a[idx_tuple] = b

print('Sum of first column: ', a[idx_tuple].sum().item())
print('Sum of source data:  ', b.sum().item())
```

## Output
```
Full shape:           torch.Size([4, 2])
Assigning to indices: (tensor([0, 1, 2, 3]), tensor(0))
Sum of first column:  0.0
Sum of source data:   4.0
```


## Expected behaviour

Output should be the same as on CUDA with `use_deterministic_algorithms(False)` and CPU:
```
Full shape:           torch.Size([4, 2])
Assigning to indices: (tensor([0, 1, 2, 3]), tensor(0))
Sum of first column:  4.0
Sum of source data:   4.0
```

## Additional context

When `a[idx_tuple] = b` is replaced with an explicit call to `a.index_put_(idx_tuple, b)`, the bug does not present itself, despite, I would assume, deferring to the same underlying calls.

For completeness, the bug also does not happen with simpler indexing, e.g. when `a[idx_tuple] = b` is replaced with:
```python
for j, i in enumerate(idx_tuple[0]):
    a[(i, ) + idx_tuple[1:]] = b[j]
```

Interestingly, if the source tensor has more than one effective dimension as well, the bug also does not occur.

To confirm:

```python
torch.use_deterministic_algorithms(True)
device = 'cuda'

n = 4
d = 2
m = 2

a = torch.zeros((n, *(1, )*(d-2), m), device=device)
b = torch.ones((n, m), device=device)
print('Full shape:          ', a.shape)

# Copy (n, m) tensor with 0-indices in between
idx_tuple = (torch.arange(n), *(torch.tensor(0), )*(d-2))
print('Assigning to indices:', idx_tuple)

a[idx_tuple] = b

print('Sum of first column: ', a[idx_tuple].sum().item())
print('Sum of source data:  ', b.sum().item())
```

Note that changing the source shape from (n, m) to (n, 1) and attempting to do the same results in a known `INTERNAL_ASSERT_FAILED`. See comment 2 below.

### Versions

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.12 (default, Oct 12 2021, 13:49:34)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-40-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3070 Laptop GPU
Nvidia driver version: 510.60.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.11.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.1            py38hd3c417c_0
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] numpy                     1.21.2           py38h20f2e39_0
[conda] numpy-base                1.21.2           py38h79a1101_0
[conda] pytorch                   1.11.0           py3.8_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0              cuda    pytorch

cc @ezyang @gchanan @zou3519 @kadeng @mruberry @kurtamohler