Repository: pytorch
Issue: Inconsistent computation of gradient in MaxUnPooling
Labels: module: autograd,triaged,module: determinism,actionable,module: correctness (silent)

### 🐛 Describe the bug

Hey

I think there is a inconsistent definition how MaxUnPool and it's gradient are computed. Here is an example code.

```python
import torch

torch.manual_seed(123)

A = torch.rand(1, 1, 9, 9)

_, I = torch.nn.MaxPool2d(3, 1, return_indices=True)(A)
print("Indices", I)

B = torch.arange(I.numel(), 0, -1).to(torch.float).view(I.shape).detach()
B.requires_grad = True
print("MaxUnPool Input", B)

C = torch.nn.MaxUnpool2d(3, 1)(B, I)
print("MaxUnPool Output", C)

D = C * torch.arange(C.numel()).to(torch.float).view(C.shape)

# now compute the gradient
E = D.sum()
E.backward()

print("MaxUnPool Gradient", B.grad)
```

The output is:
```
Indices tensor([[[[20, 20, 20, 23, 23, 23, 17],
          [28, 30, 30, 23, 23, 23, 34],
          [28, 30, 30, 23, 23, 23, 43],
          [28, 48, 48, 48, 49, 43, 43],
          [47, 48, 48, 48, 49, 43, 43],
          [47, 48, 48, 48, 49, 70, 70],
          [74, 57, 57, 57, 69, 70, 70]]]])
MaxUnPool Input tensor([[[[49., 48., 47., 46., 45., 44., 43.],
          [42., 41., 40., 39., 38., 37., 36.],
          [35., 34., 33., 32., 31., 30., 29.],
          [28., 27., 26., 25., 24., 23., 22.],
          [21., 20., 19., 18., 17., 16., 15.],
          [14., 13., 12., 11., 10.,  9.,  8.],
          [ 7.,  6.,  5.,  4.,  3.,  2.,  1.]]]], requires_grad=True)
MaxUnPool Output tensor([[[[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
          [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0., 43.],
          [ 0.,  0., 47.,  0.,  0., 37.,  0.,  0.,  0.],
          [ 0., 35.,  0., 33.,  0.,  0.,  0., 36.,  0.],
          [ 0.,  0.,  0.,  0.,  0.,  0.,  0., 15.,  0.],
          [ 0.,  0., 14., 18., 17.,  0.,  0.,  0.,  0.],
          [ 0.,  0.,  0.,  5.,  0.,  0.,  0.,  0.,  0.],
          [ 0.,  0.,  0.,  0.,  0.,  0.,  3.,  8.,  0.],
          [ 0.,  0.,  7.,  0.,  0.,  0.,  0.,  0.,  0.]]]],
       grad_fn=<MaxUnpool2DBackward0>)
MaxUnPool Gradient tensor([[[[20., 20., 20., 23., 23., 23., 17.],
          [28., 30., 30., 23., 23., 23., 34.],
          [28., 30., 30., 23., 23., 23., 43.],
          [28., 48., 48., 48., 49., 43., 43.],
          [47., 48., 48., 48., 49., 43., 43.],
          [47., 48., 48., 48., 49., 70., 70.],
          [74., 57., 57., 57., 69., 70., 70.]]]])
```

In "Indices" we can see that the index 20 is used 3 times.

Now in "MaxUnPool Output" we see that the index 20 does only use the very last value 47 in this case. So I would expect the gradient for these three to be [0, 0, 20].

I put in a multiplication, so that in the gradient we get the actual index where the value was taken from. As we can see, the gradient for these three values is instead [20, 20, 20].

To wrap this up. To my understanding MaxUnPool computes the following during the forward pass:

```c++
for(int batch ...)
	for(int channel ...)
		for(int index ...)
			output[batch, channel, indices[index]] = input[batch, channel, index];
```

However, when we look at the gradient, all get the value propagated, which is inconsistent with the forward pass, where only the last gets used.

This seems to be inconsistent to me. So I think either the forward pass needs to use a ```+=``` instead of ```=```, or the backward pass needs to only propagate the value to the last occurrence of the index.

Best

### Versions

Collecting environment information...
PyTorch version: 1.12.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 10.3.0
Clang version: Could not collect
CMake version: version 3.23.1
Libc version: glibc-2.17

Python version: 3.7.12 (default, Feb  6 2022, 20:29:18)  [GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] (64-bit runtime)
Python platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
Is CUDA available: False
CUDA runtime version: 11.3.109
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.12.0
[pip3] torchmetrics==0.9.1
[pip3] torchvision==0.13.0
[conda] Could not collect

cc @ezyang @gchanan @zou3519 @albanD @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @mruberry @kurtamohler