Repository: pytorch
Issue: scatter_add_ 6000-times slower with int64 compared to int32
Labels: module: performance,module: docs,module: cuda,triaged,module: scatter & gather ops

## 🐛 Bug

`scatter_add_` is unreasonably slow on GPU for the `torch.long` data type. I understand this might be closely related to the hardware layout of GPUs, however, not getting a user warning or a mention in the docs is extremely irritating.

## To Reproduce

Steps to reproduce the behavior:

1. Run the below script and observe:
1. Observation for me:
```text
scatter_add_long:  0.6243471608590334
scatter_add:  0.00011378360912203789
scatter_long:  2.164090983569622e-05
```

```python
from time import perf_counter

import numpy as np
import torch

repeats = 100
p = [
    0.9,
    0.09,
    0.009,
    0.0009,
    0.0001,
    0.0,
    0.0,
    0.0,
    0.0,
    0.0,
    0.0,
    0.0,
    0.0,
    0.0,
    0.0,
    0.0,
    0.0,
]
n = len(p)
labels = torch.tensor(np.random.choice(n, size=200000, p=p)).to("cuda")

assert labels.dtype == torch.long

torch.cuda.synchronize()
t0 = perf_counter()
for _ in range(repeats):
    scatter_add_long_results = torch.zeros(
        n, dtype=labels.dtype, device=labels.device
    ).scatter_add_(0, labels, torch.ones_like(labels))
torch.cuda.synchronize()
t1 = perf_counter()
print("scatter_add_long: ", (t1 - t0) / repeats)

torch.cuda.synchronize()
t0 = perf_counter()
for _ in range(repeats):
    scatter_add_results = torch.zeros(
        n, dtype=torch.int, device=labels.device
    ).scatter_add_(0, labels, torch.ones_like(labels, dtype=torch.int))
torch.cuda.synchronize()
t1 = perf_counter()
print("scatter_add: ", (t1 - t0) / repeats)

torch.cuda.synchronize()
t0 = perf_counter()
for _ in range(repeats):
    scatter_long_results = torch.zeros(
        n, dtype=labels.dtype, device=labels.device
    ).scatter_(0, labels, torch.ones_like(labels))
torch.cuda.synchronize()
t1 = perf_counter()
print("scatter_long: ", (t1 - t0) / repeats)

assert (scatter_add_results == scatter_add_long_results).all(), (
    scatter_add_results,
    scatter_add_long_results,
)
assert (scatter_add_results.clip(max=1) == scatter_long_results).all(), (
    scatter_add_results,
    scatter_long_results,
)
```

## Expected behavior

I expect at least one of three things but ideally of course all three:
- documentation of this slow down (specifying the order of magnitude being 3-4) for `torch.long` values
- a user warning when a potential slow down might be arising due to the data type
- I understand only very roughly what the hardware reason for this could be, but still a slow down of 3-4 orders of magnitude seems far too big, so I expect there might also be a software fix involved reducing this slow down to more like only 1 order of magnitude.

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py).

```text
Collecting environment information...
PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.18.4

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 460.32.03
cuDNN version: /usr/local/cuda-10.0/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.7.0
[pip3] torch-cluster==1.5.9
[pip3] torch-geometric==1.7.0
[pip3] torch-scatter==2.0.6
[pip3] torch-sparse==0.6.9
[pip3] torch-spline-conv==1.2.1
[pip3] torchvision==0.8.2
[conda] Could not collect
```

cc @ngimel @brianjo @mruberry @VitalyFedyunin