Repository: pytorch
Issue: Reduction for `torch.int8` is super slow on CUDA
Labels: module: performance,module: cuda,triaged,module: TensorIterator

## 🐛 Bug

Reduction for `torch.int8` is super slow on CUDA

## To Reproduce

```python
import torch
print(torch.__version__)
print()

for i in range(1000):
    torch.arange(10000, device='cuda')

def benchmark(dtype, i):
    numel = 2 ** i
    a = torch.zeros(numel, device='cuda', dtype=dtype)
    torch.cuda.synchronize()
    %timeit a.sum(); torch.cuda.synchronize()

for dtype in [torch.int8, torch.half, torch.float, torch.double]:
    print(dtype)
    for i in range(18, 30):
        benchmark(dtype, i)
    print()
```

gets

```
1.6.0a0+2f5b523

torch.int8
24.1 µs ± 76.7 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
25 µs ± 367 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
52.8 µs ± 67.1 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
88.4 µs ± 183 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
171 µs ± 11.5 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
296 µs ± 6.06 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
568 µs ± 1.09 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.11 ms ± 752 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
2.22 ms ± 22.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
4.55 ms ± 192 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
9.34 ms ± 1.39 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
20.4 ms ± 200 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

torch.float16
16.7 µs ± 336 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
16.7 µs ± 30.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
17 µs ± 48.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
20.1 µs ± 193 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
34.2 µs ± 49.2 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
50.6 µs ± 122 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
81.9 µs ± 784 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
151 µs ± 3.29 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
277 µs ± 9.18 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
537 µs ± 17.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.13 ms ± 35.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
2.11 ms ± 45.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

torch.float32
17.5 µs ± 721 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
17.5 µs ± 893 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
20.6 µs ± 523 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
36.9 µs ± 4.17 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
47.7 µs ± 797 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
75.6 µs ± 381 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
148 µs ± 7.8 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
262 µs ± 16.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
496 µs ± 13.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.1 ms ± 29.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
2.23 ms ± 56.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
4.05 ms ± 196 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

torch.float64
23.5 µs ± 331 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
23.5 µs ± 1.19 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
38.3 µs ± 1.7 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
57.4 µs ± 1.92 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
88.2 µs ± 2.57 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
151 µs ± 3.16 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
281 µs ± 6.73 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
507 µs ± 21.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.06 ms ± 30 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
2.03 ms ± 119 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
3.77 ms ± 31.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
7.59 ms ± 51.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

## Environment

```
PyTorch version: 1.6.0a0+2f5b523
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Arch Linux
GCC version: (Arch Linux 9.3.0-1) 9.3.0
CMake version: version 3.17.1

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 10.2.89
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 440.82
cuDNN version: /usr/lib/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.18.1
[pip3] torch==1.6.0a0+e5746ee
[pip3] torchani==1.3.dev26+g5b3d892.d20200321
[pip3] torchvision==0.6.0+cu101
[conda] Could not collect
```

cc @ngimel @VitalyFedyunin