Repository: pytorch
Issue: torch.nn.CrossEntropyLoss with "reduction" sum/mean is not deterministic on segmentation outputs / labels
Labels: module: cuda,triaged

## 🐛 Bug
torch.nn.CrossEntropyLoss doesn't output deterministic results on segmentation outputs / labels, when using reduction other than 'none'.
Happens only on GPU. CPU does give a consistent behavior. 

## To Reproduce
```
import numpy as np
import torch

outputs = np.random.rand(16, 1, 256, 256)
outputs = np.hstack((outputs, 1.0 - outputs))
targets = np.random.randint(2, size=(16, 256, 256))

seed = 0
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

for reduction in ['none', 'sum', 'mean']:
    print(reduction)

    for i in range(10):
        torch.manual_seed(seed)
        np.random.seed(seed)

        outputs_t, targets_t = torch.from_numpy(outputs), torch.from_numpy(targets)
        outputs_t, targets_t = outputs_t.cuda(0), targets_t.cuda(0)

        loss_fn = torch.nn.CrossEntropyLoss(reduction=reduction)
        loss_fn = loss_fn.cuda(0)

        loss = loss_fn(outputs_t, targets_t)
        loss = loss.detach().cpu().numpy()
        print(i, outputs.sum(), targets.sum(), outputs.mean(), targets.mean(), loss.sum(), loss.mean())
```
## Output
```
none
0 1048576.0 524341 0.5 0.5000505447387695 769533.4950007759 0.7338843297965774
1 1048576.0 524341 0.5 0.5000505447387695 769533.4950007759 0.7338843297965774
2 1048576.0 524341 0.5 0.5000505447387695 769533.4950007759 0.7338843297965774
3 1048576.0 524341 0.5 0.5000505447387695 769533.4950007759 0.7338843297965774
4 1048576.0 524341 0.5 0.5000505447387695 769533.4950007759 0.7338843297965774
5 1048576.0 524341 0.5 0.5000505447387695 769533.4950007759 0.7338843297965774
6 1048576.0 524341 0.5 0.5000505447387695 769533.4950007759 0.7338843297965774
7 1048576.0 524341 0.5 0.5000505447387695 769533.4950007759 0.7338843297965774
8 1048576.0 524341 0.5 0.5000505447387695 769533.4950007759 0.7338843297965774
9 1048576.0 524341 0.5 0.5000505447387695 769533.4950007759 0.7338843297965774
sum
0 1048576.0 524341 0.5 0.5000505447387695 769533.4950007756 769533.4950007756
1 1048576.0 524341 0.5 0.5000505447387695 769533.4950007756 769533.4950007756
2 1048576.0 524341 0.5 0.5000505447387695 769533.4950007757 769533.4950007757
3 1048576.0 524341 0.5 0.5000505447387695 769533.4950007756 769533.4950007756
4 1048576.0 524341 0.5 0.5000505447387695 769533.4950007756 769533.4950007756
5 1048576.0 524341 0.5 0.5000505447387695 769533.4950007756 769533.4950007756
6 1048576.0 524341 0.5 0.5000505447387695 769533.4950007756 769533.4950007756
7 1048576.0 524341 0.5 0.5000505447387695 769533.4950007754 769533.4950007754
8 1048576.0 524341 0.5 0.5000505447387695 769533.4950007756 769533.4950007756
9 1048576.0 524341 0.5 0.5000505447387695 769533.4950007756 769533.4950007756
mean
0 1048576.0 524341 0.5 0.5000505447387695 0.733884329796577 0.733884329796577
1 1048576.0 524341 0.5 0.5000505447387695 0.733884329796577 0.733884329796577
2 1048576.0 524341 0.5 0.5000505447387695 0.733884329796577 0.733884329796577
3 1048576.0 524341 0.5 0.5000505447387695 0.733884329796577 0.733884329796577
4 1048576.0 524341 0.5 0.5000505447387695 0.733884329796577 0.733884329796577
5 1048576.0 524341 0.5 0.5000505447387695 0.7338843297965769 0.7338843297965769
6 1048576.0 524341 0.5 0.5000505447387695 0.733884329796577 0.733884329796577
7 1048576.0 524341 0.5 0.5000505447387695 0.733884329796577 0.733884329796577
8 1048576.0 524341 0.5 0.5000505447387695 0.733884329796577 0.733884329796577
9 1048576.0 524341 0.5 0.5000505447387695 0.733884329796577 0.733884329796577
```

## Expected behavior
I believe the expected behavior of the reduction='sum' and 'mean' should be as consistent as the 'none' option (where I use numpy for reduction).

## Environment
```
PyTorch version: 1.0.1.post2
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: GPU 0: Tesla K80
Nvidia driver version: 384.111
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.16.1
[pip3] torch==1.0.1.post2
[conda] mkl                       2018.0.0             hb491cac_4
[conda] mkl-service               1.1.2            py36h17a0993_4
[conda] torch                     0.4.0                     <pip>
```
## Additional context
Ran on amazon K80 instance (p2.xlarge).
I know it does seem like a very tiny error but the result of this is that two training sessions of my segmentation network (with identical parameters, initialization, order of image-batches, random seed, etc') doesn't produce identical results. That is problematic when I want to investigate a specific training session.