Repository: pytorch
Issue: pytorch and c++ inference disagree
Labels: module: cpp,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

The output from a traced model using (1) python and (2) c++ are different.  

xpost: https://discuss.pytorch.org/t/c-and-pytorch-inference-discrepancy/77388?u=jonrbates

## To Reproduce

Steps to reproduce the behavior:

**1. Create model in python**
```
import torch
from transformers import GPT2LMHeadModel
model = GPT2LMHeadModel.from_pretrained("distilgpt2", 
           pad_token_id=50256,
           torchscript=True)
traced_model = torch.jit.trace(model, [torch.tensor([[464, 6842, 1816,  625,  262, 8598,  284,  766]])])
torch.jit.save(traced_model, "traced_distilgpt2.pt")
```

**2. Inference in pytorch**
```
loaded_model = torch.jit.load("traced_distilgpt2.pt")
outputs = loaded_model(torch.tensor([[464, 6842, 1816,  625,  262, 8598,  284,  766]]))

torch.max(outputs[0],2)
>> torch.return_types.max(values=tensor([[-26.7050, -53.3268, -66.3666, -50.1084, -54.8050, -72.4056, -55.2586,-63.5215]], grad_fn=<MaxBackward0>),
indices=tensor([[ 383,  373,  319,  262, 1353,  290,  262,  262]]))

torch.min(outputs[0][:, :, :],2)
>>torch.return_types.min(
values=tensor([[ -47.3794,  -77.6341,  -95.8365,  -75.3026,  -81.6899, -103.6701, -83.0335,  -93.0259]], grad_fn=<MinBackward0>),
indices=tensor([[  154,  7134, 31204, 22997, 10298, 31204, 22997, 31573]]))
```

**3. Inference in c**
```
int main(int argc, const char* argv[]) {

  torch::jit::script::Module module;
  module = torch::jit::load(argv[1]);
  std::cout << "Model loaded.\n";
  module.eval(); // just in case?

  torch::Tensor x = torch::tensor({464, 6842, 1816,  625,  262, 8598,  284,  766},
    torch::dtype(torch::kInt64)).reshape({8, 1});
  std::vector<torch::jit::IValue> inputs;
  inputs.push_back(x);

  // Execute the model and turn its output into a tensor (all_encoder_layers).
  torch::Tensor out = module.forward(inputs).toTuple()->elements()[0].toTensor();

  auto z = torch::max(out, /*dim=*/2);
  std::cout << std::get<0>(z) << '\n';
  std::cout << std::get<1>(z) << '\n';

  z = torch::min(out, /*dim=*/2);
  std::cout << std::get<0>(z) << '\n';
  std::cout << std::get<1>(z) << '\n';

  return 0;
}
```

Output from running c exe (with argv[1] = "traced_distilgpt2.pt") :
```
-26.7050
-28.0251
-29.0539
-28.2608
-26.3226
-28.0652
-26.8328
-28.5087
[ Variable[CPUFloatType]{8,1} ]
 383
 383
 383
 383
 383
 383
 383
 383
[ Variable[CPULongType]{8,1} ]
-47.3794
-48.2470
-49.5565
-49.0826
-46.3104
-48.5934
-47.3013
-49.1308
[ Variable[CPUFloatType]{8,1} ]
 154
 154
 154
 154
 154
 154
 154
 154
[ Variable[CPULongType]{8,1} ]
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior
The models agree for the first token (position=0) but for positions 1-7 the encoder outputs disagree.  All outputs should be equal. 
<!-- -->

## Environment (Pytorch)
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 5.4.0
CMake version: version 3.14.0

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Quadro M500M
Nvidia driver version: 442.23
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip] Could not collect
[conda] _tflow_select             2.3.0                       mkl
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.1.243             h74a9793_0
[conda] libmklml                  2019.0.5                      0
[conda] mkl                       2019.4                      245
[conda] mkl-include               2020.0                      166
[conda] mkl-service               2.3.0            py37hb782905_0
[conda] mkl_fft                   1.0.14           py37h14836fe_0
[conda] mkl_random                1.1.0            py37h675688f_0
[conda] numpy                     1.16.5           py37h19fb1c0_0
[conda] numpy-base                1.16.5           py37hc3f5095_0
[conda] numpydoc                  0.9.1                      py_0
[conda] pytorch                   1.4.0           py3.7_cuda101_cudnn7_0    pytorch
[conda] tensorflow                2.1.0           mkl_py37ha977152_0
[conda] tensorflow-base           2.1.0           mkl_py37h230818c_0
[conda] torchvision               0.5.0                py37_cu101    pytorch

## Environment (C++)

PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2

Python version: 2.7
Is CUDA available: N/A
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect


cc @yf225