Repository: pytorch
Issue: [cudatoolkit 11.0] segfaults
Labels: module: build,module: cuda,triaged

## 🐛 Bug

`malloc(): invalid size (unsorted)`

happens with `cudatoolkit==11.0`, works fine with  `cudatoolkit==10.2` w/ pt-1.6 or pt-nightly - no difference.

Installed with:
```
conda install -y pytorch torchvision cudatoolkit=11.0 -c pytorch-nightly
```
I see that if I try:
 ```
conda install -y pytorch torchvision cudatoolkit=11.0 -c pytorch
```
I end up with pytorch-1.4! I guess there is no 1.6 build for cuda-11

## To Reproduce

Steps to reproduce the behavior:

```
git clone https://github.com/huggingface/transformers/
cd transformers/
pip install -e ".[dev]"
pytest -sv tests/test_benchmark_tf.py::TFBenchmarkTest::test_inference_encoder_decoder_with_configs
```

I don't know anything about this code.  Looks mostly like tf, but for some reason it crashes with `cudatoolkit==11.0, perhaps some odd interaction?

log:
```
================================================================ test session starts ================================================================
platform linux -- Python 3.8.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1 -- /home/stas/anaconda3/envs/main-38/bin/python
cachedir: .pytest_cache
rootdir: /mnt/nvme1/code/huggingface/transformers-master
plugins: typeguard-2.10.0, forked-1.3.0, xdist-2.1.0, instafail-0.4.2
collecting ... 2020-10-23 23:30:22.604434: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-23 23:30:24.507065: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-10-23 23:30:24.507197: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:24.507610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX TITAN X computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 24 deviceMemorySize: 11.93GiB deviceMemoryBandwidth: 313.37GiB/s
2020-10-23 23:30:24.507666: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:24.508033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties: 
pciBusID: 0000:02:00.0 name: GeForce GTX TITAN X computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 24 deviceMemorySize: 11.93GiB deviceMemoryBandwidth: 313.37GiB/s
2020-10-23 23:30:24.508055: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-23 23:30:24.509160: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-23 23:30:24.509195: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-23 23:30:24.509216: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-23 23:30:24.509238: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-23 23:30:24.509885: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-23 23:30:24.512312: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-23 23:30:24.512395: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:24.512821: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:24.513213: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:24.513598: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:24.513949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
collected 1 item                                                                                                                                    

tests/test_benchmark_tf.py::TFBenchmarkTest::test_inference_encoder_decoder_with_configs 1 / 1
2020-10-23 23:30:24.918919: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-23 23:30:24.941780: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3199980000 Hz
2020-10-23 23:30:24.943151: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55dcff3b4330 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-23 23:30:24.943205: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-23 23:30:24.943717: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:24.944967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX TITAN X computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 24 deviceMemorySize: 11.93GiB deviceMemoryBandwidth: 313.37GiB/s
2020-10-23 23:30:24.945045: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-23 23:30:24.945122: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-23 23:30:24.945154: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-23 23:30:24.945177: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-23 23:30:24.945202: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-23 23:30:24.945251: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-23 23:30:24.945307: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-23 23:30:24.945494: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:24.946839: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:24.947863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-23 23:30:26.095801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-23 23:30:26.095827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-10-23 23:30:26.095833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-10-23 23:30:26.096004: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:26.096443: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:26.096839: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-23 23:30:26.097263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10967 MB memory) -> physical GPU (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0, compute capability: 5.2)
2020-10-23 23:30:26.098546: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55dd491d4430 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-10-23 23:30:26.098558: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX TITAN X, Compute Capability 5.2
malloc(): invalid size (unsorted)
Fatal Python error: Aborted

Current thread 0x00007fd58c704740 (most recent call first):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 1187 in var_handle_op
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 154 in _variable_handle_from_shape_and_dtype
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 242 in eager_safe_variable_handle
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1661 in _init_from_args
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1507 in __init__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 264 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/ops/variable_scope.py", line 2633 in default_variable_creator_v2
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 237 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 244 in _variable_v2_call
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 262 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 317 in _init_batch_counters
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 457 in _method_wrapper
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 308 in __init__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 457 in _method_wrapper
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/modeling_tf_utils.py", line 339 in __init__
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/modeling_tf_t5.py", line 1145 in __init__
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/benchmark/benchmark_tf.py", line 141 in _prepare_inference_func
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/benchmark/benchmark_tf.py", line 111 in _inference_memory
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/benchmark/benchmark_utils.py", line 643 in inference_memory
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/benchmark/benchmark_utils.py", line 674 in run
  File "/mnt/nvme1/code/huggingface/transformers-master/tests/test_benchmark_tf.py", line 147 in test_inference_encoder_decoder_with_configs
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/unittest/case.py", line 633 in _callTestMethod
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/unittest/case.py", line 676 in run
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/unittest/case.py", line 736 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/unittest.py", line 278 in runtest
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 163 in pytest_runtest_call
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 256 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 310 in from_call
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 216 in call_and_report
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 127 in runtestprotocol
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 110 in pytest_runtest_protocol
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 338 in pytest_runtestloop
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 313 in _main
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 257 in wrap_session
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 306 in pytest_cmdline_main
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 164 in main
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 187 in console_main
  File "/home/stas/anaconda3/envs/main-38/bin/pytest", line 8 in <module>
Aborted (core dumped)
```

## Environment

```
PyTorch version: 1.8.0.dev20201023
Is debug build: True
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: GeForce GTX TITAN X
GPU 1: GeForce GTX TITAN X

Nvidia driver version: 455.23.04
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] pytorch-lightning==0.9.0
[pip3] torch==1.8.0.dev20201023
[pip3] torchvision==0.8.0.dev20201023
[conda] _tflow_select             2.3.0                       mkl
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.0.221             h6bb024c_0
[conda] mkl                       2020.2                      256
[conda] mkl-service               2.3.0            py38he904b0f_0
[conda] mkl_fft                   1.2.0            py38h23d657b_0
[conda] mkl_random                1.1.1            py38h0573a6f_0
[conda] numpy                     1.18.5                   pypi_0    pypi
[conda] pytorch                   1.8.0.dev20201023 py3.8_cuda11.0.221_cudnn8.0.3_0    pytorch-nightly
[conda] pytorch-lightning         0.9.0                    pypi_0    pypi
[conda] torchvision               0.8.0.dev20201023      py38_cu110    pytorch-nightly
```


cc @malfet @seemethere @walterddr @ngimel