Repository: pytorch
Issue: matmul causing cuDNN error: CUDNN_STATUS_INTERNAL_ERROR at future, unrelated Conv2d
Labels: needs reproduction,module: cudnn,module: cuda,triaged

## 🐛 Bug

Performing a matmul and then separately performing a Conv2d (which does not use any data from the matmul) is causing a `RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR`, among other errors (depending on batch size and dimension, can also get `RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`)

The error also only happens for certain sizes of inputs (demonstrated in the provided reproduction script).

## To Reproduce

Steps to reproduce the behavior:

Run this snippet _twice_:
```
import torch

bs = 16
h = 24
w = 8
hch = 115
cch = 131

nh = 3
n = 66

##### Ops involving matmul
# Data /conv definitions
mask = (torch.tril(torch.ones(w, w)).view(1, 1, w, w) == 0).cuda()
x = torch.randn([bs, nh, h, w, n], device='cuda', requires_grad=True)
matmul_out_conv = torch.nn.Conv2d(nh*n, 2*hch, kernel_size=[1,1]).cuda()
P = torch.randn([bs, nh, h, w, w], device='cuda', requires_grad=True)
P = P.masked_fill(mask, -1e10)
P = torch.nn.functional.softmax(P, dim=-1)

# Ops
out_1 = P @ x
out_1 = out_1.permute(0, 1, 4, 2, 3)
out_1 = out_1.reshape(bs, -1, h, w)
out_1 = matmul_out_conv(out_1)

##### Separate ops unrelated to matmul
# Data / conv definitions
input_a = torch.randn([bs, hch, h, w], device='cuda', requires_grad=True)
pad_a = torch.nn.ZeroPad2d((0, 0, 3, 0)).cuda()
conv_a = torch.nn.Conv2d(hch, 2*hch, kernel_size=[4, 1]).cuda()
conv_a_2 = torch.nn.Conv2d(2*hch, 2*hch, kernel_size=[1,1]).cuda()

# Ops
out_2 = conv_a(pad_a(input_a))
out_2 = conv_a_2(out_2)

out_2.backward(torch.randn_like(out_2))
torch.cuda.synchronize()
```
When I put that into a jupyter notebook, it runs the first time, and then has this output the second time: 
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-4284e7e7c6ee> in <module>
     34 # Ops
     35 out_2 = conv_a(pad_a(input_a))
---> 36 out_2 = conv_a_2(out_2)
     37 
     38 out_2.backward(torch.randn_like(out_2))

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    397 
    398     def forward(self, input: Tensor) -> Tensor:
--> 399         return self._conv_forward(input, self.weight, self.bias)
    400 
    401 class Conv3d(_ConvNd):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    394                             _pair(0), self.dilation, self.groups)
    395         return F.conv2d(input, weight, bias, self.stride,
--> 396                         self.padding, self.dilation, self.groups)
    397 
    398     def forward(self, input: Tensor) -> Tensor:

RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([16, 230, 24, 8], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(230, 230, kernel_size=[1, 1], padding=[0, 0], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams 
    data_type = CUDNN_DATA_FLOAT
    padding = [0, 0, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x6e825ad0
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 16, 230, 24, 8, 
    strideA = 44160, 192, 8, 1, 
output: TensorDescriptor 0x6deb7d40
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 16, 230, 24, 8, 
    strideA = 44160, 192, 8, 1, 
weight: FilterDescriptor 0x6cdfc190
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 230, 230, 1, 1, 
Pointer addresses: 
    input: 0x7f5c47c00000
    output: 0x7f5c46dde200
    weight: 0x7f5c45600000
Forward algorithm: 5
```

If you change the value of `w` to something greater than or equal to 17, the code will run just fine every time. I've found that for `w <= 16`, that's when it breaks.

I've also found that if you comment out the first set of ops involving the matmul (so everything up until the `Separate ops unrelated to matmul` line) that the code runs just fine every time.

If I try to run my full model (so not just the code from the snippet above, but rather my full model which I can't post here) from the command line with `cuda-memcheck` this is what I get:
```
Program hit cudaErrorInvalidConfiguration (error 9) due to "invalid configuration argument" on CUDA API call to cudaLaunchKernel.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x355b43]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so [0x1c5d975]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so (_Z23gemmSN_TN_kernel_64addrIfLi128ELi16ELi2ELi4ELi8ELi9ELb0E30cublasGemvTensorStridedBatchedIKfES2_S0_IfEEv22cublasGemmSmallNParamsIT7_T8_T9_T_E + 0x8c) [0x124080c]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z24cublasGemm_smallN_launchIf30cublasGemvTensorStridedBatchedIKfES2_S0_IfEfE14cublasStatus_tP15cublasLtContextP11CUstream_st19cublasPointerMode_tiiiiiPKT_T0_iSD_iSC_T1_iT2_iii4dim313gemmSmallN_pfISD_SE_SF_SA_EPKc18cublasLtEpilogue_tiPT3_ + 0x2cf) [0x2d6d5df]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN4impl12gemm_small_nIf30cublasGemvTensorStridedBatchedIKfES3_S1_IfELb0EE3runEP13cublasContextP14cublasStatus_tiiiiiPS2_S3_iS3_iSA_S3_iS4_ii + 0x3f9) [0x2d6dda9]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z17cublasGemmBatchedIfffLb1EE14cublasStatus_tP13cublasContext17cublasOperation_tS3_iiiPKT1_PKPKT_S9_ixSB_S9_ixS6_PKPT0_SD_ixib + 0x98a) [0x2cb786a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (cublasSgemmStridedBatched + 0xea) [0x2ca51ea]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb0c720]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native84_GLOBAL__N__60_tmpxft_000060fe_00000000_13_LinearAlgebra_compute_86_cpp1_ii_b241008621baddbmm_out_cuda_implERNS_6TensorERKS2_S5_S5_N3c106ScalarES7_ + 0xe98) [0x1d2e808]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native12bmm_out_cudaERNS_6TensorERKS1_S4_ + 0x195) [0x1d30c85]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native8bmm_cudaERKNS_6TensorES3_ + 0x58) [0x1d31618]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5cea8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5ceef]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x1455596]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at3bmmERKNS_6TensorES2_ + 0x4f) [0x145c4bf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ba8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ecf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x18a77d6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZNK2at6Tensor3bmmERKS0_ + 0x4f) [0x18acdff]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x4a5a9b]
=========     Host Frame:python3 [0x10a5a5]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 [0x1096c8]
=========     Host Frame:python3 [0x10a3fd]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 [0x1096c8]
=========     Host Frame:python3 [0x10a3fd]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========
========= Program hit cudaErrorInvalidConfiguration (error 9) due to "invalid configuration argument" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x355b43]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so [0x1c43ec6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so (_ZNK28gemm_matmulAlgo_gemv_stridedIfffE3runERK20gemmInternalParams_tRK18matmulAlgoConfig_tPv + 0x56d) [0x1272a1d]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so [0xfb1035]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so (cublasLtMatmul + 0x60f) [0xfaec4f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0x2d9c308]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z17cublasGemmBatchedIfffLb1EE14cublasStatus_tP13cublasContext17cublasOperation_tS3_iiiPKT1_PKPKT_S9_ixSB_S9_ixS6_PKPT0_SD_ixib + 0x87c) [0x2cb775c]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (cublasSgemmStridedBatched + 0xea) [0x2ca51ea]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb0c720]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native84_GLOBAL__N__60_tmpxft_000060fe_00000000_13_LinearAlgebra_compute_86_cpp1_ii_b241008621baddbmm_out_cuda_implERNS_6TensorERKS2_S5_S5_N3c106ScalarES7_ + 0xe98) [0x1d2e808]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native12bmm_out_cudaERNS_6TensorERKS1_S4_ + 0x195) [0x1d30c85]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native8bmm_cudaERKNS_6TensorES3_ + 0x58) [0x1d31618]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5cea8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5ceef]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x1455596]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at3bmmERKNS_6TensorES2_ + 0x4f) [0x145c4bf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ba8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ecf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x18a77d6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZNK2at6Tensor3bmmERKS0_ + 0x4f) [0x18acdff]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x4a5a9b]
=========     Host Frame:python3 [0x10a5a5]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 [0x1096c8]
=========     Host Frame:python3 [0x10a3fd]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 [0x1096c8]
=========     Host Frame:python3 [0x10a3fd]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========
========= Program hit cudaErrorInvalidConfiguration (error 9) due to "invalid configuration argument" on CUDA API call to cudaLaunchKernel.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x355b43]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so [0x1c5d975]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so (_Z16gemmSN_NN_kernelIfLi128ELi2ELi4ELi8ELi4ELi4ELb0E30cublasGemvTensorStridedBatchedIKfES2_S0_IfEEv22cublasGemmSmallNParamsIT7_T8_T9_T_E + 0x8c) [0x123e8cc]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z24cublasGemm_smallN_launchIf30cublasGemvTensorStridedBatchedIKfES2_S0_IfEfE14cublasStatus_tP15cublasLtContextP11CUstream_st19cublasPointerMode_tiiiiiPKT_T0_iSD_iSC_T1_iT2_iii4dim313gemmSmallN_pfISD_SE_SF_SA_EPKc18cublasLtEpilogue_tiPT3_ + 0x2cf) [0x2d6d5df]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN4impl12gemm_small_nIf30cublasGemvTensorStridedBatchedIKfES3_S1_IfELb0EE3runEP13cublasContextP14cublasStatus_tiiiiiPS2_S3_iS3_iSA_S3_iS4_ii + 0x535) [0x2d6dee5]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z17cublasGemmBatchedIfffLb1EE14cublasStatus_tP13cublasContext17cublasOperation_tS3_iiiPKT1_PKPKT_S9_ixSB_S9_ixS6_PKPT0_SD_ixib + 0x98a) [0x2cb786a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (cublasSgemmStridedBatched + 0xea) [0x2ca51ea]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb0c720]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native84_GLOBAL__N__60_tmpxft_000060fe_00000000_13_LinearAlgebra_compute_86_cpp1_ii_b241008621baddbmm_out_cuda_implERNS_6TensorERKS2_S5_S5_N3c106ScalarES7_ + 0xe98) [0x1d2e808]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native12bmm_out_cudaERNS_6TensorERKS1_S4_ + 0x195) [0x1d30c85]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native8bmm_cudaERKNS_6TensorES3_ + 0x58) [0x1d31618]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5cea8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5ceef]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x1455596]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at3bmmERKNS_6TensorES2_ + 0x4f) [0x145c4bf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ba8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ecf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x18a77d6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZNK2at6Tensor3bmmERKS0_ + 0x4f) [0x18acdff]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0xfd47c8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6native6matmulERKNS_6TensorES3_ + 0x3a) [0xfd588a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17ba548]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17ba58f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x18a77d6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZNK2at6Tensor6matmulERKS0_ + 0x4f) [0x18abb3f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x2cf4b0]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x2cfc16]
=========     Host Frame:python3 (PyCFunction_Call + 0x103) [0x166fb3]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 [0xb1fba]
=========     Host Frame:python3 [0x8877d]
=========     Host Frame:python3 [0x19d1b2]
=========     Host Frame:python3 (PyNumber_MatrixMultiply + 0x14) [0xa3b76]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x753a) [0x11303a]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========
========= Program hit cudaErrorInvalidConfiguration (error 9) due to "invalid configuration argument" on CUDA API call to cudaLaunchKernel.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x355b43]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so [0x1c5d975]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so (_Z16gemmSN_NN_kernelIfLi128ELi2ELi4ELi8ELi4ELi4ELb0E30cublasGemvTensorStridedBatchedIKfES2_S0_IfEEv22cublasGemmSmallNParamsIT7_T8_T9_T_E + 0x8c) [0x123e8cc]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z24cublasGemm_smallN_launchIf30cublasGemvTensorStridedBatchedIKfES2_S0_IfEfE14cublasStatus_tP15cublasLtContextP11CUstream_st19cublasPointerMode_tiiiiiPKT_T0_iSD_iSC_T1_iT2_iii4dim313gemmSmallN_pfISD_SE_SF_SA_EPKc18cublasLtEpilogue_tiPT3_ + 0x2cf) [0x2d6d5df]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN4impl12gemm_small_nIf30cublasGemvTensorStridedBatchedIKfES3_S1_IfELb0EE3runEP13cublasContextP14cublasStatus_tiiiiiPS2_S3_iS3_iSA_S3_iS4_ii + 0x535) [0x2d6dee5]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z17cublasGemmBatchedIfffLb1EE14cublasStatus_tP13cublasContext17cublasOperation_tS3_iiiPKT1_PKPKT_S9_ixSB_S9_ixS6_PKPT0_SD_ixib + 0x98a) [0x2cb786a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (cublasSgemmStridedBatched + 0xea) [0x2ca51ea]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb0c720]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native84_GLOBAL__N__60_tmpxft_000060fe_00000000_13_LinearAlgebra_compute_86_cpp1_ii_b241008621baddbmm_out_cuda_implERNS_6TensorERKS2_S5_S5_N3c106ScalarES7_ + 0xe98) [0x1d2e808]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native12bmm_out_cudaERNS_6TensorERKS1_S4_ + 0x195) [0x1d30c85]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native8bmm_cudaERKNS_6TensorES3_ + 0x58) [0x1d31618]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5cea8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5ceef]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x1455596]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at3bmmERKNS_6TensorES2_ + 0x4f) [0x145c4bf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ba8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ecf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x18a77d6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZNK2at6Tensor3bmmERKS0_ + 0x4f) [0x18acdff]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0xfd47c8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6native6matmulERKNS_6TensorES3_ + 0x3a) [0xfd588a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17ba548]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17ba58f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x18a77d6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZNK2at6Tensor6matmulERKS0_ + 0x4f) [0x18abb3f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x2cf4b0]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x2cfc16]
=========     Host Frame:python3 (PyCFunction_Call + 0x103) [0x166fb3]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 [0xb1fba]
=========     Host Frame:python3 [0x8877d]
=========     Host Frame:python3 [0x19d1b2]
=========     Host Frame:python3 (PyNumber_MatrixMultiply + 0x14) [0xa3b76]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x753a) [0x11303a]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========
========= Program hit cudaErrorInvalidConfiguration (error 9) due to "invalid configuration argument" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x355b43]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so [0x1c43ec6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so (_Z8run_gemmI6float2EiR13cublasContextP11CUstream_stR14cublasStatus_t10gemmType_t17cublasOperation_tS8_iiiPKT_SB_PKv14cudaDataType_tiSD_SE_iPvSE_i12shapeTypeC_tbiibmmmPKciii + 0xe3b) [0x12ad48b]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z17cublasGemmBatchedI6float2S0_S0_Lb1EE14cublasStatus_tP13cublasContext17cublasOperation_tS4_iiiPKT1_PKPKT_SA_ixSC_SA_ixS7_PKPT0_SE_ixib + 0x9e7) [0x2cb87a7]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (cublasCgemm3mStridedBatched + 0xea) [0x2ca5c6a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x276086a8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x274f26f3]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26faa7f3]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26f9ca80]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26faa7f3]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x27420fbc]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x27432e01]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26faa7f3]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x273a51db]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26f8a60c]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26f854f9]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26f917b6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26f9226c]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x25fac5dd]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x25faf1c7]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x25faf59c]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x25fa8539]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so (_ZN2at6native17cudnn_convolutionERKNS_6TensorES3_N3c108ArrayRefIlEES6_S6_lbbb + 0xb1) [0x25fa8d61]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5e173]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5e200]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at17cudnn_convolutionERKNS_6TensorES2_N3c108ArrayRefIlEES5_S5_lbbb + 0x1f9) [0x14b7589]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2cccb05]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2ccd010]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at17cudnn_convolutionERKNS_6TensorES2_N3c108ArrayRefIlEES5_S5_lbbb + 0x1f9) [0x14b7589]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6native12_convolutionERKNS_6TensorES3_S3_N3c108ArrayRefIlEES6_S6_bS6_lbbbb + 0x2e1f) [0xefbb3f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17b3676]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17de3c9]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at12_convolutionERKNS_6TensorES2_RKN3c108optionalIS0_EENS3_8ArrayRefIlEES9_S9_bS9_lbbbb + 0x27e) [0x14af99e]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6native11convolutionERKNS_6TensorES3_S3_N3c108ArrayRefIlEES6_S6_bS6_l + 0xd6) [0xef58b6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17b3537]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17de762]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x162002e]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at11convolutionERKNS_6TensorES2_RKN3c108optionalIS0_EENS3_8ArrayRefIlEES9_S9_bS9_l + 0xb9) [0x14ae999]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6native6conv2dERKNS_6TensorES3_S3_N3c108ArrayRefIlEES6_S6_l + 0x67) [0xef55d7]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17b3a82]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17ded1f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x1620833]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6conv2dERKNS_6TensorES2_RKN3c108optionalIS0_EENS3_8ArrayRefIlEES9_S9_l + 0x9e) [0x14b242e]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x48f96a]
=========     Host Frame:python3 [0x10a5a5]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 [0x1096c8]
=========     Host Frame:python3 [0x10a3fd]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========
========= Program hit cudaErrorInvalidConfiguration (error 9) due to "invalid configuration argument" on CUDA API call to cudaLaunchKernel.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x355b43]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so [0x1c5d975]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so (_Z16gemmSN_NN_kernelIfLi128ELi2ELi4ELi8ELi4ELi4ELb0E30cublasGemvTensorStridedBatchedIKfES2_S0_IfEEv22cublasGemmSmallNParamsIT7_T8_T9_T_E + 0x8c) [0x123e8cc]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z24cublasGemm_smallN_launchIf30cublasGemvTensorStridedBatchedIKfES2_S0_IfEfE14cublasStatus_tP15cublasLtContextP11CUstream_st19cublasPointerMode_tiiiiiPKT_T0_iSD_iSC_T1_iT2_iii4dim313gemmSmallN_pfISD_SE_SF_SA_EPKc18cublasLtEpilogue_tiPT3_ + 0x2cf) [0x2d6d5df]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN4impl12gemm_small_nIf30cublasGemvTensorStridedBatchedIKfES3_S1_IfELb0EE3runEP13cublasContextP14cublasStatus_tiiiiiPS2_S3_iS3_iSA_S3_iS4_ii + 0x535) [0x2d6dee5]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z17cublasGemmBatchedIfffLb1EE14cublasStatus_tP13cublasContext17cublasOperation_tS3_iiiPKT1_PKPKT_S9_ixSB_S9_ixS6_PKPT0_SD_ixib + 0x98a) [0x2cb786a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (cublasSgemmStridedBatched + 0xea) [0x2ca51ea]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb0c720]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native84_GLOBAL__N__60_tmpxft_000060fe_00000000_13_LinearAlgebra_compute_86_cpp1_ii_b241008621baddbmm_out_cuda_implERNS_6TensorERKS2_S5_S5_N3c106ScalarES7_ + 0xe98) [0x1d2e808]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native12bmm_out_cudaERNS_6TensorERKS1_S4_ + 0x195) [0x1d30c85]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native8bmm_cudaERKNS_6TensorES3_ + 0x58) [0x1d31618]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5cea8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5ceef]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x1455596]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at3bmmERKNS_6TensorES2_ + 0x4f) [0x145c4bf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ba8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ecf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x18a77d6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZNK2at6Tensor3bmmERKS0_ + 0x4f) [0x18acdff]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0xfd47c8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6native6matmulERKNS_6TensorES3_ + 0x3a) [0xfd588a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17ba548]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17ba58f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x18a77d6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZNK2at6Tensor6matmulERKS0_ + 0x4f) [0x18abb3f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x2cf4b0]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x2cfc16]
=========     Host Frame:python3 (PyCFunction_Call + 0x103) [0x166fb3]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 [0xb1fba]
=========     Host Frame:python3 [0x8877d]
=========     Host Frame:python3 [0x19d1b2]
=========     Host Frame:python3 (PyNumber_MatrixMultiply + 0x14) [0xa3b76]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x753a) [0x11303a]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========
========= Program hit cudaErrorInvalidConfiguration (error 9) due to "invalid configuration argument" on CUDA API call to cudaLaunchKernel.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x355b43]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so [0x1c5d975]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so (_Z16gemmSN_NN_kernelIfLi128ELi2ELi4ELi8ELi4ELi4ELb0E30cublasGemvTensorStridedBatchedIKfES2_S0_IfEEv22cublasGemmSmallNParamsIT7_T8_T9_T_E + 0x8c) [0x123e8cc]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z24cublasGemm_smallN_launchIf30cublasGemvTensorStridedBatchedIKfES2_S0_IfEfE14cublasStatus_tP15cublasLtContextP11CUstream_st19cublasPointerMode_tiiiiiPKT_T0_iSD_iSC_T1_iT2_iii4dim313gemmSmallN_pfISD_SE_SF_SA_EPKc18cublasLtEpilogue_tiPT3_ + 0x2cf) [0x2d6d5df]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN4impl12gemm_small_nIf30cublasGemvTensorStridedBatchedIKfES3_S1_IfELb0EE3runEP13cublasContextP14cublasStatus_tiiiiiPS2_S3_iS3_iSA_S3_iS4_ii + 0x535) [0x2d6dee5]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z17cublasGemmBatchedIfffLb1EE14cublasStatus_tP13cublasContext17cublasOperation_tS3_iiiPKT1_PKPKT_S9_ixSB_S9_ixS6_PKPT0_SD_ixib + 0x98a) [0x2cb786a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (cublasSgemmStridedBatched + 0xea) [0x2ca51ea]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb0c720]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native84_GLOBAL__N__60_tmpxft_000060fe_00000000_13_LinearAlgebra_compute_86_cpp1_ii_b241008621baddbmm_out_cuda_implERNS_6TensorERKS2_S5_S5_N3c106ScalarES7_ + 0xe98) [0x1d2e808]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native12bmm_out_cudaERNS_6TensorERKS1_S4_ + 0x195) [0x1d30c85]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_ZN2at6native8bmm_cudaERKNS_6TensorES3_ + 0x58) [0x1d31618]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5cea8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5ceef]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x1455596]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at3bmmERKNS_6TensorES2_ + 0x4f) [0x145c4bf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ba8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2c63ecf]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x18a77d6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZNK2at6Tensor3bmmERKS0_ + 0x4f) [0x18acdff]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0xfd47c8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6native6matmulERKNS_6TensorES3_ + 0x3a) [0xfd588a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17ba548]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17ba58f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x18a77d6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZNK2at6Tensor6matmulERKS0_ + 0x4f) [0x18abb3f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x2cf4b0]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x2cfc16]
=========     Host Frame:python3 (PyCFunction_Call + 0x103) [0x166fb3]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 [0xb1fba]
=========     Host Frame:python3 [0x8877d]
=========     Host Frame:python3 [0x19d1b2]
=========     Host Frame:python3 (PyNumber_MatrixMultiply + 0x14) [0xa3b76]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x753a) [0x11303a]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 [0x14a621]
=========     Host Frame:python3 (_PyObject_FastCallKeywords + 0x19c) [0x1a9d2c]
=========     Host Frame:python3 [0x10a533]
=========
========= Program hit cudaErrorInvalidConfiguration (error 9) due to "invalid configuration argument" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x355b43]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so [0x1c43ec6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so (_Z8run_gemmI6float2EiR13cublasContextP11CUstream_stR14cublasStatus_t10gemmType_t17cublasOperation_tS8_iiiPKT_SB_PKv14cudaDataType_tiSD_SE_iPvSE_i12shapeTypeC_tbiibmmmPKciii + 0xe3b) [0x12ad48b]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (_Z17cublasGemmBatchedI6float2S0_S0_Lb1EE14cublasStatus_tP13cublasContext17cublasOperation_tS4_iiiPKT1_PKPKT_SA_ixSC_SA_ixS7_PKPT0_SE_ixib + 0x9e7) [0x2cb87a7]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so (cublasCgemm3mStridedBatched + 0xea) [0x2ca5c6a]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x276086a8]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x274f26f3]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26faa7f3]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26f9ca80]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26faa7f3]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x27420fbc]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x27432e01]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26faa7f3]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x273a51db]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26f8a60c]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26f854f9]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26f917b6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x26f9226c]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x25fac5dd]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x25faf0da]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x25faf59c]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so [0x25fa8539]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cpp.so (_ZN2at6native17cudnn_convolutionERKNS_6TensorES3_N3c108ArrayRefIlEES6_S6_lbbb + 0xb1) [0x25fa8d61]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5e173]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda_cu.so [0xb5e200]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at17cudnn_convolutionERKNS_6TensorES2_N3c108ArrayRefIlEES5_S5_lbbb + 0x1f9) [0x14b7589]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2cccb05]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x2ccd010]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at17cudnn_convolutionERKNS_6TensorES2_N3c108ArrayRefIlEES5_S5_lbbb + 0x1f9) [0x14b7589]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6native12_convolutionERKNS_6TensorES3_S3_N3c108ArrayRefIlEES6_S6_bS6_lbbbb + 0x2e1f) [0xefbb3f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17b3676]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17de3c9]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at12_convolutionERKNS_6TensorES2_RKN3c108optionalIS0_EENS3_8ArrayRefIlEES9_S9_bS9_lbbbb + 0x27e) [0x14af99e]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6native11convolutionERKNS_6TensorES3_S3_N3c108ArrayRefIlEES6_S6_bS6_l + 0xd6) [0xef58b6]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17b3537]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17de762]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x162002e]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at11convolutionERKNS_6TensorES2_RKN3c108optionalIS0_EENS3_8ArrayRefIlEES9_S9_bS9_l + 0xb9) [0x14ae999]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6native6conv2dERKNS_6TensorES3_S3_N3c108ArrayRefIlEES6_S6_l + 0x67) [0xef55d7]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17b3a82]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x17ded1f]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so [0x1620833]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so (_ZN2at6conv2dERKNS_6TensorES2_RKN3c108optionalIS0_EENS3_8ArrayRefIlEES9_S9_l + 0x9e) [0x14b242e]
=========     Host Frame:/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so [0x48f96a]
=========     Host Frame:python3 [0x10a5a5]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 [0x1096c8]
=========     Host Frame:python3 [0x10a3fd]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x444) [0x10bf44]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0xf5) [0x108dc5]
=========     Host Frame:python3 [0x195221]
=========     Host Frame:python3 (PyObject_Call + 0x3e) [0x1a058e]
=========     Host Frame:python3 (_PyEval_EvalFrameDefault + 0x19ff) [0x10d4ff]
=========     Host Frame:python3 [0x107cd4]
=========     Host Frame:python3 (_PyFunction_FastCallDict + 0x2e2) [0x108fb2]
=========
Traceback (most recent call last):
  ...*line references within my code*
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([8, 230, 24, 8], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(230, 230, kernel_size=[1, 1], padding=[0, 0], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
    data_type = CUDNN_DATA_FLOAT
    padding = [0, 0, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0xbdee9c10
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 8, 230, 24, 8,
    strideA = 44160, 192, 8, 1,
output: TensorDescriptor 0xbdee4c50
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 8, 230, 24, 8,
    strideA = 44160, 192, 8, 1,
weight: FilterDescriptor 0xbddd79b0
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 230, 230, 1, 1,
Pointer addresses:
    input: 0x7f402327c000
    output: 0x7f40233d5000
    weight: 0x7f402b800000
Forward algorithm: 5
```

## Expected behavior

It should just work

## Environment

PyTorch version: 1.8.1+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.25

Python version: 3.6 (64-bit runtime)
Python platform: Linux-5.4.0-1044-gcp-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 9.1.85
GPU models and configuration:
GPU 0: NVIDIA Tesla P100-PCIE-16GB
GPU 1: NVIDIA Tesla P100-PCIE-16GB

Nvidia driver version: 465.19.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.18.1
[pip3] torch==1.8.1+cu111
[conda] Could not collect

## Additional context


cc @ngimel @csarofeen @ptrblck @xwang233