Repository: mlcommons/inference
Issue: Deep Speech Inference with GPU
Root cause: Environment Incompatibility

Hi all, I want to run the deep speech inference using T4 and V100 GPUs models but I got the below warning and error:

**Warning**
```
  warnings.warn(incorrect_binary_warn % (d, name, 9000, CUDA_VERSION))
/usr/local/lib/python2.7/dist-packages/torch/cuda/__init__.py:114: UserWarning:
    Found GPU3 Tesla V100-SXM2-16GB which requires CUDA_VERSION >= 9000 for
     optimal performance and fast startup time, but your PyTorch was compiled
     with CUDA_VERSION 8000. Please install the correct PyTorch binary
     using instructions from http://pytorch.org
```

**Error:**
`RuntimeError: CUDNN_STATUS_MAPPING_ERROR`

So, I decided to build the docker image using Pytorch with Cuda 9.0 or Cuda 10.0, none of them worked. Some suggestion?



Ok, I have modified the Dockerfile to run the benchmark with Pytorch version 1.0.1.post2 and CUDA Version 9.0.176, the docker image was built but when running the test I got the error `RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM` see below the partial trace log:

```
number of parameters: 103841568
=======================================================
***use_set                   = libri
***slice                     = False
***batch_size_val            = 1
***continue_from             = ./trained_model_deepspeech2.pth
***force_duration            = -1.0
***checkpoint                = True
***seed                      = 1
***warmups                   = 5
***model_path                = ./deepspeech_final.pth
***device                    = gpu
***batch_1_file              = None
=======================================================
Exporting inference to: /home/dell/MLPerf_inference/cloud/speech_recognition/pytorch/inference/inference_bs1_gpu.csv
Warmups (1/5) Traceback (most recent call last):
  File "inference.py", line 159, in <module>
    main(args)
  File "inference.py", line 112, in main
    meta=True)
  File "../model/eval_model.py", line 117, in eval_model_verbose
    out = model(inputs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 143, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 153, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
    raise output
RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM

```



Hi vilmara, 
can you check the version of cuda installed inside docker. if you have cuda 9 installed then instead of default cu8 version of pytorch (http://download.pytorch.org/whl/cu80/torch-0.4.0-cp27-cp27mu-linux_x86_64.whl) try using  cu9 version of pytorch (http://download.pytorch.org/whl/cu90/torch-0.4.0-cp27-cp27mu-linux_x86_64.whl).  let me know if this helped. 

Thanks @avinashchakravarthi1 , it worked using cu9 and pytorch 0.4.0 (http://download.pytorch.org/whl/cu90/torch-0.4.0-cp27-cp27mu-linux_x86_64.whl) 

i found the same issue, cuda9.0+torch0.4.0, RuntimeError: CUDNN_STATUS_MAPPING_ERROR,
but when i using cuda9 + new pytorch from this link(http://download.pytorch.org/whl/cu90/torch-0.4.0-cp27-cp27mu-linux_x86_64.whl), i got new error "**RuntimeError: CUDNN_STATUS_EXECUTION_FAILED**" like this,

any ideas? 

`Traceback (most recent call last):
  File "inference.py", line 160, in <module>
    main(args)
  File "inference.py", line 76, in main
    model = torch.nn.DataParallel(model).cuda()
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 105, in __init__
    self.module.cuda(device_ids[0])
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 249, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/rnn.py", line 112, in _apply
    self.flatten_parameters()
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/rnn.py", line 105, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED
`

@vilmara @avinashchakravarthi1 

I found out it works fine on V100 & P40 when I used this pytorch version (http://download.pytorch.org/whl/cu90/torch-0.4.0-cp27-cp27mu-linux_x86_64.whl), but when I set CUDA_VISIBLE_DEVICES on T4, I got this error RuntimeError: CUDNN_STATUS_EXECUTION_FAILED, just like I wrote above, why is that?


Hi @suntao2012, for now the only inferences integrated to the LoadGen are image classification, object detection, and language translation. DeepSpeech v2 is a model candidate for coming releases