Repository: mlcommons/inference
Issue: Speech recognition docker image does not build
Root cause: External Dependency Breakage

Hi there,

I have the same error than when compiling by the training side, i.e. :+1: 

 ---> 5cb90201dd79
Step 13/14 : RUN cd ctcdecode; pip install .
 ---> Running in d406e114fed2
Processing /tmp/ctcdecode
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-Sq4p2m/setup.py", line 55, in <module>
        os.path.join(this_file, "build.py:ffi")
      File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 272, in __init__
        _Distribution.__init__(self,attrs)
      File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
        self.finalize_options()
      File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 327, in finalize_options
        ep.load()(self, ep.name, value)
      File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 204, in cffi_modules
        add_cffi_module(dist, cffi_module)
      File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
        execfile(build_file_name, mod_vars)
      File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 25, in execfile
        exec(code, glob, glob)
      File "/tmp/pip-req-build-Sq4p2m/build.py", line 28, in <module>
        'third_party/boost_1_67_0.tar.gz')
      File "/tmp/pip-req-build-Sq4p2m/build.py", line 19, in download_extract
        tar = tarfile.open(dl_path)
      File "/usr/lib/python2.7/tarfile.py", line 1678, in open
        raise ReadError("file could not be opened successfully")
    tarfile.ReadError: file could not be opened successfully
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-Sq4p2m/
The command '/bin/sh -c cd ctcdecode; pip install .' returned a non-zero code: 1


@francois-wellenreiter did you solve the issue? I have a similar issue building the docker image and I solved it editing the Dockerfile and adding the command `RUN pip install --upgrade setuptools`  to "# Make pip happy about itself"

@vilmara it does indeed fix this problem, even there remains many further. Thank you for your feedback.

@francois-wellenreiter what device are you using cpu or gpu?, if gpu, what model?

gpu, nvidia v100

@francois-wellenreiter , I have reported the issue also running the benchmark on T4/V100 gpus #60 

I ran the benchmark via docker on Tesla P4, to do so I had to modify the Dockerfile to install cuda 9.0 (and Pytorch 0.4.0) instead of cuda 8.0 and it worked. 

For T4 and V100 the system recommends using cuda_version >=9 . So, I decided to modify the Dockerfile to install Cuda 9.0 or Cuda 10.0  and either Pytorch 0.4.0 or 1.0.0, in all cases, it built the docker image but got the error when running the inference `RuntimeError: CUDNN_STATUS_BAD_PARAM`. See below the trace log:

Tracelog
```
Number of parameters: 103841568
=======================================================
***use_set                   = libri
***slice                     = False
***batch_size_val            = 1
***continue_from             = ./trained_model_deepspeech2.pth
***force_duration            = -1.0
***checkpoint                = True
***seed                      = 1
***warmups                   = 5
***model_path                = ./deepspeech_final.pth
***device                    = gpu
***batch_1_file              = None
=======================================================
Exporting inference to: /home/dell/MLPerf_inference/cloud/speech_recognition/pytorch/inference/inference_bs1_gpu.csv
Warmups (1/5) Traceback (most recent call last):
  File "inference.py", line 159, in <module>
    main(args)
  File "inference.py", line 112, in main
    meta=True)
  File "../model/eval_model.py", line 117, in eval_model_verbose
    out = model(inputs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 143, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 153, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
    raise output
RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM
```
In the Dockerfile the developer has commented that there are 3 versions of pytorch, but they only had success with 0.4.0.

Please keep me posted if you get progress on this test. Thanks



@vilmara apparently my previous posts weren't clear enough, actually, the docker build stage still fails. I'm not able to run this bench.

@francois-wellenreiter I saw your installation stopped in Step 13/14 with the command `RUN cd ctcdecode; pip install .`. Have you tried with the latest version of the scripts?. I see the new Dockerfile has the updated command:

`RUN cd ctcdecode; git reset --hard 140b45860cec6671fb0bf6dbb675073241c0f9b0; pip install . # This option uses the required commit`

@vilmara I am not able to explain why but inference/cloud built correctly, but neither inference /edge nor training versions


right now speech recgonition is not in v0.5 of the benchmark. Please note that we won't visit this issue until MLPerf resolved all the v0.5 related issues. please re-open if you are still working on this today.