Repository: mlcommons/inference
Issue: MLPerf Benchmark suite with Bert ONNX Model on the Squad dataset hangs at `Running LoadGen test...` step and does not seem to use the GPU.
Root cause: Configuration Error

I am trying to use the MLPerf benchmark suite through the instruction given in the ReadMe file [here](https://github.com/mlcommons/inference/tree/master/language/bert). I want to run my experiments inside a docker container. I followed the steps:

1. `$ make setup # to initialize submodule, download datasets, and download models.`
2. `$ make build_docker # to build docker image.`
3. `$ make launch_docker #  to launch docker container with an interaction session.`
4. Within the docker container shell: 
                  `$ python3 run.py --backend=onnxruntime --scenario=SingleStream --accuracy`
When I try to launch the `run.py` script, I monitor the CPU and GPU.
The `htop` shows a maxed-out CPU, with 0% GPU utilization (`nvtop`), after the following line is written out to the console:
`Running LoadGen test...`
![image](https://github.com/mlcommons/inference/assets/43134757/184b3d30-211b-487b-b1a1-b693ca25f71b)
_The output of htop_
 
![image](https://github.com/mlcommons/inference/assets/43134757/b16762d6-d197-4880-9065-c3f64c703732)
_The output of nvtop_

The `nvtop` does not even show a process using the GPU. When I am using the GPU and loading the onnx model, the model should be loaded onto the GPU, if I am not wrong.

![image](https://github.com/mlcommons/inference/assets/43134757/4e458f44-a8e0-49a0-8a2d-2bc5c1c1c663)
_The code modification to check whether GPU is used or not_

I just added a print statement to check the context which the onnx model loading is using, GPU or CPU. But I see, it shows that it is using GPU (based on the print statement)
```
...
Loading ONNX model...
Using GPU
2023-07-30 11:06:04.757505473 [W:onnxruntime:, graph.cc:2413 CleanUnusedInitializers] Removing initializer 'bert.pooler.dense.weight'. It is not used by any node and should be removed from the model.
2023-07-30 11:06:04.757578318 [W:onnxruntime:, graph.cc:2413 CleanUnusedInitializers] Removing initializer 'bert.pooler.dense.bias'. It is not used by any node and should be removed from the model.
Constructing SUT...
Finished constructing SUT.
Constructing QSL...
Loading cached features from 'eval_features.pickle'...
Finished constructing QSL.
Running LoadGen test...
```
What might be the issue?

--------

This is contrary to what happens when I try to use the [Resnet-50 model on ImageNet validation dataset](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection). 

There, the moment I execute the script, it occupies memory and shows some activity on the GPU:
![image](https://github.com/mlcommons/inference/assets/43134757/c2541678-0af3-4283-a410-40b342e45b0f)


It is not nice, when you are a bit sloppy. :(
How was I sloppy? I was checking whether the system is using CPU or GPU just based on `if` condition which was getting fired.
The correct way was to find the device which the `onnxruntime` was using as follows:
![image](https://github.com/mlcommons/inference/assets/43134757/4b619ac4-0c3a-4764-af2e-b2ee80381a8e)
After this I saw that even if the `if` condition was hitting the GPU case, but the actual device was `CPU`:
```
...
Loading ONNX model...
Using GPU
2023-07-30 13:47:50.148816667 [W:onnxruntime:, graph.cc:2413 CleanUnusedInitializers] Removing initializer 'bert.pooler.dense.weight'. It is not used by any node and should be removed from the model.
2023-07-30 13:47:50.148898101 [W:onnxruntime:, graph.cc:2413 CleanUnusedInitializers] Removing initializer 'bert.pooler.dense.bias'. It is not used by any node and should be removed from the model.
CPU                                                 <------------------------------------------------------- Device is CPU
Constructing SUT...
Finished constructing SUT.
Constructing QSL...
Loading cached features from 'eval_features.pickle'...
Finished constructing QSL.
Running LoadGen test...
```

The solution to this issue lies in the `Dockerfile`:
![image](https://github.com/mlcommons/inference/assets/43134757/82ac056b-2316-481a-92ac-093b9f487e50)

The Dockerfile was using simply `onnxruntime`. But it should have been `onnxruntime-gpu`
And this fixes the issue.