Repository: mlcommons/inference
Issue: BERT: failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
Root cause: Environment Incompatibility

[NVIDIA's new BERT contribution](https://github.com/mlperf/inference/tree/master/v0.7/language/bert) is very neat!

Unfortunately it failed to run with TensorFlow in the Offline scenario on an HP workstation with a GTX1080 GPU:

```bash
anton@velociti:/datasets/mlperf-bert/bert$ make launch_docker
 
================
== TensorFlow ==
================
 
NVIDIA Release 19.10 (build 8471601)
TensorFlow Version 1.14.0
 
Container image Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
Copyright 2017-2019 The TensorFlow Authors.  All rights reserved.
 
Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.
 
NOTE: MOFED driver for multi-node communication was not detected.
     Multi-node communication performance may be reduced.
 
(mlperf) anton@mlperf-inference-bert-anton:/workspace$ python3 run.py --backend=tf --scenario=Offline
...
2020-04-30 11:42:47.386193: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7533 MB memory) -> physical GPU (device: 0
, name: GeForce GTX 1080, pci bus id: 0000:02:00.0, compute capability: 6.1)
2020-04-30 11:42:57.015515: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.
If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not vi
a TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
2020-04-30 11:42:58.438500: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-04-30 11:43:01.548433: E tensorflow/stream_executor/cuda/cuda_blas.cc:250] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
...
2020-04-30 11:43:01.882717: W tensorflow/stream_executor/stream.cc:1916] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
 File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call
   return fn(*args)
 File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
   options, feed_dict, fetch_list, target_list, run_metadata)
 File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
   run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
 (0) Internal: Blas GEMM launch failed : a.shape=(384, 1024), b.shape=(1024, 1024), m=384, n=1024, k=1024
        [[{{node bert/encoder/layer_0/attention/self/value/MatMul}}]]
        [[Reshape_1/_2373]]
 (1) Internal: Blas GEMM launch failed : a.shape=(384, 1024), b.shape=(1024, 1024), m=384, n=1024, k=1024
        [[{{node bert/encoder/layer_0/attention/self/value/MatMul}}]]
0 successful operations.
0 derived errors ignored.
```

[Full log](https://pastebin.com/Yu4gCNXK).

/cc @nvpohanh

It appears to be running with ONNX:
```
(mlperf) anton@mlperf-inference-bert-anton:/workspace$ python3 run.py --backend=onnxruntime --scenario=Offline
...
Converting examples to features...
Constructing QSL...
Finished constructing QSL.
Finished constructing SUT.
Running Loadgen test...
```
but not using the GPU:
```
Thu Apr 30 12:11:36 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64       Driver Version: 440.64       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:02:00.0 Off |                  N/A |
| 25%   37C    P0    36W / 180W |      0MiB /  8117MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
```

Here's the output with ONNX:
```
(mlperf) anton@mlperf-inference-bert-anton:/workspace$ python3 run.py --backend=onnxruntime --scenario=Offline

================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : Performance
Samples per second: 0.897283
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes

================================================
Additional Stats
================================================
Min latency (ns)                : 1209996834
Max latency (ns)                : 2353771756198
Mean latency (ns)               : 1159909911223
50.00 percentile latency (ns)   : 1160382794479
90.00 percentile latency (ns)   : 2095698616088
95.00 percentile latency (ns)   : 2220531197517
97.00 percentile latency (ns)   : 2273992115648
99.00 percentile latency (ns)   : 2327542666488
99.90 percentile latency (ns)   : 2351674363082

================================================
Test Parameters Used
================================================
samples_per_query : 2112
target_qps : 32
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 60000
max_duration (ms): 0
min_query_count : 1
max_query_count : 0
qsl_rng_seed : 3133965575612453542
sample_index_rng_seed : 665484352860916858
schedule_rng_seed : 3622009729038561421
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
print_timestamps : false
performance_issue_unique : false
performance_issue_same : false
performance_issue_same_index : 0
performance_sample_count : 10833

No warnings encountered during test.

No errors encountered during test.
Done!
```

With PyTorch:
```
(mlperf) anton@mlperf-inference-bert-anton:/workspace$ python3 run.py --backend=pytorch --scenario=Offline --accuracy
...
Loading BERT configs...
Loading PyTorch model...
Constructing SUT...
Creating tokenizer...
Reading examples...
WARNING:tensorflow:From /workspace/DeepLearningExamples/TensorFlow/LanguageModeling/BERT/utils/create_squad_data.py:154: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.

Converting examples to features...
Constructing QSL...
Finished constructing QSL.
Finished constructing SUT.
Running Loadgen test...
...
```
the GPU is used properly:
```
Thu Apr 30 13:16:03 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64       Driver Version: 440.64       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:02:00.0 Off |                  N/A |
| 54%   82C    P2   147W / 180W |   1939MiB /  8117MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0    111197      C   python3                                     1929MiB |
+-----------------------------------------------------------------------------+
```
... and to good effect:
```
No warnings encountered during test.

No errors encountered during test.
...
Creating tokenizer...
Reading examples...
WARNING:tensorflow:From /workspace/DeepLearningExamples/TensorFlow/LanguageModeling/BERT/utils/create_squad_data.py:154: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.

Converting examples to features...
Loading loadgen logs...
Post-processing predicions...
Writing predictions to: build/result/predictions.json
Evaluating predictions...
{"exact_match": 83.68968779564806, "f1": 90.87487229720105}
Done!
```
which I'd round to `f1=90.875%`.

Thanks for trying the codes!

**TF:** 
I am not very sure what's going on, but after some research, it seems that TF always have issues with cuBLAS init. Could you try some fixes suggested by online resources? Like this one: https://stackoverflow.com/questions/38303974/tensorflow-running-error-with-cublas or this one: https://github.com/tensorflow/tensorflow/issues/9489

**onnx-runtime:**
GPU version of onnxruntime does not support Quant/Dequant ops, so I had to use CPU-only. I think we might be able to run on GPU with the non-quantized onnx model. Please feel free to add that support.


Thanks Po-Han! I'll try nudging TF to do the right thing.

The quantized version definitely runs on the CPU, and, boy, it does so slowly:
```
real    407m53.752s
user    2445m28.928s
sys     13m6.388s
```
(It took almost 7 hours on an admittedly old but 20 virtual-core CPU.)

By the way, the resulting score is only `f1=90.067` (99.11% of the reference):
```
Converting examples to features...
Loading loadgen logs...
Post-processing predicions...
Writing predictions to: build/result/predictions.json
Evaluating predictions...
{"exact_match": 82.81929990539263, "f1": 90.06671623731357}
Done!
```
while the [table](https://github.com/mlperf/inference/tree/master/v0.7/language/bert#supported-models) states `f1_score=90.482%` (99.57% of the reference). Any idea about the discrepancy?

Hmmm I will look into the accuracy discrepancy.... @guschmue Would you be the right person to ask if I have question about onnxruntime? 

@psyhtest Could you do me a favor? The master of onnxruntime seems to support GPU-accelerated Quantize/Dequantize now: https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/contrib_ops/cuda/quantize_ops.cc

Could you try the master build of onnxruntime and try running it on GPU? Thanks

Thanks @nvpohanh. I've opened a new issue https://github.com/mlperf/inference/issues/594 to track the accuracy.

I'm not sure how to enable the GPU. @guschmue?

```
pip install onnxrunime-gpu
``` 
gets you cuda support.

If you need the master you could build from source 
or try a nightly build from pypi (last was 4/25):
```
pip install --index-url https://test.pypi.org/simple/ ort-gpu-nightly
```


4/25 nightly should be worth a try. I saw that that file was added two weeks ago.

ok. If that doesn't work, let me know - I can create a wheel with the latest.

Thanks! This has enabled the GPU:
```
$ pip install --index-url https://test.pypi.org/simple/ ort-gpu-nightly --user
$ python3 run.py --backend=onnxruntime --scenario=Offline --quantized
```
although seemed slower than the non-quantized model (14 QPS vs 18 QPS)...

How big is the dataset by the way?

quantized version has Quant/Dequant ops, so it's normal that it is slower.

SQuAD v1.1 test set has 10~11k sequences, if I remember it correctly.

A-ha, it's probably the same as:
```
performance_sample_count : 10833
```
Should take 10-13 minutes on the GTX1080 with ONNX runtime then.

Apologies for converting this thread into a Q&A session :)