Repository: mlcommons/inference
Issue: unnecessary import torch in inference/language/bert/accuracy_squad.py for BERT benchmark.
Root cause: Validation Gap

Hi,

I am testing on a TensorFlow only enviroment with no pytorch and I see run time failures because of this import torch statement.
https://github.com/mlcommons/inference/blob/58025f8fde679ea864d34f96ecc9f14bf70ece53/language/bert/accuracy-squad.py#L34

I used python3 run.py --backend=tf --scenario=SingleStream --accuracy

Thanks

Does the PyT path work if that line is removed? If so, please feel free to file a PR to remove it. Thanks!

Pytorch path has another issue, it seems like it's using the TensorfLow directory. Should I raise a new issue ?
This import is calling the TensorFlow/tokenization.py instead of the PyTorch/tokenization.py
https://github.com/mlcommons/inference/blob/58025f8fde679ea864d34f96ecc9f14bf70ece53/language/bert/create_squad_data.py#L28

There is also an import tensorflow as tf  which will again break for a pytorch only build here
https://github.com/mlcommons/inference/blob/58025f8fde679ea864d34f96ecc9f14bf70ece53/language/bert/create_squad_data.py#L30

aarch64_pytorch bert> python3 run.py --backend=pytorch --scenario=SingleStream --accuracy
Traceback (most recent call last):
  File "run.py", line 25, in <module>
    from squad_QSL import get_squad_QSL
  File "/home/ubuntu/inference/language/bert/squad_QSL.py", line 23, in <module>
    from create_squad_data import read_squad_examples, convert_examples_to_features
  File "/home/ubuntu/inference/language/bert/create_squad_data.py", line 28, in <module>
    import tokenization
  File "/home/ubuntu/inference/language/bert/DeepLearningExamples/TensorFlow/LanguageModeling/BERT/tokenization.py", line 26, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

Yeah, I have been slightly lazy and assumed that both TF and PyT exist.... 

Could you try these files instead and see they work? If they do, we may want to make them the reference:
- https://github.com/mlcommons/inference_results_v0.7/tree/master/closed/NVIDIA/code/bert/tensorrt/helpers
- https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/code/bert/tensorrt/accuracy-bert.py
- https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/code/bert/tensorrt/evaluate.py


Could you let me know which files I should replace ? It's confusing.
Should I replace accuracy-squad.py by accuracy-bert.py ?
Should I replace bert/DeepLearningExamples/PyTorch/Detection/SSD/src/evaluate.py or bert/evaluate-v1.1.py ?
There are several tokenization.py file

aarch64_pytorch bert> find -name tokenization.py
./DeepLearningExamples/PyTorch/LanguageModeling/BERT/tokenization.py
./DeepLearningExamples/TensorFlow/LanguageModeling/BERT/tokenization.py
./DeepLearningExamples/TensorFlow/LanguageModeling/BERT/trt/helpers/tokenization.py

But there are also, other import tensorflow or usage of tensorflow in create_squad.py or tf_SUT.py

Replace accuracy-squad.py with https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/code/bert/tensorrt/accuracy-bert.py 

And it includes:
```
from code.bert.tensorrt.evaluate import f1_score, exact_match_score
from code.bert.tensorrt.helpers.data_processing import get_predictions, read_squad_json, convert_example_to_features
from code.bert.tensorrt.helpers.tokenization import BertTokenizer
from code.common import logging
```
Replace the above with:
```
from evaluate import f1_score, exact_match_score
from data_processing import get_predictions, read_squad_json, convert_example_to_features
from tokenization import BertTokenizer
import logging
```
where 

- `evaluate` is https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/code/bert/tensorrt/evaluate.py
- `data_processing` is https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/code/bert/tensorrt/helpers/data_processing.py
- `tokenization` is https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/code/bert/tensorrt/helpers/tokenization.py

So just put these three in the same directory as accuracy-bert.py 

in squad_QSL.py this line that will look for the tensorflow directory even with python3 run.py --backend=pytorch
I think it should be changed.
sys.path.insert(0, os.path.join(os.getcwd(), "DeepLearningExamples", "TensorFlow", "LanguageModeling", "BERT"))

But even if I remove this line and use your tokenization.py file, create_squad_data.py is later called and this file seems to use tensorflow.

@nvpohanh Is there any update to decouple TF and PyTorch builds?

I don't currently have plan to do that. To work around the issue, you can install pytorch in tf container and vice versa. 

I run into this issue when trying to use both pytorch and tf
```
python3 run.py --backend=pytorch --scenario=SingleStream --max_examples=1
Traceback (most recent call last):
  File "run.py", line 107, in <module>
    main()
  File "run.py", line 53, in main
    from pytorch_SUT import get_pytorch_sut
  File "/home/ubuntu/inference/language/bert/pytorch_SUT.py", line 25, in <module>
    import torch
  File "/home/ubuntu/python3-venv/lib/python3.7/site-packages/torch/__init__.py", line 195, in <module>
    _load_global_deps()
  File "/home/ubuntu/python3-venv/lib/python3.7/site-packages/torch/__init__.py", line 148, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/local/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/ubuntu/python3-venv/lib/python3.7/site-packages/torch/lib/libgomp-d22c30c5.so.1: cannot allocate memory in static TLS block
```

Issue resolved with PR https://github.com/mlcommons/inference/pull/951