Repository: mlcommons/inference
Issue: Unable to reproduce the SQuAD v1.1 result with the BERT-Large PyTorch model.
Root cause: Algorithmic Error

I tried to reproduce the SQuAD v1.1 f1 score of 90.874% for the BERT-Large PyTorch model following [this instruction](https://github.com/mlcommons/inference/tree/master/language/bert#commands).
(+ upgrading 'transformers' to 4.0.0 to avoid the following error.
```
Traceback (most recent call last):
  File "run.py", line 120, in <module>
    main()
  File "run.py", line 101, in main
    lg.StartTestWithLogSettings(sut.sut, sut.qsl.qsl, settings, log_settings)
  File "/workspace/pytorch_SUT.py", line 70, in issue_queries
    start_scores = model_output.start_logits
AttributeError: 'tuple' object has no attribute 'start_logits'
Makefile:149: recipe for target 'run_pytorch_accuracy' failed
```
)

But, I am getting the F1 score of less than 90 only as follows.
```
$ make run_pytorch_accuracy
...
Reading examples...
Loading cached features from 'eval_features.pickle'...
Loading LoadGen logs...
Post-processing predictions...
Writing predictions to: build/result/predictions.json
Evaluating predictions...
{"exact_match": 82.26111636707664, "f1": 89.91125746620186}
```
I'd appreciate if you could let me know any clue or solution to the 90.874 F1 score.
I have no problem with the TensorFlow BERT-Large model.

Which GPU are you using? Are you using NVIDIA GPUs? If so, could you try running on CPU and see if the accuracy is recovered?

> Which GPU are you using? Are you using NVIDIA GPUs? If so, could you try running on CPU and see if the accuracy is recovered?

That result {"exact_match": 82.26111636707664, "f1": 89.91125746620186} was from NVIDIA Titan XP.
I'm getting similar values of {"exact_match": 82.21381267738883, "f1": 89.85617167730943} with NVIDIA A100 and
{"exact_match": 82.081362346263, "f1": 89.94336020017377} with CPU (by installing a CPU version of PyTorch).
Slightly different scores from different hardwares, but all of them are still lower than the expected F1 score, 90.874.

Hmm that's weird. Could you try with `transformers==2.4.0` which is what we tested before?

also cc @pgmpablo157321 

> Hmm that's weird. Could you try with `transformers==2.4.0` which is what we tested before?
> 
> also cc @pgmpablo157321

As I said before, I couldn't run the [PyTorch inference code](https://github.com/mlcommons/inference/blob/9e89b395ddf6d823c8804d9a00afff26c67a06b9/language/bert/pytorch_SUT.py) without the following error in the provided docker environment with transformers==2.4.0.
```
Traceback (most recent call last):
  File "run.py", line 121, in <module>
    main()
  File "run.py", line 102, in main
    lg.StartTestWithLogSettings(sut.sut, sut.qsl.qsl, settings, log_settings)
  File "/workspace/pytorch_SUT.py", line 71, in issue_queries
    start_scores = model_output.start_logits
AttributeError: 'tuple' object has no attribute 'start_logits'
```
So, I modified [two lines](https://github.com/mlcommons/inference/blob/9e89b395ddf6d823c8804d9a00afff26c67a06b9/language/bert/pytorch_SUT.py#L70) in [pytorch_SUT.py](https://github.com/mlcommons/inference/blob/9e89b395ddf6d823c8804d9a00afff26c67a06b9/language/bert/pytorch_SUT.py) as follows:
``` Python
                # Before modification
                #start_scores = model_output.start_logits
                #end_scores = model_output.end_logits

                # After modification
                start_scores = model_output[0]
                end_scores = model_output[1]
```
and got the following numbers:
```
{"exact_match": 82.05298013245033, "f1": 89.79477285675065}
```

I see. Maybe @pgmpablo157321 can take a look what happened.

Thanks. I'm looking forward to it.

@yschoi-dev Hi sorry for taking so long, I was busy with other stuff. First, it seems that this way of accessing the model's inputs and outputs is wrong for version 2.4.0.
https://github.com/mlcommons/inference/blob/078ac2de2548f664b40cc4513b61441e132d16fb/language/bert/pytorch_SUT.py#L70-L71
The `model_output` variable for this version is a tuple, and they should be accessed like:
```
start_scores = model_output[0]
end_scores = model_output[1]
```
However, even with this `transformers` version, I still can't get the accuracy claimed in the readme file. Here are two runs I made with a Tesla P4 GPU and the pytorch model. 

Using docker:
![screenshot-docker](https://user-images.githubusercontent.com/30699846/158928973-d1b1c037-ebd0-4a54-8a8a-bd98ba67ec59.png)

Without docker:
![screenshot-no-docker](https://user-images.githubusercontent.com/30699846/158929075-c4799257-afbd-4dc9-b178-42db01285e24.png)

It does vary a little bit but not a lot. @nvpohanh do you know who implemented this? Or what GPU they used?
I'll inspect the model further to see if there is something we are missing there.

@pgmpablo157321 Thank you for the response.
Interestingly, I can get the F1 score much closer to one in the README (i.e. 90.784) when evaluating using the Hugging Face transformers' [QA inference script](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py) with the same checkpoint file from the MLPerf.
<img width="836" alt="image" src="https://user-images.githubusercontent.com/77429674/158933868-a67b1e44-e988-4a39-b30c-bc553044961a.png">

I used the master branch in the transformers' github (commit [8f3ea7a1e1a85e80210b3d4423b674d9a61016ed](https://github.com/huggingface/transformers/commit/8f3ea7a1e1a85e80210b3d4423b674d9a61016ed)) and NVIDIA TITAN Xp.

> It does vary a little bit but not a lot. @nvpohanh do you know who implemented this? Or what GPU they used?

I was the one who implemented it. I believe I was using TitanRTX, but that should not change the accuracy.

> Interestingly, I can get the F1 score much closer to one in the README (i.e. 90.784) when evaluating using the Hugging Face transformers' [QA inference script](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py) with the same checkpoint file from the MLPerf.

That's a good finding. This makes me think that there must be an issue in our accuracy checking script. I am wondering if it was broken by #1047 and/or #951 . 

I will spend some timing dubugging this when I have a chance. Before then, would be great if someone can try reverting #951 and #1047 and see if the accuracy comes back.

I ran some experiments reverting those commits. I get the expected metric when running `run.py` from this [commit](https://github.com/mlcommons/inference/commit/1f53d40aecd15b412d9eb16e4382520746c1549f) (the parent from #951):

![screenshot-commit-1f53d40aecd15b412d9eb16e4382520746c1549f ](https://user-images.githubusercontent.com/30699846/159326200-5dd85dcb-bacf-4a71-a5b9-f0b8b4f55fee.png)

I get this error when I run from this other [commit](https://github.com/mlcommons/inference/commit/0afaa3807a3d2d182310b6fd15f592d4e98886cc) (the parent from #1047):

![screenshot-commit-0afaa3807a3d2d182310b6fd15f592d4e98886cc](https://user-images.githubusercontent.com/30699846/159327355-649f9317-921c-4727-b94d-a43546fce8f8.png)

Finally, I get this metric when I run the script from this other [commit](https://github.com/mlcommons/inference/commit/5ec3ac922556107ce0f6ca63c175d379017ba3d8) (this belongs to #1047)

![screenshot-commit-5ec3ac922556107ce0f6ca63c175d379017ba3d8](https://user-images.githubusercontent.com/30699846/159331492-58746600-6763-41e4-a5f7-ad08c718dae5.png)



@pgmpablo157321 I recently got the same error (`No module named 'tokenization'`). It turned out that for whatever reason the `DeepLearningExamples` submodule [here](https://github.com/mlcommons/inference/tree/master/language/bert) was missing from my checkout...

Some updates:
I also ran the script from this [commit](https://github.com/mlcommons/inference/commit/73e0e81f8c56b29b89ba04ecc110ee7fa1053e9f) that belongs to #951 and got initially the same error:
```
Traceback (most recent call last): 
File "/workspace/accuracy-squad.py", line 30, in <module> import tokenizationModuleNotFoundError: No module named 'tokenization'
Traceback (most recent call last): 
File "run.py", line 107, in <module> main() File "run.py", line 96, in main subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.6/subprocess.py", line 311, in check_call raise CalledProcessError(retcode, cmd)subprocess.CalledProcessError: Command 'python3 /workspace/accuracy-squad.py ' returned non-zero exit status 1.
```
Then I added this line:
```
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "DeepLearningExamples", "TensorFlow", "LanguageModeling", "BERT"))
```
And got this:
```
Evaluating predictions...
{"exact_match": 82.18543046357615, "f1": 89.88174847655135}
Done!
Destroying SUT...
Destroying QSL...
Finished destroying SUT.
Finished destroying QSL.
```
It seems that something was introduced in #951 that changed the accuracy or how it was measured.

cc @LukeIreland1 for visibility since his changes were the cause

Adding these two lines seems to solve the metric divergence problem:
```
self.model.eval()
self.model.load_state_dict(torch.load("build/data/bert_tf_v1_1_large_fp32_384_v2/model.pytorch"), strict=True)
```
Currently, it is like this:
https://github.com/mlcommons/inference/blob/078ac2de2548f664b40cc4513b61441e132d16fb/language/bert/pytorch_SUT.py#L55
I'll make a PR for this fix