Repository: mlcommons/inference
Issue: Running RNNT using CMR fails because of missing accuracy.txt
Root cause: Configuration Error

Running this RNNT benchmark command (based on [this README](https://github.com/mlcommons/inference/blob/e39003a9c4c89a2215db0ca57ad7a57b16f9a785/speech_recognition/rnnt/README_cm.md) in the RNNT directory and on [this working command from the BERT directory](https://github.com/mlcommons/inference/blob/e39003a9c4c89a2215db0ca57ad7a57b16f9a785/language/bert/README.md?plain=1#L61) :

```
cm run script --tags=run,mlperf,inference,generate-run-cmds,_submission,_short \
        --quiet --submitter="Community" \
        --hw_name=default \
        --implementation=reference \
        --model=rnnt\
        --backend=pytorch\
        --device=cuda \
        --scenario=Offline \
        --test_query_count=10000
```

gets to the end of the run and then fails with error:

```
python3 '/root/CM/repos/local/cache/6695c95780014f10/inference/tools/submission/truncate_accuracy_log.py' --input '/root/mlperf_submission' --submitter 'Community' --backup '/root/mlperf_submission_logs'
ERROR:main:open/Community/results/default-reference-gpu-pytorch-v1.13.0-default_config/rnnt/offline/accuracy/accuracy.txt missing, generate to continue
ERROR:main:open/Community/results/default-reference-gpu-pytorch-v1.13.0-default_config/rnnt/offline/accuracy/accuracy.txt missing, generate to continue
ERROR:main:no submission in open/Community/compliance
INFO:main:Make sure you keep a backup of /root/mlperf_submission_logs in case mlperf wants to see the original accuracy logs
* cm run script "submission inference checker mlc _short-run"
  * cm run script "get python3"
  * cm run script "get mlcommons inference src"
  * cm run script "get generic-python-lib _xlsxwriter"
  * cm run script "get generic-python-lib _pandas"
/root/cm/bin/python3 /root/CM/repos/local/cache/6695c95780014f10/inference/tools/submission/submission_checker1.py --input /root/mlperf_submission --submitter Community  
[2024-02-29 00:13:14,797 submission_checker1.py:2877 ERROR] open/Community/results/default-reference-gpu-pytorch-v1.13.0-default_config/rnnt/offline/accuracy has no accuracy.txt. Generate it with accuracy-imagenet.py or accuracy-coco.py o
r process_accuracy.py                                      
```

I tried to create the file `accuracy.txt` by writing the output of `accuracy_eval.py` into it. I put it in the `mlperf_submission` folder in the `mlperf_submission/*...*/accuracy` subdirectory as asked for in the error above, but when I rerun the mlperf command, it still fails.

Could anyone help me fix this issue? Thank you.

Hi @kevinnegy Can you please do this change in your CM script and try a rerun of the same command?  The file to be edited is this: `$HOME/CM/repos/ctuning@mlcommons-ck/cm-mlops/app-mlperf-inference/_cm.yaml`

https://github.com/mlcommons/ck/pull/1142/files#diff-aca162ab4c82668036c3ce7d596b83b7e2f003c35962150d928676e36effa806L510

Hi @arjunsuresh, that fixed it! Thank you! Does this mean I had an outdated version of CM? I have issues running a couple other benchmarks in mlperf inference with CM (e.g. `import cv2` error for the vision benchmarks). Should I try to update CM or do I need to open a separate issue (perhaps in the CM repo) for this?  Thank you!

Great. 
No, actually this was a bug and the fix is yet to be merged in CM. But at any time you can just do `cm pull repo mlcommons@ck` to update the CM repo for MLPerf - it just pulls in the latest changes.

`import cv2` error should normally don't happen unless somehow the installed pip package got removed. `cm rm cache --tags=generic-python-lib,_opencv-python` will clear the stale cache in case something got removed.