zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Pytorch DLRM reference scripts failing in accuracy mode 
Root cause: Environment Incompatibility

Dear All,
We are trying to run inference reference scripts for pytorch DLRM (recommendation) in accuracy mode using mlperf official model (tb00_40M.pt). In order to check accuracy, we are using day_23 dataset of Criteo Terabyte datasets (https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf/), as given on https://github.com/mlcommons/inference/tree/master/recommendation/dlrm/pytorch.

We are finding that this dataset accesses embedding tables beyond their limits, thereby causing the reference scripts to fail. 
Any help on how to run DLRM to check accuracy using Criteo Terabyte data will be highly appreciated.

Thanks in advance.  

Hi @sudarshan-shinde are you using only day_23 dataset (no download of other days)?

> Hi @sudarshan-shinde are you using only day_23 dataset (no download of other days)?

Yes, since we are running only inference.



Unfortunately that is not possible using the current scripts. We need to download all 24 days dataset even though we use only half a day's dataset for inference. This issue is expected to be solved with DLRM 2 from 3.1 round but as of now I do not think there is an easier way out. Downloading and preprocessing of the 24 day's dataset will need about 3 TB of disk space and running the model needs about 200 GB of RAM - not something that can be easily tried on a workstation.

How can pre-processing all 24 day's data will help is not clear to me. If certain samples in day_23 data are such that their categorical features go beyond embedding table limits, then how pre-processing all 24 days data can help? Kindly elaborate.  

Unfortunately I haven't checked that and hence can't answer your query. But downloading the whole dataset was the way out for us when we faced this issue. May be someone familiar with DLRM model can answer that - or you can bring up this issue in Tuesdays inference WG meeting. 

@TheKanter Is it possible for mlcommons to share the pre-processed files with members?

Due to licensing constraints from Criteo, this request cannot be met.

I believe this is same issue https://github.com/mlcommons/inference/issues/1303

Using `torch==1.10` and `numpy==1.19` should fix the error.
Repository: mlcommons/inference
Issue: Query: How to run BERT INT8 TF model  
Root cause: Documentation Deficiency

Hi,

I am trying to run BERT INT8 with TF backend. However, I don't see TF INT8 model info in below link.  
https://github.com/mlcommons/inference/tree/master/language/bert

Any help on how to run will be highly appreciated.

Not all quantized models are added in the inference repo. You're free to take any fp32 model and do your own quantization method. More details regarding retraining can be found [here](https://github.com/mlcommons/inference_policies/blob/master/inference_retraining_rules.adoc#6-model)
Repository: mlcommons/inference
Issue: Simpler download script for openimages
Root cause: Unimplemented Feature Gap

The current [script](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/tools/openimages.py) to download openimages is really resource heavy and it needs mongodb database which unfortunately is not easy to set up on many platforms including Apple M1, RHEL 9, Ubuntu 22.04 etc. Just for downloading a dataset do we really need this?

Though this is not a big issue currently the script for openimages is installing the below python packages
```
Installing collected packages: joblib, threadpoolctl, scipy, scikit-learn, tzdata, backports.zoneinfo, 
pytz-deprecation-shim, tzlocal, pytz, dill, Pillow, six, retrying, ndjson, patool, argcomplete, python-dateutil, imageio, 
networkx, PyWavelets, tifffile, scikit-image, future, sortedcontainers, opencv-python-headless, glob2, tabulate, 
voxel51-eta, xmltodict, pymongo, motor, sniffio, anyio, starlette, sse-starlette, tenacity, plotly, dacite, pprintpp, 
cycler, contourpy, kiwisolver, fonttools, matplotlib, wrapt, Deprecated, aiofiles, h11, httpcore, rfc3986, httpx, 
universal-analytics-python3, kaleido, psutil, pandas, pygments, graphql-core, python-multipart, typing-extensions, 
sentinel, backports.cached-property, click, strawberry-graphql, MarkupSafe, Jinja2, fiftyone-db, wsproto, priority, 
toml, hpack, hyperframe, h2, hypercorn, fiftyone-brain, mongoengine, greenlet, dnspython, eventlet, jmespath, 
botocore, s3transfer, boto3, sseclient-py, fiftyone
```

Thank you @pgmpablo157321 for looking into this. We have the instructions to manually download the openimages dataset [here](https://storage.googleapis.com/openimages/web/download_v7.html). What's missing is a way to convert that to coco format. If this is not easy to do (without fiftyone package) what we can do is to provide a way to directly download this dataset in coco format.
Repository: mlcommons/inference
Issue: BERT model int8 comparison
Root cause: Documentation Deficiency

Can onnx（from tf） and torch（from hugging） match the corresponding model under the int8 model? The operators of the last few layers of the onnx model in the current int8 mode and the torch model are completely inconsistent with the final model output? Inconvenient to use and performance analysis comparison

@lixiaolx  Can you provide more details on the differences?

Not relevant anymore, closing issue
Repository: mlcommons/inference
Issue: Wrong accuracy for onnxruntime reference implementation of resnet50 on aarch64
Root cause: Environment Incompatibility

Running the onnxruntime reference implementation gives zero accuracy on aarch64 architecture. This is shown in [one of the 2.1 round submissions](https://github.com/mlcommons/inference_results_v2.1/blob/master/open/OctoML/results/apple-m1-onnxruntime-cpu/resnet50/offline/accuracy/accuracy.txt) too.

Fixed now.
Repository: mlcommons/inference
Issue: Vision example doesn't work
Root cause: Documentation Deficiency

The vision example does not work and the documentation is not up to date with the repository.

The example is the following: https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/GettingStarted.ipynb

Does it start by referencing a non-existing folder `v0.5`?

<img width="434" alt="image" src="https://user-images.githubusercontent.com/33181424/197813536-65ad5715-f945-426f-8b5a-a19409fe36d9.png">

>__Note__: This instruction is pointing to an old repository that has been dead for 2 years. 

It also breaks when I try to build the docker container by running the command 
```bash
!./run_and_time.sh onnxruntime mobilenet cpu
```

I fixed this by adding  the following line before the pip installs.

```dockerfile
RUN pip install --upgrade pip setuptools wheel
```

When I fixed this the image was built correctly but the sh file stopped working.
The following error appears:
```bash
main.py: error: unrecognized arguments: --queries-offline 20
```
This is probably due to an outdated parameter that was never changed.

I removed the extra parameters from the command so I could run it at least but the following error appeared as well:

```bash
  File "/mlperf/python/imagenet.py", line 56, in __init__
    CNT = count if count <= self.count else self.count
TypeError: '<=' not supported between instances of 'int' and 'NoneType'
```




The docker part of the example is not the only one with wrong instructions. The local part doesn't work as well.

For example, after running this command:

```bash
!cd ../../loadgen; CFLAGS="-std=c++14" python setup.py develop; cd {root}
!python setup.py develop
```
The following error appears:

<img width="1042" alt="image" src="https://user-images.githubusercontent.com/33181424/197843107-d4c2f25e-161c-43a1-bf50-5305af46e861.png">

Probably, the are additional installation packages that are not contemplated in the example instructions.


Looking at this issue, it seems there are two problems.

1. The notebook is outdated. This seems easy to fix with a couple of small changes I just included in #1267 
2. Running the benchmark using docker seems to give another error. I am not sure exactly what is happening here, but I'll take a look

Thank you for your quick response. I will wait for the PR to be completed
Repository: mlcommons/inference
Issue: Inference v0.5 software verification notes

Latest verified:
date 5/15
commit: fd037e6b008cc5e42e833116661474a3fcefe975

tested commands:
under cloud/image_classification
./run_local.sh tf cpu --time 100
./run_local.sh tf gpu --time 100

--time 100 allows you to run a shorter version of the code

what to expect:
- Single stream mode working end to end on Resnet 50. Latency results reported in mlperf_log_summary.txt, which seems to be correct.
- Seeing QPS printouts for SingleStream, MultiStream, Server printouts in the CLI, with latency at 0.01 to 0.4 for each scenario.
- No MultiStream or Server result reporting from LoadGen yet. 
- The program exits properly.
- No SSD-mobilenet code has been tested.
Repository: mlcommons/inference
Issue: Classification via Docker on CPU fails
Root cause: Environment Incompatibility

On r2.1, the Docker container run fails as shown:

```
$ MODEL_DIR=$PWD DATA_DIR=$HOME/data/ILSVRC2012_img_val ./run_and_time.sh tf mobilenet cpu
.
.
.
Collecting onnx
  Downloading https://files.pythonhosted.org/packages/2c/6a/39b0580858589a67c3322aabc2634f158391ffbf98fa410127533e7f1495/onnx-1.12.0.tar.gz (10.1MB)
    Complete output from command python setup.py egg_info:
    fatal: Not a git repository (or any of the parent directories): .git
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-v2jvnd2z/onnx/setup.py", line 81, in <module>
        assert CMAKE, "Could not find cmake executable!"
    AssertionError: Could not find cmake executable!
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-v2jvnd2z/onnx/
The command '/bin/sh -c pip install future pillow onnx opencv-python-headless tensorflow onnxruntime' returned a non-zero code: 1
Unable to find image 'mlperf-infer-imgclassify-cpu:latest' locally
docker: Error response from daemon: pull access denied for mlperf-infer-imgclassify-cpu, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
```


By the way, I do have _cmake_ installed:

```
$ which cmake
/usr/bin/cmake
$ 
```

Prepending the above command line with `CMAKE=/usr/bin/cmake` still results in the same error.

mobilenet is not a supported benchmark in r2.1.

I get the same error with resnet50.

@pgmpablo157321 Could you help?
Repository: mlcommons/inference
Issue: Classification via Docker on GPU fails
Root cause: Environment Incompatibility

On r2.1, the Docker container run fails as shown:

```
$ MODEL_DIR=$PWD DATA_DIR=$HOME/data/ILSVRC2012_img_val ./run_and_time.sh tf mobilenet gpu
Sending build context to Docker daemon  424.4kB
Step 1/17 : FROM nvidia/cuda:11.1-cudnn8-devel-ubuntu16.04
manifest for nvidia/cuda:11.1-cudnn8-devel-ubuntu16.04 not found: manifest unknown: manifest unknown
docker: Error response from daemon: Unknown runtime specified nvidia.
See 'docker run --help'.
```

Changing the base image from "nvidia/cuda:11.1-cudnn8-devel-ubuntu16.04" to "nvidia/cuda:11.1.1-cudnn8-devel-ubuntu16.04" causes a later error:

```
.
.
.
Successfully tagged mlperf-infer-imgclassify-gpu:latest
docker: Error response from daemon: Unknown runtime specified nvidia.
See 'docker run --help'.
```

Changing the "--runtime=nvidia" Docker run flag to "--gpus all" causes a yet later error:

```
.
.
.
2022-11-02 17:39:15.158369: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
ImportError: numpy.core.multiarray failed to import

The above exception was the direct cause of the following exception:

SystemError: <built-in method __contains__ of dict object at 0x7f862f7b13f0> returned a result with an error set

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "python/main.py", line 596, in <module>
    main()
  File "python/main.py", line 454, in main
    backend = get_backend(args.backend)
  File "python/main.py", line 274, in get_backend
    from backend_tf import BackendTensorflow
  File "/mlperf/python/backend_tf.py", line 7, in <module>
    import tensorflow as tf
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/__init__.py", line 37, in <module>
    from tensorflow.python.eager import context
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/context.py", line 35, in <module>
    from tensorflow.python.client import pywrap_tf_session
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/pywrap_tf_session.py", line 19, in <module>
    from tensorflow.python.client._pywrap_tf_session import *
ImportError: initialization failed
ENDING RUN AT 2022-11-02 05:39:15 PM
```

mobilenet is not a supported benchmark in r2.1.

I get the same error with resnet50.

@nvetcheng Could you help?

> @nvetcheng Could you help?

@nv-ananjappa, I couldn't find a contributor with the username "nvetcheng." Was that a typo?
Repository: mlcommons/inference
Issue: Bert Pytorch model fails on run
Root cause: Validation Gap

I'm trying to run the reference implementation of [Bert pytorch model](https://zenodo.org/record/3733896). But it is failing with the below error. I have tried Torch versions from 1.4.0 to 1.12 and python 3.6 on Ubuntu 18.04 and python3.10 on Ubuntu 22.04. 
Is [this config file](https://github.com/mlcommons/inference/blob/master/language/bert/bert_config.json) valid for pytorch model? The onnx and tensorflow models run fine.

By using `strict=False` in [this line](https://github.com/mlcommons/inference/blob/master/language/bert/pytorch_SUT.py#L59), pytorch model is able to run.

```
cd '/home/cmuser/CM/repos/local/cache/1a0e3c34a993400b/inference/language/bert' && /usr/bin/python3 run.py --backend=pytorch --scenario=Offline --max_examples 5 --user_conf '/home/cmuser/CM/repos/mlcommons@ck/cm-mlops/script/app-mlperf-inference/tmp/0d55e974ae634ac6a7dcb7b5810aea94.conf'
Loading BERT configs...
Loading PyTorch model...
Traceback (most recent call last):
  File "run.py", line 120, in <module>
    main()
  File "run.py", line 63, in main
    sut = get_pytorch_sut(args)
  File "/home/cmuser/CM/repos/local/cache/1a0e3c34a993400b/inference/language/bert/pytorch_SUT.py", line 93, in get_pytorch_sut
    return BERT_PyTorch_SUT(args)
  File "/home/cmuser/CM/repos/local/cache/1a0e3c34a993400b/inference/language/bert/pytorch_SUT.py", line 59, in __init__
    self.model.load_state_dict(torch.load(model_file), strict=True)
  File "/home/cmuser/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 830, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for BertForQuestionAnswering:
	Missing key(s) in state_dict: "bert.embeddings.position_ids". 
	Unexpected key(s) in state_dict: "bert.pooler.dense.weight", "bert.pooler.dense.bias". 
Finished destroying SUT.
```

Adding the flag doesn't change the metric:
```
No warnings encountered during test.

No errors encountered during test.
Reading examples...
Loading cached features from 'eval_features.pickle'...
Loading LoadGen logs...
Post-processing predictions...
Writing predictions to: build/result/predictions.json
Evaluating predictions...
{"exact_match": 83.68968779564806, "f1": 90.87487229720105}
Done!
Destroying SUT...
Destroying QSL...
Finished destroying SUT.
Finished destroying QSL.
```
Same as the one in the [readme](https://github.com/mlcommons/inference/tree/master/language/bert)

> | model | framework | accuracy | dataset | model link | model source | precision | notes |
> | ----- | --------- | -------- | ------- | ---------- | ------------ | --------- | ----- |
> | BERT-Large | PyTorch | f1_score=90.874% | SQuAD v1.1 validation set | [from zenodo](https://zenodo.org/record/3733896) | [BERT-Large](https://github.com/google-research/bert), trained with [NVIDIA DeepLearningExamples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT), converted with [bert_tf_to_pytorch.py](bert_tf_to_pytorch.py) | fp32 | |
Repository: mlcommons/inference
Issue: Permission error in  inference/speech_recognition/rnnt/
Root cause: Validation Gap

I am trying to run "MlPref /speech_recognition " in my CPU. But when I run the ./run.sh file from this link "https://github.com/mlcommons/inference/tree/r1.1/speech_recognition/rnnt" , it creates few .flack files but the permission for this file is read only. I change the permission and rerun that but it overwrite the existing permission. I am not sure how to solve this issue. Is there any advice or suggestion for me? Thank you.



022-11-28 15:09:00 (13.0 MB/s) - ‘/MlPref/inference/speech_recognition/rnnt/rnnt.pt’ saved [543976536/543976536]

0%| | 0/2943 [00:00<?, ?it/s]
Downloading dev-clean.tar.gz:
File exists, skipping download.
Verifing dev-clean.tar.gz: Checksum correct for dev-clean.tar.gz
Decompressing /MlPref/inference/speech_recognition/rnnt/local_data/LibriSpeech/dev-clean.tar.gz:
Traceback (most recent call last):
File "pytorch/utils/download_librispeech.py", line 74, in <module>
extract(fpath=fpath, dest_folder=args.e)
File "/MlPref/inference/speech_recognition/rnnt/pytorch/utils/download_utils.py", line 69, in extract
tar.extract(path=dest_folder, member=member)
File "/home/anaconda3/envs/mlperf-rnnt/lib/python3.6/tarfile.py", line 2053, in extract
numeric_owner=numeric_owner)
File "/home/anaconda3/envs/mlperf-rnnt/lib/python3.6/tarfile.py", line 2123, in _extract_member
self.makefile(tarinfo, targetpath)
File "/home/anaconda3/envs/mlperf-rnnt/lib/python3.6/tarfile.py", line 2164, in makefile
with bltn_open(targetpath, "wb") as target:
PermissionError: [Errno 13] Permission denied: '/MlPref/inference/speech_recognition/rnnt/local_data/LibriSpeech/dev-clean/2277/149896/2277-149896-0026.flac'

@pgmpablo157321 Could you check?

Actually we are using the reference script in [CM workflow](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-preprocessed-dataset-librispeech) itself and they work fine. 

MlPref -> MLPerf?

On Sat, 28 Jan 2023, 19:40 Arjun Suresh, ***@***.***> wrote:

> Actually we are using the reference script in CM workflow
> <https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-preprocessed-dataset-librispeech>
> itself and they work fine.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/mlcommons/inference/issues/1293#issuecomment-1407471468>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABSKZOSZJPPNRQPQPEAORQ3WUVY3DANCNFSM6AAAAAASQ3TRGI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Not relevant anymore, closing issue
Repository: mlcommons/inference
Issue: How to get imagenet 2012 validation dataset
Root cause: Documentation Deficiency

Hi, I am trying to run the inference benchmark in [vision/classification_and_detection](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection) for `resnet50-v1.5` with onnx backend. I don't know how to get the imagenet 2012 validation dataset. The link in [GettingStarted.ipynb](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/GettingStarted.ipynb) under step 2 is broken (404).

@pgmpablo157321 Could you help?

I think this is the only way: https://image-net.org/download.php

Okay thanks, it seems the dataset is not publicly available anymore.

Hi @god-is-a-crab .

Actually, you can still download ImageNet validation dataset via Academic torrents. 

Please check this [README](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/get-dataset-imagenet-val/README-extra.md) .

Hope it's of any help. If you need any further help with the MLPerf submission, feel free to join our [public MLCommons taskforce on education and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md) and the ["Reproducibility" Discord server](https://discord.gg/7SmbHwuC) - we are developing these [portable and reusable scripts](https://github.com/mlcommons/ck/tree/master/cm-mlops/script) to automate all the steps required to run MLPerf inference benchmarks and automate design space exploration experiments ...

CC @arjunsuresh.
Repository: mlcommons/inference
Issue: Publish official prebuilt whl files for python bindinds
Root cause: Unimplemented Feature Gap

It is a bit painful to sync code from github and manually building it for every project where it has to be used.
Would it be possible to build and release official whl files on pypi for each stable version?

Someone already added a version here: https://pypi.org/project/mlperf-loadgen-cb/#description

But it doesn't look like it is an official one.

@rnaidu02 Can we discuss the motivation and feasibility of this proposal in the first WGM in 2023 and see if @pgmpablo157321 could do it?
Repository: mlcommons/inference
Issue: Using alternative sequence lengths for SQuAD-based models in the Open division
Root cause: Unimplemented Feature Gap

For BERT Large, we tokenize the SQuAD v1.1 dataset into sequences of up to 384 symbols. For smaller models such as BERT Base, the sequence length of 128 is often used. Would it be allowed to tokenize the dataset into shorter sequences for submissions to the Open division?

@rnaidu02 Let us discuss this in the first WGM in 2023.

No disagreement with the proposal from the 1/3/2023 IWG meeting

I think that this idea breaks uniformity and should not be in the open division.

By reducing sequence length, you get a "free" speedup and bypass one of the main problems of transformers, which is the attention mechanism (time complexity of 384^2 >> 128^2 for example).

In addition to that, this makes "advancements" and comparisons with previous mlperf submissions irrelevant as it is not an apples-to-apples comparison if we compare performance of seq_len 384 with seq_len 128 for example.

From the 1/10/2023 IWG meeting, it is determined that seq is part of the benchmark definition (seq length of 384).