zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: [automation and reproducibility taskforce] progress report for 20240116

**Current progress:**
* CM coverage to automate and reproduce MLPerf inference: [GitHub](https://github.com/mlcommons/ck/issues/1052)
  * All reference implementations are supported by CM including GPT-J and Stable Diffusion (though didn’t run LLAMA). 
    * Added support to run RNNT via GPU
    * Added AMD GPU support for RNNT and BERT
  * Nvidia implementation v3.1 is supported by CM 
  * Qualcomm implementation v3.1 is supported by CM
  * Intel: only BERT is done (it took us much longer to decompose their container, expose all dependencies and provide CM automation recipes for Conda, OneDNN, etc). We continue working on adding CM for other models including GPT-J. Mostly done but we encountered and reported an issue with some Intel extensions.
* Network reference implementation for BERT is added to inference repo and automated via CM
* The latest MLPerf power measurements are automated by CM and supports Nvidia, Qualcomm, Intel and NeuralMagic
* GitHub actions to test end-to-end submission for all applicable MLPerf inference reference implementations via CM are added to the MLPerf inference repo: https://github.com/mlcommons/ck/tree/master/.github/workflows

**MLCommons offers the following free help to the v4.0 submitters via our taskforce:**
* We can help any submitter automatically benchmark Nvidia, Qualcomm or Intel systems on any cloud via CM using v3.1 implementation
* We can also fully automate submissions for v4.0 implementations via CM if we get public or private access to the code (particularly for Stable Diffusion and LLAMA-2)
* We can provide access to the Thundercom RB6 (Qualcomm AI-100) and Nvidia Jetson AJX Orin with power meter and CM automation
* Automatically generate custom Docker images via CM to run MLPerf out-of-the-box with different OS and software stack (we just need feedback from submitters which implementations and models they need)
* Extend GUI to automate MLPerf inference experimentation and visualization

We also have a proposal for reproducibility badges for MLPerf inference v4.0 based on our experience with artifact evaluation at the recent ACM/IEEE hardware conferences including [MICRO’23](https://ctuning.org/ae/micro2023.html) and ASPLOS - we will send more details later.

Public Discord channel for the automation and reproducibility TF: https://discord.gg/JjWNWXKxwT
Repository: mlcommons/inference
Issue: [Stable Diffusion Reference]: `qps` can't set to float
Root cause: Validation Gap

I run and have an error:
```
 time python3 main.py --dataset "coco-1024" --dataset-path coco2014_full --profile stable-diffusion-xl-pytorch --model-path model/stable_diffusion
_fp16/  --dtype fp16 --device cuda  --scenario SingleStream --model-name stable-diffusion-xl --qps 0.022  --output results --ids-path tools/sample_ids.txt

usage: main.py [-h] [--dataset {coco-1024}] --dataset-path DATASET_PATH [--profile {defaults,debug,stable-diffusion-xl-pytorch,stable-diffusion-xl-pytorch-dist}] [--scenario SCENARIO] [--max-batchsize MAX_BATCHSIZE] [--threads THREADS]
               [--accuracy] [--find-peak-performance] [--backend BACKEND] [--model-name MODEL_NAME] [--output OUTPUT] [--qps QPS] [--model-path MODEL_PATH] [--dtype {fp32,fp16,bf16}] [--device {cuda,cpu}]                                                [--latent-framework {torch,numpy}] [--mlperf_conf MLPERF_CONF] [--user_conf USER_CONF] [--audit_conf AUDIT_CONF] [--ids-path IDS_PATH] [--time TIME] [--count COUNT] [--debug]
               [--performance-sample-count PERFORMANCE_SAMPLE_COUNT] [--max-latency MAX_LATENCY] [--samples-per-query SAMPLES_PER_QUERY]
main.py: error: argument --qps: invalid int value: '0.022'
```
Here https://github.com/mlcommons/inference/blob/master/text_to_image/main.py#L102
qps should be `int`.

We need to set `qps` to `float` in some cases.


Ideally we should just delete that option and just use `user.conf` right? Handling qps/latency etc inside the implementation is unnecessary extra work. 

@arjunsuresh , thank you. It is better to use user.conf only.
Repository: mlcommons/inference
Issue: How can I  decrease the expected latency?
Root cause: Configuration Error

In the file 'mlperf_log_summary.txt', it shows '1 ERROR encountered. See detailed log.'. And the detailed log shows :
:::MLLOG {"key": "error_runtime", "value": "Ending early: Ran out of generated queries to issue before the minimum query count and test duration were reached. Please update the relevant expected latency or target qps in the TestSettings so they are more accurate.", "time_ms": 348305.965731, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "loadgen.cc", "line_no": 505, "pid": 13554, "tid": 13554}}

Which MLPerf inference implementation are you trying?

> Which MLPerf inference implementation are you trying?
vision/classification_and_detection，the Model is mobilenet, and the Datasets is imagenet2012. 
Looking forward to you helping me out with this, thanks

Actually mobilenet is not officially part of MLPerf inference. But it can still be submitted under open division. You can follow the following README which makes use of CM automation to run mobilenet models. 
https://github.com/mlcommons/ck/blob/master/docs/mlperf/inference/resnet50/README_tflite.md
Repository: mlcommons/inference
Issue: Questions about Llama benchmarks
Root cause: Documentation Deficiency

Really exciting to see progress on LLM benchmarking in the `loadgen` codebase. I do wonder that:

a) Is `First Token latency` going to be the only metric? Sometimes we might also need per-token latency, or other metrics.
b) The "significant overhead" mentioned in https://github.com/mlcommons/inference/blob/master/language/llama2-70b/SUT.py#L333 , is it because adding `FirstTokenStreamer` to `generate` slows it down significantly?
c) Is there an expected release date on mlperf-loadgen 4.0?

Thanks for the great work :)

@pgmpablo157321 to answer

@ljk3210 For the Server scenario the **first-token-latency** is one of the constraints that have to be met. The other constraint is the **time-per-output-token**. Their statistics during the run will also be reported, but the actual metric of interest here is the **target qps** which determines the frequency at which queries are dispatched to the client system.

Regarding b), yes, the overhead is a perceived one, since the implementation is not optimized. The tokens streamer runs in a live thread.

@pgmpablo157321 Can comment on the release date for the loadgen 4.0
Repository: mlcommons/inference
Issue: Error in submission checker
Root cause: Validation Gap

```
ERROR:main:no submission in open/CTuning/compliance
INFO:main:Make sure you keep a backup of /home/arjun/inference_submission_tree_logs in case mlperf wants to see the original accuracy logs
* cm run script "preprocess mlperf submission"
  * cm run script "get python3"
  * cm run script "get mlcommons inference src"
python3 '/home/arjun/CM/repos/local/cache/f911c234b3e94513/inference/tools/submission/preprocess_submission.py' --input '/home/arjun/inference_submission_tree' --submitter 'CTuning' --output '/home/arjun/inference_submission_tree_processed'
Traceback (most recent call last):
  File "/home/arjun/CM/repos/local/cache/f911c234b3e94513/inference/tools/submission/preprocess_submission.py", line 283, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/arjun/CM/repos/local/cache/f911c234b3e94513/inference/tools/submission/preprocess_submission.py", line 268, in main
    config = checker.Config(
             ^^^^^^^^^^^^^^^
  File "/home/arjun/CM/repos/local/cache/f911c234b3e94513/inference/tools/submission/submission_checker.py", line 1530, in __init__
    self.accuracy_upper_limit = self.base["accuracy-upper-limit"]
                                ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'accuracy-upper-limit'
```

@pgmpablo157321 Can you please check this?

Fixed now
Repository: mlcommons/inference
Issue: [Stable Diffusion] Mismatch in the 10 image files checked by the submission checker and produced by the reference script
Root cause: Algorithmic Error

Currently submission checker is throwing the below error for stable diffusion - the image names being checked are different from the ones generated.
```
[2024-01-24 10:40:24,289 submission_checker.py:2915 ERROR] closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy expected to have the following extra files (['closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/4459.png', 'closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/4015.png', 'closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/2705.png', 'closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/1682.png', 'closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/4048.png', 'closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/4683.png', 'closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/3757.png', 'closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/1578.png', 'closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/3319.png', 'closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/95.png'])
[2024-01-24 10:40:24,289 submission_checker.py:2929 ERROR] closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy, accuracy not valid
```

```
$ ls inference_submission_tree/closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/offline/accuracy/images/
1633.png  2301.png  2392.png  2521.png  4244.png  4297.png  4379.png  4499.png  4871.png  881.png
```

@pgmpablo157321 can you please check?
Repository: mlcommons/inference
Issue: [Stable Diffusion] No compliance directory error 
Root cause: Validation Gap

The below error is currently coming while running submission checker for closed division
```
[2024-01-24 16:32:08,677 submission_checker.py:3075 ERROR] no compliance dir for closed/CTuning/results/PCSPECIALIST_AMD_AM5_with_Nvidia_RTX_4090-reference-gpu-pytorch-v2.1.0-default_config/stable-diffusion-xl/singlestream

```

[This PR](https://github.com/mlcommons/inference/pull/1576) should fix this.
@pgmpablo157321 please review.

Merged LGTM
Repository: mlcommons/inference
Issue: Issue with llama2-70b reference implementation in float16
Root cause: Environment Incompatibility

Running the llama2-70b reference implementation on CPU is giving this error

```
CMD: /home/arjun/cm/bin/python3 main.py  --scenario Offline --dataset-path /home/arjun/CM/repos/local/cache/bbeede6826cb4cf1/processed-openorca/open_orca_gpt4_tokenized_llama.sampled_24576.pkl --device cpu   --mlperf-conf '/home/arjun/CM/repos/local/cache/f0e43d80c94d491d/inference/mlperf.conf' --user-conf '/home/arjun/CM/repos/ctuning@mlcommons-ck/cm-mlops/script/generate-mlperf-inference-user-conf/tmp/ca22220211a147f09ecd1e27779c819b.conf' --output-log-dir /home/arjun/test_results/arjun_spr-reference-cpu-pytorch-v2.1.1-default_config/llama2-70b-99.9/offline/performance/run_1 --model-path /home/arjun/CM/repos/local/cache/7e45fdca432d4fed/repo

Loading dataset...
Finished loading dataset.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:29<00:00,  1.97s/it]
Loaded model
WARNING:accelerate.big_modeling:You shouldn't move a model when it is dispatched on multiple devices.
Traceback (most recent call last):
  File "/home/arjun/CM/repos/local/cache/f0e43d80c94d491d/inference/language/llama2-70b/main.py", line 96, in <module>
    main()
  File "/home/arjun/CM/repos/local/cache/f0e43d80c94d491d/inference/language/llama2-70b/main.py", line 69, in main
    sut = sut_cls(
          ^^^^^^^^
  File "/home/arjun/CM/repos/local/cache/f0e43d80c94d491d/inference/language/llama2-70b/SUT.py", line 123, in __init__
    self.load_model()
  File "/home/arjun/CM/repos/local/cache/f0e43d80c94d491d/inference/language/llama2-70b/SUT.py", line 243, in load_model
    self.model = self.model.to(memory_format=torch.channels_last)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arjun/cm/lib/python3.11/site-packages/accelerate/big_modeling.py", line 415, in wrapper
    raise RuntimeError("You can't move a model that has some modules offloaded to cpu or disk.")
RuntimeError: You can't move a model that has some modules offloaded to cpu or disk.

```

To reproduce
```
pip install cmind
cm pull repo mlcommons@ck
cm run script --tags=generate-run-cmds,inference,_full --model=llama2-70b-99.9 --backend=pytorch --precision=float16 --quiet --device=cpu
```

@attafosu have you hit the same issues on Intel CPU? We haven't seen this issue on our end before.

@nvzhihanj Thank you for checking. Can you please share the python version that worked for you? We're seeing the same issue with python 3.9 and python 3.11 (other dependencies as installed by pip on Ubuntu 23.04)

We are on python3.8 and pytorch 2.0.x

Thank you @nvzhihanj. The issue was with `rouge-score` dependency missing. `float16` didn't run on CPU but `bfloat16` is working fine. 

> @attafosu have you hit the same issues on Intel CPU? We haven't seen this issue on our end before.

I didn't see this issue.
Repository: mlcommons/inference
Issue: use MLPerf inference benchmark testing on NPU
Root cause: Documentation Deficiency

May I ask if I can use MLPerf inference benchmark testing on NPU? What do I need to do? Is there any relevant documentation available？

I am also wondering this. It looks like we would have to modify the `backend` files, e.g. for classification and detection, modifying `inference/vision/classification_and_detection/python/backend_[backend].py`. For example if you're using onnxruntime, check if your hardware is supported here: https://onnxruntime.ai/docs/execution-providers/. If not, we have to modify the `backend_[backend].py` file and implement those functions, importantly `load` and `predict`. But what else: what output formats, etc.; guidance would be helpful.
Repository: mlcommons/inference
Issue: tokens_per_sample should be +/- 10% of the reference
Root cause: Validation Gap

It looks like submission checker checks only if `tokens_per_sample` is bigger than 294.45 * 0.9.
IMHO we should also check if `tokens_per_sample` is lower than 294.45 * 1.1.

Why? In LLaMA final performance numbers take into account amount of generated tokens so we need to make sure that matches the reference.

Today on the Inference WG meeting we agreed that @pgmpablo157321 will prepare PR for this.
Repository: mlcommons/inference
Issue: [automation and reproducibility taskforce] progress report for 20240130

Many thanks to MLPerf submitters and MLCommons members for their feedback during the past 2 weeks 
to help us improve the [MLCommons CM automation for MLPerf inference](https://github.com/mlcommons/ck):

## Improvements and extensions to general [CM automation recipes](https://github.com/mlcommons/ck/blob/master/docs/list_of_scripts.md) 
that all submitters can reuse to automate their benchmarking:

#### Download models

- [x] `cmr "get ml-model gptj"`
- [x] `cmr "get ml-model stable-diffusion"`
- [x] `cmr --tags=get,ml-model,llama2-70b`

#### Download datasets

- [x] `cmr "get dataset cnndm _validation"`
- [x] `cmr "get dataset coco2014 _validation"`
- [x] `cmr "get preprocessed dataset openorca _validation"`


#### Detect/install frameworks

- [x] `cmr "get generic-python-lib _package.onnxruntime" --version_min=1.16.0`
- [x] `cmr "get generic-python-lib _package.torch" --version=2.1.1`<br>
- [x] `cmr "get generic-python-lib _package.torchvision" --version=0.16.2`

## Continue [repeatibility study for v3.1](https://github.com/mlcommons/ck/issues/1052)

- [x] GPT-J from Intel - thanks to feedback we are nearly done and have CM automation
- [x] GPT-J from Nvidia - re-testing it because of some reports about potential issues
- [ ] Google submission is under testing

Feel free to report issues or suggest submissions [here](https://github.com/mlcommons/ck/issues/1052) or via our [Discord server](https://discord.gg/JjWNWXKxwT).

## Suggestions that we may include to the new CM v1.6.1 release before the next meeting:

- [x] Substitute current complex docs with a CM GUI to select implementation, target, model and then generate CM commands or show the current state with automation and reproducibility 
  - we started prototyping this GUI and will hopefully show it next time.
- [ ] Develop high-level CM script to run different MLPerf implementations with all models for a given target in sequence similar to SPEC benchmarks ([in progress](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-any-mlperf-inference-implementation)).
- [ ] Update inference v4.0 READMEs with CM commands to download/detect/install models, datasets and frameworks

## Longer term:

- [x] [Suggestion for MLPerf repetability/reproducibility badges from ACM/IEEE/NeurIPS](https://github.com/mlcommons/ck/issues/1080)
- [ ] Sugestion to restart Automation and Reproducibility TF - contact us if you would like to participate, chair, develop ...

CM is a collaborative engineering effort based on your feedback - please don't hesitate to get in touch via our [Discord server](https://discord.gg/JjWNWXKxwT) or open a ticket [here](https://github.com/mlcommons/ck/issues). Thank you!
Repository: mlcommons/inference
Issue: CNNDM download failing for Nvidia v3.1 scripts 
Root cause: External Dependency Breakage

Inside the Nvidia Docker container `BENCHMARKS=gptj make download_data` is failing as shown below. 

```
(mlperf) arjun@mlperf-inference-arjun-x86-64-411:/work$ BENCHMARKS=gptj make download_data
Inside container, start downloading...
Downloading and preparing dataset None/1.0.0 to /home/arjun/.cache/huggingface/datasets/parquet/1.0.0-b3218b6f9a2cf018/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7621.39it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1385.33it/s]
Traceback (most recent call last):
  File "build/inference/language/gpt-j/download_cnndm.py", line 24, in <module>
    dataset = load_dataset(dataset_id, name=dataset_config)
  File "/usr/local/lib/python3.8/dist-packages/datasets/load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 1003, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/info_utils.py", line 100, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=1261703785, num_examples=287113, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=3785111355, num_examples=861339, shard_lengths=[115705, 115704, 115704, 115705, 121408, 115705, 115704, 45704], dataset_name='parquet')}, {'expected': SplitInfo(name='validation', num_bytes=57732412, num_examples=13368, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=173197236, num_examples=40104, shard_lengths=None, dataset_name='parquet')}, {'expected': SplitInfo(name='test', num_bytes=49925732, num_examples=11490, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='test', num_bytes=149777196, num_examples=34470, shard_lengths=None, dataset_name='parquet')}]
```

```
python3 -m pip install --upgrade datasets
```
fixed this.
Repository: mlcommons/inference
Issue: Llama2 token logging bug with TEST06 (flag accuracy_log_sampling_target)
Root cause: Validation Gap

When running TEST06 for Llama2, we found that the `accuracy_log_sampling_target` will segfault for Server scenario. After investigation, the reason is because the first-token data is not properly logged under such mode (see: https://github.com/mlcommons/inference/blob/4bdf56f883bafa08cd571cc0f515a7362afa0b79/loadgen/loadgen.cc#L87 and https://github.com/mlcommons/inference/blob/4bdf56f883bafa08cd571cc0f515a7362afa0b79/loadgen/loadgen.cc#L143)

Proposed fix:
~When first-token data is required, the loadgen should sample accuracy logging in the tokenComplete function, and record the sample_id or unique_id in a hashset. In SampleComplete() function, if the id is in the hashset, the sample's accuracy should be logged.~
Use accuracy_log_val to determine whether the first_token needs to be logged or not.

@pgmpablo157321 @szutenberg FYI

Thanks @nvzhihanj for catching this. It might be as simple as copying & pasting the code from here? https://github.com/mlcommons/inference/blob/master/loadgen/loadgen.cc#L82-L93 

> Thanks @nvzhihanj for catching this. It might be as simple as copying & pasting the code from here? https://github.com/mlcommons/inference/blob/master/loadgen/loadgen.cc#L82-L93

You are totally right. Thanks for the suggestion.
Repository: mlcommons/inference
Issue: Broken Download Links: fp32 and fp16 Models
Root cause: Documentation Deficiency

The fp32 and fp16 model download links on this page aren't working. Would it be possible to get updated links or another way to access the models? https://github.com/mlcommons/inference/tree/master/text_to_image

Can you please try the following?

```
pip install cmind
cm pull repo mlcommons@ck
```

```
cm run script --tags=get,ml-model,sdxl,_fp16,_rclone -j
```

```
cm run script --tags=get,ml-model,sdxl,_fp32,_rclone -j
```

Thanks, it works. But I don't know whether the mlcommon hosted checkpoint are same as cm hosted checkpoint.

They are the same. Basically cm is just wrapping the rclone command which accesses the same S3 bucket.
Repository: mlcommons/inference
Issue: `lg.QuerySamplesComplete` hangs when using multiprocessing
Root cause: Resource Mishandling

I have a Python script that uses the multiprocessing module to parallelise some work. Inside this multiprocessing code, I call `lg.QuerySamplesComplete(responses)` to check if a query has completed and get the results.

When running with multiple processes, it seems that this call to `lg.QuerySamplesComplete` causes the code to hang indefinitely. The processes never exit and the script does not complete. Any idea how to fix that, or should we expect it working on multiprocessing module?

I will close it for now, it is doable with shared memory but not recommanded