zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Clarification regarding how the accuracy.txt file is generated
Root cause: Documentation Deficiency

The submission generation rules for inference says that the `accuracy.txt` file should be generated from the accuracy scripts. My interpretation of this is that one should run the reference accuracy scripts stand alone using the logs from the accuracy run and obtain this accuracy.txt file and not dump the accuracy.txt file with in the implementation code. Is this the correct interpretation? 

```
accuracy.txt # stdout of reference accuracy scripts
```

@psyhtest @ashwin @attafosu Can you please confirm?

@arjunsuresh Yes, that's correct.

I can think of a situation when an implementer refactors/integrates a reference script into their own script. For example, the reference script may hardcode using `/usr/bin/python3`, while they may want to use `/usr/local/bin/python3.8`. In this case, we can probably request that _no material changes_ should be done during such refactoring/integration, but not that the reference script must always be run stand alone?

Thank you @attafosu @psyhtest

@psyhtest yes, running the reference accuracy script standalone is fine I believe. But this is not that straightforward as it often requires the original dataset and so we do have some submissions where accuracy.txt is generated from the benchmark run itself without calling the reference script. We didn't see any accuracy issue when running the standalone script for those submissions, but I believe this should not be allowed.

@arjunsuresh 

But you admit that in some cases it may not be straightforward:
> yes, running the reference accuracy script standalone is fine I believe.
> But this is not that straightforward

So why would we disallow it in such cases?

@psyhtest I'm not telling to disallow running the reference accuracy script in a custom way - say like within another python file. But I don't think it is right to allow generation of the accuracy.txt file by mimicking the actions of the reference script - because it becomes hard to verify this for other people. 

We face this issue specifically for automating  DLRMv2 submissions where to generate the accuracy.txt file we need the day23 criteo dataset which is not possible to be downloaded in an non-interactive way. But if we are allowed to generate the accuracy.txt file from within the benchmark implementation we possibly do not need this file at all. 

@arjunsuresh to work on this
Repository: mlcommons/inference
Issue: Running Llama-2 on HPC cluster
Root cause: Documentation Deficiency

Hi,
I would like to work on the Llama-2 workload on GPU devices in an HPC cluster. Problem is that I am not able to run Docker container on the cluster. So, what is the best option for the preparation [phase](https://github.com/mlcommons/inference/tree/master/language/llama2-70b#prepare-environment)?

Hi, I don't know if this will help you or not. 
https://docs.mlcommons.org/inference/benchmarks/language/llama2-70b/#__tabbed_1_1

Can you give a little more details about the cluster? What GPUs are used and how many of them?
Why Lamma-2?

Hi @mahmoodn Sorry, it has been many years since I have been on a HPC setup - do you have podman or some other container setup?

> Hi, I don't know if this will help you or not. https://docs.mlcommons.org/inference/benchmarks/language/llama2-70b/#__tabbed_1_1
> 
> Can you give a little more details about the cluster? What GPUs are used and how many of them? Why Lamma-2?

I tired the CM commands, but it seems that it need sudo password which I don't have that on the cluster.
```
(cm) [mahmood@gpu014 scratch]$ cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev --model=llama2-70b-99 --implementation=reference --framework=pytorch --category=datacenter --scenario=Offline --execution_mode=test --device=cuda --quiet --test_query_count=50
INFO:root:* cm run script "run-mlperf inference _find-performance _full _r4.1-dev"
INFO:root:  * cm run script "detect os"
...
INFO:root:               ! call /home/mahmood/CM/repos/mlcommons@cm4mlops/script/install-cuda-prebuilt/run.sh from tmp-run.sh
sudo bash /home/mahmood/CM/repos/local/cache/65f6724ba7584a74/cuda_11.8.0_520.61.05_linux.run --toolkitpath=/home/mahmood/CM/repos/local/cache/8a208105cc1f4c34/install --defaultroot=/home/mahmood/CM/repos/local/cache/8a208105cc1f4c34/install --toolkit  --silent --override

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for mahmood:
```

> Hi @mahmoodn Sorry, it has been many years since I have been on a HPC setup - do you have podman or some other container setup?

Unfortunately, they have apptainer on the cluster.

Hi @mahmoodn I checked and we can support apptainer - but it'll take a few weeks for it. 

Are you trying to run llama2-70b or some smaller variants ? Can you please share the number of GPUs and the GPU memory you are having and I can share a solution to run without docker for now?

Thanks for the update. At this moment and after struggling with various kinds of errors, I was able successfully run Llama workload on GPU. I briefly wrote the commands [here](https://github.com/mahmoodn/inference/blob/master/language/llama2-70b/README_mahmood.md) in case someone else needs that.

I appreciate for an apptainer container as working with native commands are painful for big workloads and software stacks.
The GPUs are some A100 with 80 GB of memory.
Repository: mlcommons/inference
Issue: vision/classification setup failed due to incorrect name of package in setup.py.
Root cause: Configuration Error

In the inference/vision/classification_and_detection/setup.py

there is no package with the name **mlperf_loadgen**?

Instead it is **mlcommons_loadgen.**

I believe that file `setup.py` is a stale one. @pgmpablo157321 can you please confirm?

And yes, `mlcommons_loadgen` is the correct pypi package for loadgen. But submitters are encouraged to use the build from src method until we have the automated release workflow working in the github repo for the loadgen build. 

Not sure why this was closed, I've submitted a fix now, assuming it was as simple as this :) 

Cheers,
Annie
Repository: mlcommons/inference
Issue: [4.1 postmortem] Unit test module for inference repo
Root cause: Unimplemented Feature Gap

We propose to add some basic unit test framework (likely pytest) and tests to the inference repo. Ideally, it should test:

- All configuration (mlperf.conf, user.conf) is valid and working (i.e. without typo, no items outside of an existing dictionary etc)
- All reference benchmark dataset and model download modules are working
- All reference models are running and producing reasonable output
- Submission checker can catch errors as expected.
- etc..

To get this started, we can aim at some simple tests:

- configuration walkthrough and validation
- submission checker tests for some corner cases (once #1670 has started)

@pgmpablo157321 to help implement
@nv-alicheng for more suggestions.

We also need a deadline for these to be done before every submission round. May be the same as code freeze date?

Currently we do have github tests for reference benchmark runs which includes download of models and datasets and even the submission checker - it is completed for small models and we also have GPTJ and SDXL tests running on self hosted github runners. We hope to cover LLAMA2, DLRMv2 and Mixtral this month. 

The main concern is with upcoming benchmarks - say if the benchmark needs multiple GPUs and Terabytes of memory like say the GNN - we do not have an infrastructure to do the runs. There unit tests for the conf files and log parsing may be the only option.  



@arjunsuresh Thank you for the suggestions, I think we can split into steps and cover the most important parts first (submission checker, config files, loadgen)
For the benchmarks we can implement 1 by 1 according to urgency
Regarding the deadline, I agree it should align with freeze date

@nvzhihanj 

Yes, we do need unit tests for submission checker. Currently we check it by running it on the [previous submission repository](https://github.com/mlcommons/inference/pull/1867) which can catch most of the issues but not all. It is currently failing because "images" folder was removed before publication for stable diffusion but this was not updated in the submission checker. 

Similar situation for loadgen - we currently have tests for it and we run it on the previous benchmarks. But if a new benchmark comes and new configs are added - they are not tested. 



@pgmpablo157321 to work on this
Repository: mlcommons/inference
Issue: Number of sample runs for Llama-2
Root cause: Documentation Deficiency

For Llama-2 inference, I set the total sample count to 1024 to see how much time is needed and what is the final output. I see multiple sample runs:

```
Samples run: 2
	BatchMaker time: 0.556433916091919
	Inference time: 9502.616274356842
	Postprocess time: 0.36942481994628906
	==== Total time: 9503.54213309288
Saving outputs to run_outputs/q182_77.pkl
Samples run: 4
	BatchMaker time: 0.00017786026000976562
	Inference time: 13552.655136823654
	Postprocess time: 0.18449878692626953
	==== Total time: 13552.83981347084
Saving outputs to run_outputs/q257_571.pkl
Samples run: 6
	BatchMaker time: 0.0001952648162841797
	Inference time: 9649.980134010315
	Postprocess time: 0.10514378547668457
	==== Total time: 9650.085473060608
...
```

Questions are:
1- Why sample runs are even numbers? What does that mean?
2- How can I limit the total run number to 1? I didn't see any option in the command line for that.

@pgmpablo157321 to respond

@mahmoodn 
> 1- Why sample runs are even numbers? What does that mean?

This might be due to batch size that you are using. Could you post the command that you are running and I can try to reproduce it

> 2- How can I limit the total run number to 1? I didn't see any option in the command line for that.

You can limit the number of queries with a few parameters in the [mlperf.conf](https://github.com/mlcommons/inference/blob/master/loadgen/mlperf.conf) file (but it depends on the scenario that you are running):

1. `min_duration`: this sets the minimum duration of the run
2. `min_query_count`: this sets the minimum number of queries to finish the run.
3.  For `Offline` and `Server`, `target_qps` sets the expected queries per second. Internally, this creates (along with `min_query_count`) another condition time condition 

Once the `min_duration`, `min_query_count` and the other condition are met, the run finish. So you can set `min_query_count` to 1 and the other values appropriately

Thanks. As you pointed out, I used batch size of 2 according to the following command:
```
python3 -u main.py --scenario Offline --model-path ${CHECKPOINT_PATH} \
   --mlperf-conf mlperf.conf --user-conf user.conf --total-sample-count 1024 \
   --batch-size 2 --num-workers 1 --dataset-path ${DATASET_PATH} --output-log-dir $OUTPUT_PATH \
   --dtype bfloat16 --device cuda:0
```
Repository: mlcommons/inference
Issue: Docker with a lost cache
Root cause: Environment Incompatibility

Hello everyone. I recently moved space on my system and removed files that weighed especially much. Now when I launch "3-d Unet" I get an error.

Is there any way to download all the necessary files again, instead of accessing the lost file?

docker: **Error response from daemon: open /var/lib/docker/overlay2/dec3bj6gjyikkcxl2guzxa8pd/.tmp-committed2865404872: no such file or directory**.
See 'docker run --help'.
INFO:root:       ! call "postprocess" from /home/user/CM/repos/mlcommons@cm4mlops/script/run-mlperf-inference-app/customize.py


This is a docker issue, not MLPerf
Repository: mlcommons/inference
Issue: Question about text_to_image
Root cause: Documentation Deficiency

When I run the text_to_image command
```
python3 main.py --dataset "coco-1024" --dataset-path coco2014 \
   --profile stable-diffusion-xl-pytorch --model-path ./model/stable_diffusion_fp32 \
   --dtype fp32 --device cuda --scenario Offline
```
I see a bunch of output lines like this:
```
Loading pipeline components...: 100%|██████████| 7/7 [00:29<00:00,  4.21s/it]
/scratch/mahmood/inference/text_to_image/coco.py:58: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  torch.load(f"{data_path}/latents/latents.pt")

100%|██████████| 20/20 [00:18<00:00,  1.10it/s]
100%|██████████| 20/20 [00:12<00:00,  1.54it/s]
...

```
I have counted the number of progress bars in 24 hours on an A100 GPU and it was about 5000 lines. The question is are these progress bars some kind of creating chunks before the real execution or those or the actual outputs from the inference execution?

The progress bars indicate the inference, where 20 is the number of steps you go through the UNet part of the model. You can find that it's defined as self.steps=20 in the backend_pytorch.py . 5000 is the number of images you have processed.


Thank you.
Repository: mlcommons/inference
Issue: 3dunet segmentation fault
Root cause: External Dependency Breakage

When I want to run the 3dunet workload, it ends up with a segmentation fault which is quite not clear what is the exact problem. See the output below:

```
$ python run.py --backend=pytorch --scenario=Offline --model=./build/model/3dunet_kits19_pytorch.ptc
Constructing SUT...
Finished constructing SUT.
Constructing QSL...
Found 43 preprocessed files
Using performance count = 43
Finished constructing QSL.
Loading PyTorch model...
Running Loadgen test...
Loading file case_00400
Loading file case_00125
Loading file case_00128
Loading file case_00189
Loading file case_00005
Loading file case_00111
Loading file case_00006
Loading file case_00052
Loading file case_00161
Loading file case_00012
Loading file case_00171
Loading file case_00070
Loading file case_00207
Loading file case_00034
Loading file case_00187
Loading file case_00000
Loading file case_00169
Loading file case_00185
Loading file case_00049
Loading file case_00160
Loading file case_00112
Loading file case_00003
Loading file case_00162
Loading file case_00203
Loading file case_00157
Loading file case_00138
Loading file case_00078
Loading file case_00065
Loading file case_00086
Loading file case_00041
Loading file case_00206
Loading file case_00084
Loading file case_00024
Loading file case_00066
Loading file case_00061
Loading file case_00056
Loading file case_00076
Loading file case_00087
Loading file case_00044
Loading file case_00176
Loading file case_00092
Loading file case_00198
Loading file case_00080
Traceback (most recent call last):
  File "/scratch/mahmood/inference/vision/medical_imaging/3d-unet-kits19/run.py", line 180, in <module>
    main()
  File "/scratch/mahmood/inference/vision/medical_imaging/3d-unet-kits19/run.py", line 161, in main
    lg.StartTestWithLogSettings(sut.sut, sut.qsl.qsl, settings, log_settings, args.audit_conf)
  File "/scratch/mahmood/inference/vision/medical_imaging/3d-unet-kits19/base_SUT.py", line 173, in issue_queries
Segmentation fault (core dumped)
```
Prior to that, i ran `make setup` and `make preprocess_data`.
I don't know if it is related to missing file or parameter or a runtime problem like memory limit. Any idea about that?

3D-Unet is automatically tested. Please re-test and re-open if still having the issue

Ok I did a `git pull` and seems to be fine now.

```
Loading file case_00198
Loading file case_00080
    1/   86 -- Processing sample id 30 with shape = (1, 192, 384, 384),  50 sub-volumes took    9.65431 sec
    2/   86 -- Processing sample id 37 with shape = (1, 448, 320, 320),  96 sub-volumes took    4.44507 sec
    3/   86 -- Processing sample id 27 with shape = (1, 128, 320, 320),  16 sub-volumes took    0.93298 sec
...
```
Repository: mlcommons/inference
Issue: How Built a CM Script
Root cause: Documentation Deficiency

Hi, 

I have a customized inference script to run SDXL Datacenter Category code. How can I incorporate it into my local CM directory as part of a CM Run script? 

Thank you

If you are talking about modifications to reference implementation, you can do the changes, push them to your fork in github and then add the below option to your CM run commands.

`--adr.mlperf-inference-implementation.repo=https://github/arjunsuresh/inference,_branch.master`.

If you want to do some customizations to the options, you can do it in the below file
https://github.com/mlcommons/cm4mlops/blob/main/script/app-mlperf-inference-mlcommons-python/customize.py#L287

which will be mapped to `$HOME/CM/repos/mlcommons@cm4mlops/script/app-mlperf-inference-mlcommons-python/customize.py` locally. 

Thank you Arjun, 

Which of the methods do you recommend if this is for running the SCC24 SDXL Coco dataset scripts? We have a customized and new backend_sut. We looked into the `cm4mlops/script`, it seems that each new script requires a `customize.py` with `preprocess` and `postprocess` function. Is there a clear instructions on how to incorporate `customized backend_sut.py` into `customize.py` ?

If you are adding a custom backend for the reference implementation, you can do the first option given [here](https://github.com/mlcommons/inference/issues/1880#issuecomment-2425013116). There should be no need to modify any CM script. 

Closing, per @arjunsuresh
Repository: mlcommons/inference
Issue: How to Change Offline samples_per_query
Root cause: Documentation Deficiency

Hi, 

How do I change the values of `samples_per_query` given to the sut by loadgen? 
https://github.com/mlcommons/inference/blob/master/text_to_image/main.py#L421

```
    def issue_queries(query_samples):
        runner.enqueue(query_samples)
```

Hi @zixianwang2022 it's not trivial to change that as it's scenario specific and for each scenario it's value is calculated by loadgen using multiple parameters. In offline scenario, target_qps is the main parameters which determines the number of samples in a query. 

If you are running via CM, you can provide `--test_query_count=N` to the run command to control this. 

You can do that by additing the lines in your user.conf :
*.Offline.min_query_count = \<desired number of samples>
*.Offline.min_duration = 1
Details for this can be found in loadgen.cpp which controls the number of samples in the query according to the target qps, min_duration and min_query_count in user.conf.
Repository: mlcommons/inference
Issue: Bug in the gptj accuracy script
Root cause: Validation Gap

Currently the [accuracy.txt](https://github.com/mlcommons/cm4mlperf-inference/blob/mlperf-inference-results-scc24/open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.4.1-default_config/gptj-99/offline/accuracy/accuracy.txt) file is having the below content
```
Constructing QSL
Encoding Samples

Results

{'rouge1': np.float64(32.2581), 'rouge2': np.float64(6.6667), 'rougeL': np.float64(22.5806), 'rougeLsum': np.float64(25.8065), 'gen_len': np.int64(258), 'gen_num': 1}
Finished destroying QSL.
hash=5470ad247d575ae6dd919dbaec13f5d488c15366f13a7dc52cb0e73e062cc958
```
Repository: mlcommons/inference
Issue: Create a pypi package for inference-submission-tools
Root cause: Unimplemented Feature Gap

Currently we need to clone the inference repository to get the python files to do `accuracy_truncation`, `preprocess_submission` and to run the `submission_checker`. It'll be easier if these tools are distributed as a `pypi` package and one can do `pip install mlperf-inference-submission-tools`.

Since this can be too complicated to replicate across the submission tools and individual benchmark accuracy scripts closing this issue as we can instead use CM commands to get similar effect.
Repository: mlcommons/inference
Issue: coco upscale script size
Root cause: Documentation Deficiency

@guschmue 

Why is the readme for 700x700? I thought we want to do 1200x1200. 

Please verify.

Thanks!

this is the original code. We should update classification_and_detection/README.md 

please update the readme of the resize steps and we can close this afterwards. Thank you @guschmue
Repository: mlcommons/inference
Issue: mixtral-8x7b baseline reproduction : gsm8k :  72.00
Root cause: Environment Incompatibility

Hi, I tried to reproduce your baseline on mixtral-8x7b, but the accuracy on gsm8k is 72.00 instead of 73.66. Can you reproduce it?
Also, what is the version of transformers?

@DehuaTang : What system are you running on? Also, can you re-test?
Repository: mlcommons/inference
Issue: This system is not registered with an entitlement server
Root cause: Environment Incompatibility

I get the following error  when I run
cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev \
   --model=dlrm-v2-99 \
   --implementation=reference \
   --framework=pytorch \
   --category=datacenter \
   --scenario=Offline \
   --execution_mode=test \
   --device=cpu  \
   --docker --quiet \
   --test_query_count=10

**This system is not registered with an entitlement server. You can use subscription-manager to register.**

How can I prevent the warning?

Hi @Qixiao1234 can you please share the console output here? 

But you are running dlrmv2 reference implementation on CPU which can take weeks to complete. 

Closing, not an MLPerf issue.
Repository: mlcommons/inference
Issue: SDXL: index 21 is out of bounds for dimension 0 with size 21  
Root cause: Resource Mishandling

Hi, 

Run command: 
```
python3 main.py --dataset "coco-1024" --dataset-path coco2014 --profile stable-diffusion-xl-pytorch --model-path /work1/zixian/ziw081/CM/models/SDXL/official_pytorch/fp16/stable_diffusion_fp16/ --dtype fp16 --device cuda --time 30 --scenario Offline --threads 2
```

When I am running with `--threads>1`, we are seeing 
```
ERROR:main:thread: failed on contentid=['/work1/zixian/ziw081/inference/text_to_image/coco2014/validation/data/COCO_val2014_000000386589.jpg'], index 21 is out of bounds for dimension 0 with size 21
```

I saw similar event here: https://github.com/mlcommons/inference/pull/1519#issue-1993090825 , but there's no actual solutions posted regarding this error. 

We believed that it was a direct cause of [threading.Thread](https://github.com/mlcommons/inference/blob/dev/text_to_image/main.py#L290) of the source code. 

Hi @zixianwang2022, yes improving the code for multi-threading can be a useful change but that's unlikely to improve the performance. 

> Hi @zixianwang2022, yes improving the code for multi-threading can be a useful change but that's unlikely to improve the performance.

Hello @arjunsuresh , 

When we have `--threads>1`, we are seeing a significant improvement in terms of performance. However, the run isn't valid as it is not passing the accuracy test for the above error. Are you aware of any changes that can avoid getting the error? 

@zixianwang2022 unfortunately no. Since the performance was expected using 1 thread we haven't checked the reference implementation since.