zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: llama3-8b download failing
Root cause: External Dependency Breakage

```
Downloading: bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh)  -d '/home/mlcuser/MLC/repos/local/cache/download-file_llama3-dataset-_94f9f6e2/llama3-1-8b-instruct.uri'  https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri
Checking if authentication is required...
Authentication required
Checking dependencies...
cloudflared not found. Attempting to install...
Installing cloudflared...
Downloading cloudflared...
cloudflared installed successfully to /usr/local/bin/cloudflared
Logging in to Cloudflare Access at https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri...
Please open the following URL and log in with your Cloudflare account:

https://llama3-1.mlcommons-storage.org/cdn-cgi/access/cli?aud=892c2283ed5ea77ad3dc38590b0684f177421d8347c8a13835848c1bfd08cb23&edge_token_transfer=true&redirect_url=https%3A%2F%2Fllama3-1.mlcommons-storage.org%2Fmetadata%2Fllama3-1-8b-instruct.uri%3Faud%3D892c2283ed5ea77ad3dc38590b0684f177421d8347c8a13835848c1bfd08cb23%26token%3DvefiNItyXkPkoWu0rnx8Vp0LjC-uC_MH1tJZdB5-d3s%253D&send_org_token=true&token=vefiNItyXkPkoWu0rnx8Vp0LjC-uC_MH1tJZdB5-d3s%3D

Leave cloudflared running to download the token automatically.
2025-07-28T20:17:50Z INF Waiting for login...
2025-07-28T20:18:44Z INF Waiting for login...
2025-07-28T20:19:38Z INF Waiting for login...
2025-07-28T20:20:32Z INF Waiting for login...
2025-07-28T20:21:26Z INF Waiting for login...
2025-07-28T20:22:20Z INF Waiting for login...
2025-07-28T20:23:15Z INF Waiting for login...
2025-07-28T20:24:09Z INF Waiting for login...


2025-07-28T20:25:03Z INF Waiting for login...
2025-07-28T20:25:57Z INF Waiting for login...
2025-07-28T20:25:57Z ERR Could not verify token error="Could not create access request: failed to run transfer service: Failed to fetch resource"
Could not create access request: failed to run transfer service: Failed to fetch resource
Unable to find token for provided application. Please run login command to generate token.
open /home/mlcuser/.cloudflared/llama3-1.mlcommons-storage.org-892c2283ed5ea77ad3dc38590b0684f177421d8347c8a13835848c1bfd08cb23-token: no such file or directory
```
Repository: mlcommons/inference
Issue: SDXL multithreading support?
Root cause: Documentation Deficiency

Hi,

I'm a bit unclear on whether the SDXL inference benchmark is supposed to support threading or not. On the one hand, `meta.yaml` forces the [`MLC_NUM_THREADS` env var to always be `1`](https://github.com/mlcommons/mlperf-automations/blob/61c933e9323cca04a4823572252d319b0e865f3e/script/app-mlperf-inference-mlcommons-python/meta.yaml#L1057), which prevents specifying the number of threads to the benchmark. However, the `customize.py` script has [logic to forward the `MLC_NUM_THREADS` env var to the underlying implementation](https://github.com/mlcommons/mlperf-automations/blob/61c933e9323cca04a4823572252d319b0e865f3e/script/app-mlperf-inference-mlcommons-python/customize.py#L131) specifically for SDXL, which would suggest it is supposed to support multithreading.

If the current implementation is intentional, I think it would be nice to document that SDXL does not support multithreading.

Thanks.

yes, the reference implementation doesn't support multiple threads for SDXL. @anandhu-eng 

Thanks for pointing that out, @lunbun. I’ve added a note regarding the SDXL reference implementation in [this PR](https://github.com/mlcommons/inference/pull/2292/files#diff-b10564ab7d2c520cdd0243874879fb0a782862c3c902ab535faabe57d5a505e1).

Regarding the option in MLC Scripts, I think we can keep it as it is for anyone running inference on optimized code from their forked repo, it could still be useful.

Thanks for the quick response! Sounds good. I'll go ahead and close this.
Repository: mlcommons/inference
Issue: wrong path for tf ResNet-34 backend
Root cause: Configuration Error

in run_common.sh
it seems tf backend is hardcoded with an ONNX model. Is that correct?

if [ $name == "ssd-resnet34-tf" ] ; then
    model_path="$MODEL_DIR/ssd_resnet-34_from_onnx.pb"
    profile=ssd-resnet34-tf
fi

should it be 'ssd_resnet34_mAP_20.2.pb' from the readme?

I think this is wrong but we are waiting for https://github.com/mlperf/inference/pull/183 that will fix this.

FYI, @woinck uploaded a new checkpoint and pb file to Zenodo. See https://zenodo.org/record/3262269#.XRqSIy2ZPUI. Based on that @pkanwar23 updated Guenther's table. See #232. 

just tested. The code is working now. Closing this issue.
Repository: mlcommons/inference
Issue: Huggingface Rate Limit Issue when Running Accuracy Checker for llama
Root cause: Environment Incompatibility

We were running llama2 for submission, but we kept running into this error and are still hitting this issue after the submission date. I appear to be rate limited by hugginface when I try to run the accuracy checker for any of the llama models. For example if I run the accuracy checker for llama2 with the command below,

> python3 -m code.common.mlcommons.accuracy_checker --benchmark LLAMA2 --scenario Offline --precision FP8 --log_dir /work/build/logs/default/AccuracyOnly/SR780a_V3_B200SXMx8_TRT/llama2-70b-99/Offline  

I get this as my output to the terminal: [llama2_accuracy_check.log](https://github.com/user-attachments/files/21742166/llama2_accuracy_check.log)

The main error seems to be this:

> Traceback (most recent call last):
>   File "/work/build/inference/language/llama2-70b/evaluate-accuracy.py", line 156, in <module>
>     main()
>   File "/work/build/inference/language/llama2-70b/evaluate-accuracy.py", line 127, in main
>     results_list = pool.map(compute_rouge_chunk, chunks)
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.12/multiprocessing/pool.py", line 367, in map
>     return self._map_async(func, iterable, mapstar, chunksize).get()
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.12/multiprocessing/pool.py", line 774, in get
>     raise self._value
>   File "/home/perf/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--rouge/86e7a402cd7dfe065e73052a1fb4af8a2f610ef14bad9fa08ccc5dd334237bcf/rouge.py", line 30
>     min-height: 100vh;
>                   ^
> SyntaxError: invalid decimal literal

>"/home/perf/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--rouge/86e7a402cd7dfe065e73052a1fb4af8a2f610ef14bad9fa08ccc5dd334237bcf/rouge.py" 

When I open the file names in the error message listed out above, I see the message "We had to rate limit you. To continue using our service, please wait a few minutes and log in, or create an account" in that file. Deleting "/home/perf/.cache" just starts the download again and I get rate limited again. I belive that I am logged in to my HF account.

We were told that the line that its erroring at is this. 
https://github.com/mlcommons/inference/blob/master/language/llama2-70b/evaluate-accuracy.py#L58 


My main questions:
a)Is there a way to manually download the files needed and replace the files it is looking for in order to run it offline? If so what are all the files I need?

b)Is this an issue on MLCommons side or Huggingface? 

c)Why am I now having this issue? I didn't see this error before when I ran the acuracy checker earlier in the week before submission. But even after a week after submission, I still see this error.

I found what my issue was. I tried running it the first time while not logged in and got rate limited. After loging in, I tried again, but the wrong files were cached. When I deleted the .cache, it also deleted the login info and I didn't notice till now.
Repository: mlcommons/inference
Issue: Revisit Early stopping
Root cause: Documentation Deficiency

Early stopping and current validity check reporting is confusing.

For example, this deepseek run has 1.9 second TTFT. The validity checker still reports it as invalid because early stopping uses some different threshold as a validity criterion. This is not communicated clearly. Please refer to the example below. 

Server Scenario:

+--------------------------------------+-------------+-----------+----------+-------------------------+----------------------------------\

--+------------------+------------------+

| System Name                          | Benchmark   | Setting   | Valid?   | Per-query time usage    | Metric Name                      \

  |   Measured Value | Avg. Power (W)   |

+======================================+=============+===========+==========+=========================+==================================\

==+==================+==================+

| vSYS-422GA-NBRT-B200-SXM-180GBx8_TRT | deepseek-r1 | cp990     | INVALID  | TTFT: 99.6%, TPOT: 75.8 | result_completed_tokens_per_secon\

d |         12331.70 | N/A              |

+--------------------------------------+-------------+-----------+----------+-------------------------+----------------------------------\

--+------------------+------------------+

  * Note: 'Per-query time usage' is the measured 99th-percentile latency divided by the requested server latency. This value should not e\

xceed 100% for a 'VALID' result.

Let's take this to the working group meeting

Add this as a part of  LoadGen Readme doc
Repository: mlcommons/inference
Issue: Request to reduce size of LLM checkpoints hosted by MLCommons
Root cause: Unimplemented Feature Gap

The current checkpoints hosted by MLCommons are large, and using them takes more storage space than necessary. As a rule-of-thumb, checkpoints require parameter-count * byte/parameter storage, so an 8B model with bf16 weights requires 16GB storage, 70B requires 140GB storage, and so on. 

Let's take this to working group meeting
Repository: mlcommons/inference
Issue: Inference Power Submission Checker Fails for Interactive
Root cause: Validation Gap

I noticed that the submission checker doesn't work when I try to submit Power results with Interactive scenario. I get a message that says Interactive is not a recognized Scenario.

[2025-08-13 16:39:22,074 submission_checker.py:2755 ERROR] closed/Lenovo/results/.../llama2-70b-99/Interactive/performance/run_1 caused exception in check_power_dir: Unknown scenario: Interactive

Is this an error in the inference repo or power-dev?

Should be fixed by [this](https://github.com/mlcommons/inference/pull/2326)

Looks good to me. It works when I tested it.

The PR was verified by Pablo and has been successfully merged.
Repository: mlcommons/inference
Issue: Keep all LoadGen files under loadgen/

Now that we have a special `loadgen/` directory, would it make sense to move there all the related files from the root directory?

```
.gn
BUILD.gn
CONTRIBUTING.md
DEPS
LICENSE.md
Makefile
README.md
loadgen_pymodule_setup_lib.py
loadgen_pymodule_setup_src.py
```

For example, it seems that while all `*.md` files belong to the root directory, `loadgen_*` and `Makefile` clearly don't. (Not sure about `.gn`, `BUILD.gn` and `DEPS` that may need to stay in the root directory.)

@briandersn 

.gn, BUILD.gn, and DEPS have some restrictions on where they can go. I might be able to move them to the loadgen dir, but it would require some investigation and potentially wasted effort.

I should be able move the loadgen_* and Makefiles though.


> .gn, BUILD.gn, and DEPS have some restrictions on where they can go.

Yeah, that's why I was cautious to say that we may need to keep them where they are.

> I should be able move the loadgen_* and Makefiles though.

Great. The reason for creating this issue was that I spent a few moments panicking while trying to locate `Makefile` :)

very low priority for now

Can be sneaked with many LoadGen updates.
Repository: mlcommons/inference
Issue: LoadGen should use Python 3
Root cause: Architectural Constraint

I think we have said it multiple times that only Python 3 is officially supported (e.g. [here](https://github.com/mlperf/inference/issues/93#issuecomment-488360664)). Yet `inference/loadgen/BUILD.gn` has this on top:
```
# These can be overriden in the .gn file or via the "gn gen" command line.
declare_args() {
  py_include_dirs = [ "/usr/include/x86_64-linux-gnu/python2.7",
                      "/usr/include/python2.7" ]
  py_ld_flags = [ "-lpython2.7" ]
}
```

Moreover, `build/config/linux/pkg-config.py` is not Python 3 compatible, as can be seen in multiple `print`s without parentheses e.g.
```
    print "You must specify an architecture via -a if using a sysroot."
```

@briandersn please clean up.

Actually I always use python3 and there is no issue with the quick start steps at https://github.com/mlperf/inference/blob/master/loadgen/README.md

The setup.py approach to building the loadgen as a python module is compatible with python3.

The gn/ninja approach for building as a C++ library uses build files that assume python2. It'll require a bit of work to remove python2 as a dependency for the gn/ninja approach since I don't own the code in build/config. I'd really like to avoid fixing or re-implementing build/config.

As a compromise, I'll see if I can make it easy to provide a python2 path to gn/ninja if python3 is the system default. @psyhtest : would that be acceptable?

I will also update the REAMDE file so folks aren't surprised.

Alternatively, if we could add CMake bulid files - but I'm not familiar with CMake and would appreciate if someone else might be able to pick that up.

I imagine some folks already have CMake files of their own set up for internal use.


@briandersn Will have a look at providing `CMakeLists.txt`.

@briandersn Please check https://github.com/mlperf/inference/pull/239. I believe this should be sufficient to build LoadGen using tools that most people already have (e.g. CMake and make), without extra dependencies.

This work is completed.
Repository: mlcommons/inference
Issue: Not able to cross 2000 Samples/second for RGAT inferencing on tiny dataset
Root cause: Unimplemented Feature Gap

Hello All!

Please help me with the following issue:

I am carrying out inferencing for RGAT model on tiny dataset with the following values of  `--threads` and `--max-batchsize`:

```
THREAD_MIN=4
THREAD_MAX=224
THREAD_STEP=4
BATCH_MIN=10
BATCH_MAX=4096
BATCH_STEP=10
```

Here is the complete command:

```bash
#!/usr/bin/env bash
set -u

# ==== CONFIG (edit if you like) ====
export OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
COUNT=40000000        # big count so each run stabilizes
LAYOUTS=(CSC CSR)     # add COO if you want
THREAD_MIN=4
THREAD_MAX=224
THREAD_STEP=4
BATCH_MIN=10
BATCH_MAX=4096
BATCH_STEP=10

# ==== OUTPUT ====
mkdir -p output
SUMMARY_CSV="output/throughput_summary.csv"
TOP_TXT="output/throughput_top.txt"
echo "layout,threads,batch,count,samples_per_sec,output_dir" > "$SUMMARY_CSV"

# ==== SWEEP ====
for layout in "${LAYOUTS[@]}"; do
  for t in $(seq "$THREAD_MIN" "$THREAD_STEP" "$THREAD_MAX"); do
    for b in $(seq "$BATCH_MIN" "$BATCH_STEP" "$BATCH_MAX"); do
      out="output/${layout}_t${t}_b${b}"
      rm -rf "$out" && mkdir -p "$out"
      echo "Running: layout=${layout} threads=${t} batch=${b} count=${COUNT}"

      # Run; if a combo fails/OOMs, keep going
      python3 main.py \
        --dataset igbh-dgl-tiny --dataset-path igbh/ \
        --model-path model/RGAT.pt \
        --device gpu --dtype fp32 \
        --scenario Offline --layout "${layout}" \
        --count "${COUNT}" \
        --max-batchsize "${b}" --threads "${t}" \
        --output "$out" --user_conf user.conf --audit_conf audit.config || true

      # Extract throughput
      if [[ -f "$out/mlperf_log_summary.txt" ]]; then
        sps=$(grep -E "Samples per second|Completed samples per second" "$out/mlperf_log_summary.txt" \
              | tail -1 | awk -F': ' '{print $2}' | tr -d '[:space:]')
      else
        sps=""
      fi
      [[ -z "$sps" ]] && sps="NA"

      printf "%s,%s,%s,%s,%s,%s\n" "$layout" "$t" "$b" "$COUNT" "$sps" "$out" >> "$SUMMARY_CSV"
      echo "=> ${layout} t=${t} b=${b} : ${sps} samples/s"
    done
  done
done

# ==== RANK RESULTS (highest first) ====
{
  head -1 "$SUMMARY_CSV"
  tail -n +2 "$SUMMARY_CSV" | awk -F, '$5 != "NA"' | sort -t, -k5,5gr
  tail -n +2 "$SUMMARY_CSV" | awk -F, '$5 == "NA"'
} | tee "$TOP_TXT"

echo
echo "Top 10:"
awk -F, 'NR==1{print;next} $5!="NA"{print}' "$SUMMARY_CSV" | sort -t, -k5,5gr | head -10 | column -t -s,
echo
echo "Saved:"
echo "  CSV: $SUMMARY_CSV"
echo "  Sorted view: $TOP_TXT"

```


Please tell me if I am missing out on something.

I aslo see my 8 H200s underutilised, as only 1 GPU is used.

Thanks

This is an expected behavior for RGAT inferencing with reference implementation for the system being used. Could you try reproducing Nvidia implementation for RGAT for the 5.1?
Repository: mlcommons/inference
Issue: Bandwidth Requirement check in submission checker
Root cause: Validation Gap

As per our discussion in postmortem v5.1, we need to add an automatic check in the submission checker which will check the network speed of a given system ([relevant field](https://github.com/mlcommons/inference_results_v5.1/blob/main/closed/NVIDIA/systems/B200-SXM-180GBx8_TRT.json#L33) ) is meeting the BW requirement of a submitted benchmark result. As part of this we also need to make sure that the relevant system json field is non empty and a numeral. 



@pgmpablo157321 is looking into it now.
Repository: mlcommons/inference
Issue: Documenting the behaviors of various config arguments in mlperf.conf and user.conf

It would be good to document the behaviors of various config arguments in **mlperf.conf** and **user.conf** like the `min_query_count`, `max_duration`, etc so that the questions like `Would SingleStream performance run exit on attaining the minimum required duration even though the run executed only the fraction of generated_query_count` raised in issue [2308](https://github.com/mlcommons/inference/issues/2308) could be easily answered.

Could go to Readme for testSettings.
Repository: mlcommons/inference
Issue: Increase expected QPS so the loadgen pre-generates a larger (coalesced) query
Root cause: Documentation Deficiency

I used the Makefile to run the BERT example. Where can a user  Increase expected QPS ?  Thanks.
```
make run_pytorch_performance

Loading BERT configs...
Loading PyTorch model...
Constructing SUT...
Finished constructing SUT.
Constructing QSL...
No cached features at 'eval_features.pickle'... converting from examples...
Creating tokenizer...
Reading examples...
Converting examples to features...
Caching features at 'eval_features.pickle'...
Finished constructing QSL.
Running LoadGen test...
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 113.38
Result is : INVALID
  Min duration satisfied : NO
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
Recommendations:
 * Increase expected QPS so the loadgen pre-generates a larger (coalesced) query.
```
Repository: mlcommons/inference
Issue: Define standard for representing and filtering heterogeneous systems on the official result table
Root cause: Unimplemented Feature Gap

I would like to continue our discussion related to how we should display heterogeneous system results here. Given that there have already been two submitters with heterogeneous GPU configurations (e.g., systems with "GPU A + GPU B"), it is a good time to find a standard for how heterogeneous systems are shown and represented in the result table.

The goal of this is to make sure readers can:
1) Still see apples-to-apples comparisons
2) Make sure that no system is left out with the filtering logic in the results viewer (ex: when a user filters for "GPU A", a system listed as "GPU A + GPU B" is hidden. This prevents users from discovering all relevant results). The current table lists the heterogeneous system as a separate "accelerator type", which I believe is less than ideal.
3) Does not clutter the default view of the results table.

The current format gets us 1) and 3), but loses the ability to do 2). Let's use this issue to discuss and define this standard. Possible approaches could be: 

- Show the heterogeneous results when one of the accelerator is picked for the filtering.

- Allow submitters to tag submitter a "primary" accelerator designation, and the result will show up if that tagged accelerator is used as the filter.
Repository: mlcommons/inference
Issue: [Llama2-70] No matching distribution found for torch==2.2.0.dev20231006+cpu
Root cause: External Dependency Breakage

The pip command to install the correct pytorch version for Llama2-70b fails with this message:
```
ERROR: Could not find a version that satisfies the requirement torch==2.2.0.dev20231006+cpu (from versions: none)
ERROR: No matching distribution found for torch==2.2.0.dev20231006+cpu
```
See the output below:

```
~]$ conda create -n llama2-70b python=3.9
~]$ conda activate llama2-70b
(llama2-70b) ~]$ conda install pybind11==2.10.4 -c conda-forge -y
(llama2-70b) ~]$ python -m pip install torch==2.2.0.dev20231006+cpu --index-url https://download.pytorch.org/whl/nightly/cpu
Looking in indexes: https://download.pytorch.org/whl/nightly/cpu
ERROR: Could not find a version that satisfies the requirement torch==2.2.0.dev20231006+cpu (from versions: none)
ERROR: No matching distribution found for torch==2.2.0.dev20231006+cpu
```
Repository: mlcommons/inference
Issue: Conflicting documentations for accuracy thresholds for llama3.1_8b
Root cause: Documentation Deficiency

Hi,

Could you clarify the accuracy thresholds for llama3.1_8b?

1) [this page](https://mlcommons.org/benchmarks/inference-datacenter/) states that we need to be within "99% of FP32 and 99.9% of FP32 (rouge1=42.9865, rouge2=20.1235, rougeL=29.9881)" . Also which one is true 99% or 99.9%?

2) while [this page](https://mlcommons.org/2025/09/small-llm-inference-5-1/) says "We set the accuracy threshold for closed division submissions to 99% of the FP16 reference."

@pgmpablo157321 

The best place to look for these values is the submission checker and you can see the values [here](https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L438). 99.9% variant is not yet there for llama3.1_8b. 

ok thanks!
Repository: mlcommons/inference
Issue: Llama-3.1-8B inference time with different batch sizes
Root cause: Documentation Deficiency

I have run Llama-3.1-8B inference on a A100 device with fixed 1 hour run. Comparing batch size 1 and 4 shows:

b1:
INFO:Llama-8B-SUT:Samples run: 2004
INFO:Llama-8B-SUT:	BatchMaker time: 1.430511474609375e-06
INFO:Llama-8B-SUT:	Inference time: 1.5663466453552246
INFO:Llama-8B-SUT:	Postprocess time: 3.147125244140625e-05
INFO:Llama-8B-SUT:	==== Total time: 1.5663795471191406

b4:
INFO:Llama-8B-SUT:Samples run: 7348
INFO:Llama-8B-SUT:	BatchMaker time: 2.384185791015625e-06
INFO:Llama-8B-SUT:	Inference time: 1.713407278060913
INFO:Llama-8B-SUT:	Postprocess time: 9.489059448242188e-05
INFO:Llama-8B-SUT:	==== Total time: 1.7135045528411865


So, given a fixed time, b4 runs more samples which makes sense. However, I wonder why inference time of b4 is larger than b1. I expected to see less inference time. Any idea about that? Or I am misunderstanding something?

So from my understanding, b4 is running with a batch size of 4. When you see an inference time of ~1.56s for b1, that means it takes about 1.56 seconds to run inference for one sample. For b4, the ~1.71s inference time is for four samples at once. Even though the batch takes slightly longer, you’re effectively processing 4 samples in that time, which is why the total number of samples completed in the fixed run is much higher. This is also why batching improves throughput but doesn’t necessarily reduce the per-batch inference time.