zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: offline reporting with LoadGen
Root cause: Algorithmic Error

With offline mode, the metric should be reported as samples per second rather than QPS, as there is only 1 query in the offline mode.

cat mlperf_log_summary.txt 

SUT name : PySUT
Scenario : Offline
Mode     : Performance
**QPS: xxxxxxx**
Result is : INVALID
  Min duration satisfied : NO
  Min queries satisfied : Yes

@briandersn 

This is fixed now. Closing.
Repository: mlcommons/inference
Issue: Execution of MLPerf Inference text_to_image benchmark fails
Root cause: Validation Gap

text_to_image/tools/fid/fid_score.py, line 47, in <module>
sys.path.insert("..", 0)

TypeError: 'str' object cannot be interpreted as an integer

CM error: Portable CM script failed (name = benchmark-program, return code = 256)
Repository: mlcommons/inference
Issue: Relocate v0.5/tools/upscale_coco

Why is https://github.com/mlperf/inference/tree/master/v0.5/tools/upscale_coco sitting in a separate tools dir? Should we reorg it into the classification_and_detection directory?

That was intention in case there are alternative implementations.

I think we are Ok with this. please re-open if needed.
Repository: mlcommons/inference
Issue: [v4.1 inference] Detected system did not match any known systems.
Root cause: Documentation Deficiency

Hi,
I'm facing some issues when i tried running the benchmark for 3d-unet.
When i ran 
**make run RUN_ARGS="--benchmarks=3d-unet --scenarios=offline,server""**

Got the errors, which is also the system i'm working on 
**Detected system did not match any known systems. Exiting. SystemConfiguration(host_cpu_conf=CPUConfiguration(layout={CPU(name='AMD EPYC 9554 64-Core Processor', architecture=<CPUArchitecture.x86_64: AliasedName(name='x86_64', aliases=(), patterns=())>, core_count=56, threads_per_core=2): 2}), host_mem_conf=MemoryConfiguration(host_memory_capacity=Memory(quantity=1.5849542239999999, byte_suffix=<ByteSuffix.TB: (1000, 4)>, _num_bytes=1584954224000), comparison_tolerance=0.05), accelerator_conf=AcceleratorConfiguration(layout=defaultdict(<class 'int'>, {GPU(name='NVIDIA H100 PCIe', accelerator_type=<AcceleratorType.Discrete: AliasedName(name='Discrete', aliases=(), patterns=())>, vram=Memory(quantity=79.6474609375, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=85520809984), max_power_limit=350.0, pci_id='0x233110DE', compute_sm=90): 8})), numa_conf=NUMAConfiguration(numa_nodes={}, num_numa_nodes=16), system_id=None)** 

**Driver Version: 550.90.07      CUDA Version: 12.4**  

I didn't manually add any system configuration under /work/code/common/system since i didn't saw 4.1 inference result submitter who submit on H100 Pcie 80GB customized that.

any suggestion could help me to pass the error? 
thanks a lot!


Hi @loganwuw Nvidia implementation detects the system configuration which includes the GPU and CPUs and if the system is not a known one separate scripts need to be called to initialize the system. If you want to benchmark 3d-unet, you can use the below CM wrapping - where those manual steps are automated. We actually do nightly runs for these benchmarks and store the results [here](https://github.com/gateoverflow/mlperf_inference_unofficial_submissions_v5.0/) 

https://docs.mlcommons.org/inference/benchmarks/medical_imaging/3d-unet/#__tabbed_1_2
Repository: mlcommons/inference
Issue: Source distributions are missing on PyPI
Root cause: Configuration Error

Since the release 4.0.1 source distribution aren't published on PyPI.

Since the release of 4.0.1, source distributions haven't been published on PyPI:
https://pypi.org/project/mlcommons-loadgen/#files


Hi @tprrt yes, currently we are only adding the whl files via github action. But we can add the src distribution too if there's a use. 

@tprrt The source files are now uploaded. 

thx
Repository: mlcommons/inference
Issue: Updated https://github.com/mlperf/inference README

The main https://github.com/mlperf/inference link doesn't have any README on it. We need to have some important details recorded there: e.g., dir structure, how to run one or all benchmarks. Pointer back to rules documents etc.

@christ1ne and @guschmue can one of you please review the readme. I just copied it from training, made modifications and posted it.

The reference system needs to be updated. Both of you have said you are running all the benchmarks, so is it possible to just list down your machine specs.

I also have doubts if we really do support Docker containers of the reference implementations. I didn't find docker container setup in all dirs. Well, specifically I didn't find it in the GNMT codebase.
Repository: mlcommons/inference
Issue: GNN/R-GAT Inference Submission disallows sorting of the nodes.  Need a mechanism to prevent and detect sorting.
Root cause: Unimplemented Feature Gap

GNN/R-GAT Inference task force agreed that the sorting of the nodes is disallowed. Sorting of the nodes reduces number of unique nodes and artificially increases the throughput up to 6x.  We need a mechanism to prevent and detect sorting.


@ukurkure Adding TEST04 compliance can solve this. But it is not straightforward to do this and will need time. Probably this can be targeted for 5.1. The idea of the test is as follows:

A run is done with limited number of unique inputs say 1000 and repeated 10 times.
A run is done with 10000 unique inputs
Run times are compared between the two and the difference should be within 10%. 

The section 8.2.  Model Equivalence needs to be updated https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#82-model-equivalence. 
The proposed addition to the disallowed techniques:

The following techniques are disallowed:

- Techniques that only improve performance when there are identical samples in a query. For example, sorting samples in SSD.
- The Techniques that reduce the number of unique samples in a query. For example, sorting samples in Graph Neural Networks' RGAT.

@ukurkure The below line already covers the case for R-GAT too right? 

Techniques that only improve performance when there are identical samples in a query. For example, sorting samples in SSD.

If the working group agrees that "Techniques that only improve performance
when there are identical samples in a query. For example, sorting samples
in SSD." covers RGAT, we do not have to add the RGAT case explicitly. We
could make it explicit by adding the following:
Techniques that only improve performance when there are identical samples
in a query. For example, sorting samples in SSD and RGAT.

On Tue, Nov 26, 2024 at 6:09 PM Arjun Suresh ***@***.***>
wrote:

> @ukurkure <https://github.com/ukurkure> The below line already covers the
> case for R-GAT too right?
>
> Techniques that only improve performance when there are identical samples
> in a query. For example, sorting samples in SSD.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/mlcommons/inference/issues/1950#issuecomment-2502506179>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AE7XMAXQPL3GUTYZUNGF7XT2CUSVVAVCNFSM6AAAAABSKHQRN6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBSGUYDMMJXHE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


@arjunsuresh 
In the taskforce meeting held on 12/02/2024, we decided to add the extra sentence specific to unique sample in R-GAT.
The following techniques are disallowed:
Techniques that reduce the number of unique samples in a query. For example, sorting samples in Graph Neural Networks' RGAT.

I am not very familiar with MLCommons Github practices. Can you please create a PR for this?

Sure @ukurkure . [Here](https://github.com/mlcommons/inference_policies/pull/303/files) is the PR.
Repository: mlcommons/inference
Issue: [Llama3] Docker dependencies issues
Root cause: Configuration Error

The following issues appear when running the LLM reference implementation

Dependencies in the docker container:
```
Collecting mistral-common>=1.4.4 (from mistral-common[opencv]>=1.4.4->vllm==0.6.3->-r requirements.txt (line 8))
  Downloading mistral_common-1.5.0-py3-none-any.whl.metadata (4.6 kB)
  Downloading mistral_common-1.4.4-py3-none-any.whl.metadata (4.6 kB)
ERROR: Cannot install -r requirements.txt (line 8), mistral-common and sentencepiece==0.1.99 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested sentencepiece==0.1.99
    vllm 0.6.3 depends on sentencepiece
    mistral-common 1.5.1 depends on sentencepiece==0.2.0
    The user requested sentencepiece==0.1.99
    vllm 0.6.3 depends on sentencepiece
    mistral-common 1.5.0 depends on sentencepiece==0.2.0
    The user requested sentencepiece==0.1.99
    vllm 0.6.3 depends on sentencepiece
    mistral-common 1.4.4 depends on sentencepiece==0.2.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
```
Repository: mlcommons/inference
Issue: [Llama3] Error when multiple GPUs are used
Root cause: Resource Mishandling

The following issues appear when running the LLM reference implementation. Multiple GPUs issue:
```
(VllmWorkerProcess pid=1795) ERROR 12-03 18:49:03 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
(VllmWorkerProcess pid=1795) ERROR 12-03 18:49:03 multiproc_worker_utils.py:231]   File "/home/zhihanj/.local/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
(VllmWorkerProcess pid=1795) ERROR 12-03 18:49:03 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=1795) ERROR 12-03 18:49:03 multiproc_worker_utils.py:231]   File "/home/zhihanj/.local/lib/python3.10/site-packages/vllm/worker/worker.py", line 166, in init_device
(VllmWorkerProcess pid=1795) ERROR 12-03 18:49:03 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
(VllmWorkerProcess pid=1795) ERROR 12-03 18:49:03 multiproc_worker_utils.py:231]   File "/home/zhihanj/.local/lib/python3.10/site-packages/torch/cuda/__init__.py", line 420, in set_device
(VllmWorkerProcess pid=1795) ERROR 12-03 18:49:03 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
(VllmWorkerProcess pid=1795) ERROR 12-03 18:49:03 multiproc_worker_utils.py:231]   File "/home/zhihanj/.local/lib/python3.10/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
(VllmWorkerProcess pid=1795) ERROR 12-03 18:49:03 multiproc_worker_utils.py:231]     raise RuntimeError(
(VllmWorkerProcess pid=1795) ERROR 12-03 18:49:03 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
```
Repository: mlcommons/inference
Issue: what does sample/query mean in dlrm?

[inference_policies](https://github.com/mlcommons/inference_policies/tree/master)　says : 
dlrm sample meaning a batch of user-item paires with dist :100, 100, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 300, 300, 400, 500, 600, 700

![image](https://github.com/user-attachments/assets/8986973c-0cde-4865-975e-aead3e59e522)



when run mlperf dlrm debug, i got this result:
```
./run_local.sh pytorch dlrm debug cpu --scenario Offline --samples-to-aggregate-fix=2048 --max-batchsize=2048

TestScenario.Offline qps=0.04, mean=14.5861, time=24.326, queries=1, tiles=50.0:14.5861,80.0:14.5861,90.0:14.5861,95.0:14.5861,99.0:14.5861,99.9:14.5861

./run_local.sh pytorch dlrm debug cpu --scenario Server--samples-to-aggregate-fix=2048 --max-batchsize=2048

TestScenario.Server qps=0.89, mean=0.0357, time=601.533, queries=535, tiles=50.0:0.0309,80.0:0.0348,90.0:0.0398,95.0:0.0480,99.0:0.1899,99.9:0.2337

```

how can i calculate query(sample) / s from these result?
Repository: mlcommons/inference
Issue: llama3: performance_sample_count update
Root cause: Configuration Error

https://github.com/mlcommons/inference/blob/2d2eb3081dcda64e766a28dc1b6fb9c40fb9eefa/tools/submission/submission_checker.py#L432C28-L432C32

should this be 8313 as per length of 405b official dataset?

Addressed in https://github.com/mlcommons/inference/pull/1978/files
Repository: mlcommons/inference
Issue: why did the specified data precision not work for MLPerf inference?
Root cause: Documentation Deficiency

Run Resnet50 without the precision parameter in the docker successfully and get the result (measurements.json) that displays data type is int8.

Then run Resnet50 again with the precision=float16 as below in the docker successfully but still get the data type is int8 from the result (measurements.json).

It seems that the parameter of precision=float16 didn't take effect. How can I run the model in different data precision conveniently on MLPerf?

   cm run script --tags=run-mlperf,inference,_r4.1-dev \
   --model=resnet50 \
   **--precision=float16 \**
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=valid \
   --device=cuda \
   --division=closed \
   --rerun \
   --quiet

<pre>$ cat measurements.json 
{
  &quot;starting_weights_filename&quot;: &quot;https://zenodo.org/record/2592612/files/resnet50_v1.onnx&quot;,
  &quot;retraining&quot;: &quot;no&quot;,
  &quot;input_data_types&quot;: &quot;int8&quot;,
  &quot;weight_data_types&quot;: &quot;int8&quot;,
  &quot;weight_transformations&quot;: &quot;no&quot;
</pre>



Hi @Bob123Yang for nvidia implementation this is expected behaviour as the precision is automatically chosen by the implementation - often the best one satisfying the accuracy requirement for MLPerf. We don't have a choice to change this. 

Thank you @arjunsuresh , so do you mean non-nvidia implementation have the choice of changing the precision by the paramter for MLPerf?

You're welcome @Bob123Yang Actually what I told is also true for other vendor implementations like Intel, AMD, Qualcomm etc. Reference implementations usually have fp16 and fp32 options especially for the pytorch models. 

Oh it's a pity, thanks!  @arjunsuresh 

Could you help confirm one more question for NVIDIA multiple GPU scenario - how to run MLPerf inference on multiple GPUs which are connected with NVLink? Is there any parameter dedicated for that scenario or without any special parameter and just prepare the physical connection (such as NVLINK) working well for the multiple GPUs and then MLPerf running will automatically enable all GPU resources for usage?
Repository: mlcommons/inference
Issue: Metrics definition and settings used for LLM benchmarks
Root cause: Documentation Deficiency

Hello,
I have query regarding metrics definition and settings used in LLM benchmarks in MLCommons (https://mlcommons.org/benchmarks/inference-datacenter/).
For LLM-Q/A task in the benchmark table in above link:
1. How TPOT and TTFT are calculated? Can you source code for it? For TPOT how many tokens are generated?
2. For openocra dataset, input prompt provided is "system_prompt" + "question" or "question" only? Since openocra dataset has both colmuns
3. For quality metrics in table the values for ROUGE-1 is its precision , recall or fmeasure?




Hi @arjunsuresh, any information regarding this?
Repository: mlcommons/inference
Issue: mixtral-8x7b: Reference Implementation Accuracy Failure on H200 
Root cause: Algorithmic Error

When running reference implementation on H200, I see an accuracy failure: 

| Metric | Target Score | H200 Reference Implementation | Percentage Diff |
| ------- | -------------- | ----------------------------------- | ----------------- |
| rouge1 |	 45.5989 | 45.127 | 1.034893386 |
| rouge2 |	23.3526	| 22.9785 | 	1.601962951 |
| rougeL |	30.4608 |	30.4806 |	0.065001576 |
| gsm8k |	73.66 |	74.06 |	0.543035569 |
 |mbxp |	60.16 |	60.22 |	0.099734043 |
| tokens per sample |	144.84 |	283.5 |	95.73322287 |



@pgmpablo157321 @nvzhihanj @arjunsuresh : Any comments?

Hi @mrmhodak we are running the full accuracy run for this. But it won't be finishing until Thursday. 

We did the dataset update for Mixtral this round (for the EOS issue). Were you running on the latest dataset and latest settings (i.e. min_output_len=2)?
We will launch a local run to verify as well

@nvzhihanj : Yes, all latest, freshly downloaded according to latest instructions using rclone. 

@arjunsuresh @nvzhihanj @pgmpablo157321: Any update on this?

I am able to re-run the standalone script and double-check the accuracy of the model
```
Evaluating GSM8K score...
EM: 0.7366, correct: 3683 / 5000, gen_token_per_sample: 129.9604
Evaluating OpenOrca score...
OpenOrca score: {'rouge1': np.float64(45.5989), 'rouge2': np.float64(23.3526), 'rougeL': np.float64(30.4608), 'rougeLsum': np.float64(42.5396)}, gen_token_per_sample: 205.8656
Evaluating MBXP score...
100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 5000/5000 [02:33<00:00, 32.50it/s]
Processed 5000 in 153.89411109898356s
 60.16% pass@1
{'cpp': 381, 'typescript': 438, 'ruby': 419, 'python': 492, 'php': 809, 'javascript': 469}  out of  {'cpp': 743, 'typescript': 868, 'ruby': 846, 'python': 863, 'php': 846, 'javascript': 834}
gen_tokens_per_sample: 98.7026
```
The bug must be in the reference implementation FYI @pgmpablo157321 , I will check in the standalone script to the repo later.
One thing: please make sure you use the checkpoint downloaded from the mlcommon cloud, not the public one.

I added the reference standalone scripts in https://github.com/mlcommons/inference/pull/2029 and formalize the docker workflow. For the reference implementation, @pgmpablo157321 can you help the discrepancy between the standalone and the existing code?

@nvzhihanj Working on this
Repository: mlcommons/inference
Issue: nvmitten issue
Root cause: Configuration Error

Hi, days ago I was able to run the bert99 on my system, I was facing issues with other benchmarks, so I decided to rerun bert99 to make sure everything is fine. But it seems not. 

Here are the messages:
```bash
467.8 INFO:root:           ! cd /home/cmuser/CM/repos/local/cache/34233344dd0c42bc
467.8 INFO:root:           ! call /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/install.sh from tmp-run.sh
467.8 
467.8 /usr/bin/python3 -m pip install "/opt/nvmitten-0.1.3b0-cp38-cp38-linux_x86_64.whl"
468.3 WARNING: Requirement '/opt/nvmitten-0.1.3b0-cp38-cp38-linux_x86_64.whl' looks like a filename, but the file does not exist
468.3 Processing /opt/nvmitten-0.1.3b0-cp38-cp38-linux_x86_64.whl
468.3 ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/opt/nvmitten-0.1.3b0-cp38-cp38-linux_x86_64.whl'
468.3 
468.3 
468.3 Config file missing for given hw_name: 'buildkitsandbox', implementation: 'nvidia_original', device: 'gpu,  backend: 'tensorrt', copying from default
468.3             Detected version: 20.0.2
468.3 
468.3           Extra PIP CMD: 
468.3 
468.3           Detected version: 2022.2.2
468.3             Detected version: 20.0.2
468.3 
468.3           Extra PIP CMD: 
468.3 
468.3 
468.3 CM error: Portable CM script failed (name = get-generic-python-lib, return code = 256)
```

with the run command:
```bash
cm run script --tags=run-mlperf,inference,_find-performance,_full,_r5.0-dev \
   --model=bert-99 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=datacenter \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=500
```

Please help me to solve this issue

I ran the other benchmarks, and they all show the same error. I checked a previous similar issue but it was about a different environment setup.
My system is intel 14700 with A6000, it should work fine.

Update:
I tried bert99 on my other system with intel processor and a40 that also ran the benchmark properly last week. Maybe it should be any commit that alters the config?

@Xi0131 can you please try the same command but without "_r5.0-dev" in the tags?

@anandhu-eng do you know when this was modified? CM doesn't support "_r5.0-dev" for Nvidia implementation right? 

Hi @arjunsuresh , this issue arose when the inference documentation for Llama 3.1-405b was updated. I have created a PR to address it.

@Xi0131 Thanks for reporting. 

The issue is solved.
Thanks @arjunsuresh @anandhu-eng !
Repository: mlcommons/inference
Issue: Run command issue
Root cause: Unimplemented Feature Gap

Hi, I noticed there might be an upcoming version of benchmark v5.0 from the documents' run commands, such as
https://docs.mlcommons.org/inference/benchmarks/language/bert/#__tabbed_1_1

I have used the run command but it will get stuck at certain file downloads with no progress (0%)
Is v5.0's version of the run command ready to be executed?

related issue: #2024 

Hi @Xi0131 v5.0 can be tested for `reference` implementation but not for others. Officially it'll be ready for reference implementation this Friday and for other implementations by April 2025.
Repository: mlcommons/inference
Issue: Add code, measurement and results links in the result summary sheet
Root cause: Unimplemented Feature Gap

It'll be nice to add links to the Code, Measurement and Results folders in the below summary sheet. 

https://htmlpreview.github.io/?https://github.com/mlcommons/mlperf_inference_unofficial_submissions_v5.0/blob/refs/heads/auto-update/closed/MLCommons/results/mlc-server-reference-gpu-pytorch_v2.4.0-cu124/summary.html

Links are added now