zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: mlcommons_loadgen install erro
Root cause: Validation Gap

Processing /home/fangtao/inference/loadgen
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mlcommons_loadgen
  Building wheel for mlcommons_loadgen (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for mlcommons_loadgen (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running bdist_wheel
      running build
      running build_ext
      g++ -pthread -B /root/miniconda3/envs/llama2-70b/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/llama2-70b/include -I/root/miniconda3/envs/llama2-70b/include -fPIC -O2 -isystem /root/miniconda3/envs/llama2-70b/include -fPIC -I/root/miniconda3/envs/llama2-70b/include/python3.9 -c flagcheck.cpp -o flagcheck.o -std=c++17
      building 'mlperf_loadgen' extension
      g++ -pthread -B /root/miniconda3/envs/llama2-70b/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/llama2-70b/include -I/root/miniconda3/envs/llama2-70b/include -fPIC -O2 -isystem /root/miniconda3/envs/llama2-70b/include -fPIC -DMAJOR_VERSION=4 -DMINOR_VERSION=1 -I. -I/tmp/pip-build-env-ghw8xksd/overlay/lib/python3.9/site-packages/pybind11/include -I/root/miniconda3/envs/llama2-70b/include/python3.9 -c bindings/c_api.cc -o build/temp.linux-x86_64-cpython-39/bindings/c_api.o -std=c++17 -fvisibility=hidden -g0
      In file included from bindings/c_api.h:28,
                       from bindings/c_api.cc:13:
      bindings/../test_settings.h:267:1: error: version control conflict marker in file
        267 | <<<<<<< HEAD
            | ^~~~~~~
      bindings/../test_settings.h:274:1: error: version control conflict marker in file
        274 | =======
            | ^~~~~~~
      error: command '/usr/bin/g++' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for mlcommons_loadgen
Failed to build mlcommons_loadgen
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (mlcommons_loadgen)


This looks like due to a `git merge conflict`.

MLPerf loadgen installation is tested via Github actions and you can see the [latest runs here](https://github.com/mlcommons/inference/actions/workflows/test-loadgen.yml)
Repository: mlcommons/inference
Issue: TypeError
Root cause: External Dependency Breakage

python processorca.py --dataset_pq_path=${OPENORCA_PARQUET} --model_dir=${CHECKPOINT_PATH} --seqlen_limit=1024 --export_dir=${EXPORT_DIR} --num_total_samples=24576
Traceback (most recent call last):
  File "/home/fangtao/inference/language/llama2-70b/processorca.py", line 233, in <module>
    ds_gen.generate(
  File "/home/fangtao/inference/language/llama2-70b/processorca.py", line 182, in generate
    df = self.load_parquet()
  File "/home/fangtao/inference/language/llama2-70b/processorca.py", line 66, in load_parquet
    df = pd.read_parquet(self.pq_path)
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pandas/io/parquet.py", line 667, in read_parquet
    return impl.read(
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pandas/io/parquet.py", line 281, in read
    result = pa_table.to_pandas(**to_pandas_kwargs)
  File "pyarrow/array.pxi", line 885, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 5002, in pyarrow.lib.Table._to_pandas
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 781, in table_to_dataframe
    columns = _deserialize_column_index(table, all_columns, column_indexes)
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 913, in _deserialize_column_index
    columns = _pandas_api.pd.Index(
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 565, in __new__
    arr = sanitize_array(data, None, dtype=dtype, copy=copy)
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pandas/core/construction.py", line 654, in sanitize_array
    subarr = maybe_convert_platform(data)
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pandas/core/dtypes/cast.py", line 139, in maybe_convert_platform
    arr = lib.maybe_convert_objects(arr)
  File "lib.pyx", line 2538, in pandas._libs.lib.maybe_convert_objects
TypeError: Cannot convert numpy.ndarray to numpy.ndarray


@allthemight  may I ask what the solution was to this.
Repository: mlcommons/inference
Issue: TypeError
Root cause: External Dependency Breakage

python3 -u main.py --scenario Offline         --model-path ${CHECKPOINT_PATH}         --mlperf-conf mlperf.conf         --user-conf user.conf         --total-sample-count 24576         --dataset-path ${DATASET_PATH}         --output-log-dir offline-logs         --dtype float32         --device cuda:0 2>&1 | tee offline_performance_log.log
Traceback (most recent call last):
  File "/home/fangtao/inference/language/llama2-70b/main.py", line 96, in <module>
    main()
  File "/home/fangtao/inference/language/llama2-70b/main.py", line 69, in main
    sut = sut_cls(
  File "/home/fangtao/inference/language/llama2-70b/SUT.py", line 97, in __init__
    self.data_object = Dataset(self.model_path, dataset_path=self.dataset_path, total_sample_count=total_sample_count)
  File "/home/fangtao/inference/language/llama2-70b/dataset.py", line 30, in __init__
    self.load_processed_dataset()
  File "/home/fangtao/inference/language/llama2-70b/dataset.py", line 52, in load_processed_dataset
    input_tokens = processed_data['tok_input']
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pandas/core/frame.py", line 4078, in __getitem__
    return self._get_item_cache(key)
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pandas/core/frame.py", line 4639, in _get_item_cache
    res = self._ixs(loc, axis=1)
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pandas/core/frame.py", line 4010, in _ixs
    col_mgr = self._mgr.iget(i)
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 1016, in iget
    block = self.blocks[self.blknos[i]]
  File "/root/miniconda3/envs/llama2-70b/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 192, in blknos
    self._rebuild_blknos_and_blklocs()
  File "internals.pyx", line 761, in pandas._libs.internals.BlockManager._rebuild_blknos_and_blklocs
TypeError: Cannot convert numpy.ndarray to numpy.ndarray
Repository: mlcommons/inference
Issue: Loadgen Python API does not expose server_num_issue_query_threads
Root cause: Unimplemented Feature Gap

 Loadgen Python API does not expose `server_num_issue_query_threads` in mlperf::TestSettings. It seems users cannot set this attribute directly from Python.

Will create a PR to add this in pybind module.

@ever-wong Just to understand, is this parameter affecting performance for some systems?
Repository: mlcommons/inference
Issue: Disambiguating execution modes
Root cause: Documentation Deficiency

What are the difference between the `test` and `valid` modes list on the [documentation](https://docs.mlcommons.org/inference/benchmarks/language/llama2-70b/)? 

Since this is intended to be a reproducible bechmark suite, would it be possible to add clear instructions on which modes are to be run and the impact each step has? For instance, I cannot tell if the `test` execution mode is optional or if it is generating a target metric (as indicated by "record the estimated offline_target_qps" on the documentation)




Valid execution mode tries to do a valid run meeting the minimum duration of 10 minutes. But for this it relies on the user provided qps value in offline and server scenarios.

Test execution mode is just for testing (in the documentation it serves to launch the container too). But it also helps in determining the qps for the valid run. If user knows the qps, there's no need to do the test run.

We'll add these in the documentation. Please keep this issue open.
Repository: mlcommons/inference
Issue: Querying intermediate results
Root cause: Documentation Deficiency

I am running MLPerf Inference datacenter suite on a CPU only device following the instructions on the [documentation](https://docs.mlcommons.org/inference/benchmarks/language/llama2-70b/). 

The suggested sample size/query counts seem to take a very long time to reach completion. 

1. Would it be possible to query intermediate results (such as throughput) when the benchmark is executing? 
2. How are the sample sizes correlated with the accuracy of results? For instance, does llama2 CPU run need the same sample count (24576) as GPU? This is suggested [here](https://github.com/mlcommons/inference/tree/master/language/llama2-70b#offline)

I see the following prints on my terminal, but I am not sure how to interpret these results:
<img width="644" alt="image" src="https://github.com/user-attachments/assets/ce038e37-5f7e-48de-8261-68ee1817fad8">


1. You can do `--execution_mode=test - -test_query_count=100` and get a quick result. But this won't be accepted as official one.
2. Yes, minimum 24576 inputs need to be run for llama2. Accuracy value also changes if we run lower number of inputs. For this reason, no one has tried to do a llama2-70b submission on CPUs.
Repository: mlcommons/inference
Issue: CM error: Extracted TensorRT folder does not seem proper - Version information missing!
Root cause: Validation Gap

I'm getting this new error for tensorrt framework bert inference implementation on grace-hopper 200:

Here is the output:
```sh
Apptainer> cm run script "get tensorrt _dev" --tar_file=/my/path/TensorRT-10.1.0.27.Ubuntu-22.04.aarch64-gnu.cuda-12.4.tar.gz
INFO:root:* cm run script "get tensorrt _dev"
INFO:root:  * cm run script "detect os"
INFO:root:         ! cd /tmp/001/CM/repos/local/cache/e217b3c0efc44d75
INFO:root:         ! call /tmp/001/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /tmp/001/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:  * cm run script "get python3"
INFO:root:       ! load /tmp/001/CM/repos/local/cache/5126da4b74e84019/cm-cached-state.json
INFO:root:Path to Python: /usr/bin/python3
INFO:root:Python version: 3.10.12
Untaring file - can take some time ...

CM error: Extracted TensorRT folder does not seem proper - Version information missing!
Apptainer> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:24:28_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0
Apptainer>
```

I intend to run this command:
```sh
cm run script --tags=run-mlperf,inference,_r4.1 \
   --model=bert-99 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=valid \
   --device=cuda \
   --quiet \
```

Any hints?

@thehalfspace Nvidia implementation requires `--docker` option. Please follow this docs page for the latest commands.
https://docs.mlcommons.org/inference/benchmarks/language/bert/#__tabbed_50_1

But we haven't tested on GH200 yet. You might have to add `--batch_size=64` if the given run command fails.
Repository: mlcommons/inference
Issue: The documentation is not showing correctly...

Check https://docs.mlcommons.org/inference/benchmarks but found:
<img width="1422" alt="image" src="https://github.com/user-attachments/assets/4d258406-970e-48c3-a9c3-ff7a204cff79">
@ashwin @jdduke @codyaustun @badenh @koichishirahata 

The installation page does not exist...
Check https://docs.mlcommons.org/install but found:
<img width="1023" alt="image" src="https://github.com/user-attachments/assets/b565bedc-cba2-432e-b7ae-ccc21b69d2b5">


@zhimin-z hello, do you need this ? 

https://docs.mlcommons.org/inference/install/

Thank you @zhimin-z for reporting this. This PR should fix the `benchmarks` path not working. 

https://github.com/mlcommons/inference/pull/1843/files
Repository: mlcommons/inference
Issue: How to run text-to-image with multi GPUs
Root cause: Documentation Deficiency

Hello everyone, I am a newcomer to MLPerf.
I would like to know whether the text-to-image in inference supports multi-card testing.  Currently, I see that there is no parameter to set multi-card in the parameters of " python main.py --help ".
and I saw the MLPerf inference Results, it has results of  2 L40s and 8 H100 . how do they run the test? 
thanks a lot 

Hi @surbanqq! Reference code often supports only a single accelerator. But for their submissions vendors optimize including scaling to multiple accelerators. In the case of NVIDIA, please take a look at [their v4.1 submission](https://github.com/mlcommons/inference_results_v4.1/blob/main/closed/NVIDIA/code/stable-diffusion-xl/tensorrt/README.md).

thanks a  lot
Repository: mlcommons/inference
Issue: Problems with cmtools
Root cause: Environment Incompatibility

Hi! I recently changed the video cards on my test bench. When trying to run familiar tests, I encountered errors.

I entered: cm run script --tags=run-mlperf,inference,_find-performance,,_full_r4.1-dev \ --model=retinanet \ --implementation=nvidia \ --framework=tensorrt \ --category=edge \ --scenario=Offline \ --execution_mode=test \ --device=cuda \ --docker --quiet \ --test_query_count=500

And got errors: 
INFO:root:* cm run script "run-mlperf inference _find-performance _full_r4.1-dev" CM error: no scripts were found with above tags and variations variation tags ['find-performance', 'full_r4.1-dev'] are not matching for the found script run-mlperf-inference-app with variations dict_keys(['accuracy-only', 'all-modes', 'all-scenarios', 'compliance', 'dashboard', 'find-performance', 'full', 'performance-only', 'populate-readme', 'r2.1', 'r3.0', 'r3.1', 'r4.0-dev', 'r4.0', 'r4.1-dev', 'r4.1', short', 'performance-and-accuracy', 'submission']) 

Oh, I entered: 
(Last lines of the error message)

```

 /usr/include/x86_64-linux-gnu/bits/mathcalls.h(110): error: identifier "_Float32" is undefined
 
 Error limit reached.
 100 errors detected in the compilation of "print_cuda_devices.cu".
 Compilation terminated.
 
 CM error: Portable CM script failed (name = get-cuda-devices, return code = 256)
```

Full log [2a100_issue_2510.log](https://github.com/user-attachments/files/17129457/2a100_issue_2510.log)



The docs are updated now - sorry, it had a typo before. 
https://docs.mlcommons.org/inference/benchmarks/object_detection/retinanet/

Hi @arjunsuresh ! 
If the documents are currently being updated, please don't be rude, but can I ask when they will be ready?

Should I try using benchmarking methods from here? https://github.com/mlcommons/inference/tree/master/language/bert
Or from here?
https://github.com/mlcommons/inference/tree/master/vision/medical_imaging/3d-unet-kits19

@Agalakdak No, docs should be stable now. Unfortunately it had a bug in the previous update - which caused an extra "," coming. 

We are currently adding self hosted github actions to do nightly runs on Nvidia GPUs. Have added [gptj](https://github.com/mlcommons/cm4mlops/blob/main/.github/workflows/test-mlperf-inference-gptj.yml) and [stable diffusion](https://github.com/mlcommons/cm4mlops/blob/mlperf-inference/.github/workflows/test-mlperf-inference-sdxl.yaml) so far and expected to complete the rest by next week. So, the runs should be more stable after that. 

The readme files in the inference repository are meant for benchmark developers to create their own implementation. They normally are not helpful in benchmarking as the READMEs are not updated after the creation of the reference implementation.  

@arjunsuresh 
Thanks, it seems to work now!) I have accumulated quite a few questions regarding benchmarks. Should I create a separate top for them or can I list them here?
Repository: mlcommons/inference
Issue: Ensure the unique IDs assigned to each SUT in the results table are not changing when result spreadsheet is regenerated. 
Root cause: Unimplemented Feature Gap

Currently each time the result table is regenerated the unique IDs of systems are changing if any new result is added or if some result is dropped. It will be good to ensure they remain consistent.

# Proposal: Change of IDs in the Result Table

We are proposing to change the IDs in the result table as follows:

## Current IDs
```
4.1-0001     AMD     .... (System1)
4.1-0002    AMD .... (System2)
4.1-0003    ASUSTek ... (System1)
....
```  
## Proposed Change (Step 1)
```
4.1-001-001     AMD     .... (Submitter1, System1)
4.1-001-002    AMD .... (Submitter1, System2)
4.1-002-001    ASUSTek ... (Submitter2, System1)
....
```


**Explanation**:  
- We are introducing a separate ID component for every submitter. This component remains constant even when results are regenerated. 
- The format allows for up to 999 submitters and 999 systems per submitter.
- This new format ensures that all submission IDs from a given submitter will always be contiguous when we preserve the already generated IDs.

## Question: Should this ID be unique across different submission rounds?

Additionally, we need to maintain a record of previously generated IDs for each system (within the same round) so that when results are regenerated:
- IDs are first assigned based on the historical record.
- New systems will receive newly generated IDs.

### Key Considerations:
1. **Withdrawals**: Corresponding IDs will remain unused.
2. **New Results**: Any new results (e.g., added by review committee decision) will receive new IDs that have not been used in previously generated results.

## Advantages

1. **Consistency**: Immediately after submissions, submitters will receive IDs for their results that will not change based on withdrawals or additions by other submitters.
   
2. **Reproducibility**: Anyone can run the submission checker on the submissions repository and regenerate the results with the same IDs. All that is required is to save the previously generated IDs in the repository.


Good suggestion. @mrmhodak to bring for discussion next week

@arjunsuresh How should we handle when a system is renamed? I see a couple of options here, but not sure if they are a what we are looking for

- We can have a new ID when the system is renamed
- We can keep track the folder names, and assign the IDs based on that. That means, they wont change if the system is renamed, but when the folder changes. This might also happen during review
- Provide an option to map names when they are changed. Not sure how this will work, and if we have to do additional checks

To be discussed 12/3
Repository: mlcommons/inference
Issue: Enable max duration in loadgen
Root cause: Unimplemented Feature Gap

It'll be good to enable max duration in the loadgen like done [here](https://github.com/mlcommons/inference/compare/master...mobile) so that we can do a test run for a fixed duration and automatically determine the throughput of the system. @pgmpablo157321 is my understanding correct here?

@arjunsuresh and @pgmpablo157321 to work on this
Repository: mlcommons/inference
Issue: Benchmarking Question List #1

Hello everyone.
I have been using MLperf benchmarks for some time. And I have a small list of questions about them. I am asking them here because I have not found answers in other sources of information.

1) I have several video cards in my system. Can I explicitly set the number of video cards for the test?

2) This question follows from the question above. Do all tests use all available GPUs?

3) Many tests have different profiles like "edge"
"datacenter" what is the difference between them?

4) Since the space on my SSD is limited, how can I tell the benchmarks to use a different directory to store the cache?

5) The tests (in the profiles that I used) do not always use 100% of the video memory. Are there any scenarios for which all the video memory will be used or is this not necessary?

6) Perhaps there are more subtle benchmark settings, is there any user guide.

Hi @Agalakdak Some of your questions are "benchmark implementation" dependent and we currently have Nvidia, Intel, and Reference implementations for most/all of the benchmarks and other vendor implementations are available for some of the benchmarks. 

1. "no" for most of the reference implementations except some like for llama2. "yes" for Nvidia implementation though it uses all the GPUs by default.
2. For Nvidia implementation - "yes". For reference implementation, it uses 1 GPU by default and in some benchmark implementations it supports multiple GPUs.
3. Those are 2 different submission categories. The required scenarios to be run differs for them and "Offline" scenario is the only common one for both.
4. `export CM_REPOS=<NEW_PATH>` can be used to do this or we can create softlink for any folder inside $HOME/CM/repos/local/cache path. 
5. Many small inference models do not need large amount of GPU memory. [Parameter size](https://docs.mlcommons.org/inference/) given here is usually a good guide for the required GPU memory.
6. Unfortunately not much currently - as most implementations by default only support the systems on which the MLPerf results were submitted. We are trying to extend this - but it is a WIP and the implementations and the results changes every 6 months. 

@arjunsuresh , Thank you! And the last 2 questions for now. 
1) How can benchmarking results be interpreted? Is this some abstract metric or can the data be interpreted as "Model A can process x requests per second". 
2) How can I donate $10

1. For offline scenario - samples per second  is the usual metric. Requests per second or queries per second may not be correct as a single request or query can contain multiple samples. But for LLMs it is often tokens per second. 
2. I don't think MLCommons is taking donations but I might be wrong. You can contact the right people [here](https://mlcommons.org/community/)