zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: RetinaNet 800x800 ValueError: no images in image list found
Root cause: Documentation Deficiency

I try to run RetinaNet 800x800 sample.

when I run

`./run_local.sh pytorch retinanet gpu --scenario Offline`

I got the error message

```
ERROR:coco:no images in image list found
Traceback (most recent call last):
  File "/root/inference/vision/classification_and_detection/python/main.py", line 615, in <module>
    main()
  File "/root/inference/vision/classification_and_detection/python/main.py", line 482, in main
    ds = wanted_dataset(data_path=args.dataset_path,
  File "/root/inference/vision/classification_and_detection/python/openimages.py", line 113, in __init__
    raise ValueError("no images in image list found")
ValueError: no images in image list found
```

My json path is

`/root/open-images-v6-mlperf/annotations/openimages-mlperf.json`

here is files tree.

```
./
├── annotations
│   ├── class-descriptions-boxable.csv
│   ├── openimages-mlperf.json
│   └── validation-annotations-bbox.csv
├── calibration
│   ├── annotations
│   │   ├── class-descriptions-boxable.csv
│   │   ├── oidv6-train-annotations-bbox.csv
│   │   └── openimages-calibration-mlperf.json
│   └── calibration
│       └── data
│           ├── 0001a2f47097ee27.jpg
.............
│           └── ff2a750ec0048d00.jpg
└── validation
    └── data
        ├── 0001eeaf4aed83f9.jpg
..........
        └── ffff21932da3ed01.jpg

7 directories, 25287 files
```

I tried put image to different path, but still no work.

and I read the code, still no clue with image path.

where is the image actual place??

my environment
```
ubuntu 22.04
ROCm 5.6
torch 2.1.0.dev20230813+rocm5.6
onnxruntime-rocm 1.16.0
```

I found issue #930 is like my problem.
but that problem is no files.
I do have files, so I really no clue with this issue.

BTW, Resnet-50 v1 benchmark is work perfectly, so I think my python package maybe ok?

Sorry, I do not remember how the paths are supposed to be defined but these are done properly in CM. You can follow the below README and in the console output of the run command, you can see how the paths are getting used.  

https://github.com/mlcommons/ck/tree/master/docs/mlperf/inference/retinanet

Hi, I found the Images needs place in

`/root/open-images-v6-mlperf/`

So, problem is sloved.

thanks for reply.
Repository: mlcommons/inference
Issue: cannot access gpt3 checkpoint bucket
Root cause: Environment Incompatibility

I cannot get access gpt3 checkpoint on branch gpt3_reference_implementation. I followed this command https://github.com/mlcommons/inference/tree/gpt3_reference_implementation/language/gpt3/megatron#download-gpt-3-model to download the checkpoint for inference. But it seems that I didn't have the access. How can I download the checkpoint? please help. @pgmpablo157321
Repository: mlcommons/inference
Issue: Python 3.11 - Building loadgen has issues with pinned third_party/pybind version
Root cause: External Dependency Breakage

When I follow the loadgen build instructions, I get:

```
../third_party/pybind/include/pybind11/cast.h:442:36: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
        442 |                 "  " + handle(frame->f_code->co_filename).cast<std::string>() +
            |                                    ^~
      In file included from /opt/conda/envs/firesim/include/python3.11/Python.h:42,
                       from ../third_party/pybind/include/pybind11/detail/common.h:112,
                       from ../third_party/pybind/include/pybind11/pytypes.h:12,
                       from ../third_party/pybind/include/pybind11/cast.h:13,
                       from ../third_party/pybind/include/pybind11/attr.h:13,
                       from ../third_party/pybind/include/pybind11/pybind11.h:44,
                       from ../third_party/pybind/include/pybind11/functional.h:12,
                       from bindings/python_api.cc:27:
```

I looked up this issue online, and found that this was a change in Python 3.11: [https://github.com/pybind/pybind11/discussions/4333](https://github.com/pybind/pybind11/discussions/4333)

It has already been fixed in pybind11; however, the mlcommons/inference project has an old (~5 years) pybind version [pinned](https://github.com/mlcommons/inference/tree/master/third_party) in [third_party/pybind](https://github.com/pybind/pybind11/tree/25abf7efba0b2990f5a6dfb0a31bc65c0f2f4d17)

By pulling the latest pybind version, I was able to compile loadgen with no problems.

Is there a reason for this (e.g. consistency with old versions)? If not, I would suggest either:
- Update the pinned pybind version to a more recent release, so that Python 3.11 ppl can compile loadgen easier
- Put in the install/build instructions a notice that Python 3.11 will require updating pybind

Thanks!

[This PR](https://github.com/mlcommons/inference/pull/1434) should solve this. We can see this when the Inference WG meeting resumes from next week. 

[This](https://github.com/mlcommons/inference/pull/1271) is also a possible solution.
Repository: mlcommons/inference
Issue: Is it possible to calculate the LLM performance, Token per Second ?
Root cause: Documentation Deficiency

Hello, I get the inference result v3.1 and analysis the performance, throughput (token per second).
In the large language model task, the test results are measured by Queries/s and Samples/s.

I understand that query is a similar concept to batch and includes multiple samples.
Then i can estimate the number of sentences, but i don't know the specific number of input/output token length.
In the source code, there is only maximum number of input/output token length. 
Is there any way to estimate the throughput performance in terms of token per second metric ?

Thank you.

@ConstantPark 

I'm the maintainer of LiteLLM we provide an Open source proxy for load balancing Azure + OpenAI + Any LiteLLM supported LLM
**It can process (500+ requests/second)**

From this thread it looks like you're trying to maximize throughput - I hope our solution makes it easier for you. **(i'd love feedback if you're trying to do this)**

## Here's the quick start:
Doc: https://docs.litellm.ai/docs/simple_proxy#load-balancing---multiple-instances-of-1-model

## Step 1 Create a Config.yaml
```python
model_list:
  - model_name: gpt-4
    litellm_params:
      model: azure/chatgpt-v-2
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
      api_version: "2023-05-15"
      api_key: 
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: 
      api_base: https://openai-gpt-4-test-v-2.openai.azure.com/
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: 
      api_base: https://openai-gpt-4-test-v-2.openai.azure.com/
```

## Step 2: Start the litellm proxy:
```
litellm --config /path/to/config.yaml
```

## Step3 Make Request to LiteLLM proxy:
```
curl --location 'http://0.0.0.0:8000/chat/completions' \
--header 'Content-Type: application/json' \
--data ' {
      "model": "gpt-4",
      "messages": [
        {
          "role": "user",
          "content": "what llm are you"
        }
      ],
    }
'
```
Repository: mlcommons/inference
Issue: Offline scenario inferred from SingleStream gives incorrect power efficiency
Root cause: Algorithmic Error

For `<sut1>`, resnet50 Offline was inferred from SingleStream scenario, the results generated by `submission_checker.py` are shown below. The power efficiency calculated is `power_efficiency (inf/J) = 3448436.324053`, which is indeed a nonsensical figure.

```
INFO    Results closed/<submitter>/results/<sut1>/resnet50/multistream 378.649068 with power_metric = 5218.764205 and power_efficiency (inf/J) = 1.532930
INFO    Results closed/<submitter>/results/<sut1>/resnet50/offline 21.008400 with power_metric = 13.767718 and power_efficiency (inf/J) = 3448436.324053
INFO    Results closed/<submitter>/results/<sut1>/resnet50/singlestream 47.686353 with power_metric = 655.611514 and power_efficiency (inf/J) = 1.525294

INFO    Results open/<submitter>/results/<sut2>/resnet50/multistream 2.292281 with power_metric = 47.788945 and power_efficiency (inf/J) = 167.402734
INFO    Results open/<submitter>/results/<sut2>/resnet50/offline 6580.530000 with power_metric = 25.936246 and power_efficiency (inf/J) = 253.030061
INFO    Results open/<submitter>/results/<sut2>/resnet50/singlestream 0.647587 with power_metric = 13.885169 and power_efficiency (inf/J) = 72.019287
```

Thank you @sahelib25 for reporting this issue. [This PR](https://github.com/mlcommons/inference/pull/1495) should fix it.
Repository: mlcommons/inference
Issue: [DLRM_v2] Torch Reshape Error during loading model weights
Root cause: Validation Gap

I set up the (CPU) docker container, processed the dataset, and downloaded the model weights as mentioned in the README file. When I try to run the benchmark with dummy weights on the full dataset (eg, `./run_local.sh multihot-criteo cpu --max-ind-range=10000000 --debug`) it runs fine and reports stats. However, when I try to use the actual model weights (by removing the `--debug` option), it results in the following error:
```
Loading model weights...
INFO:torchsnapshot.scheduler:Set process memory budget to 34359738368 bytes.
Traceback (most recent call last):
  File "python/main.py", line 566, in <module>
    main()
  File "python/main.py", line 448, in main
    model = backend.load(args.model_path, inputs=args.inputs, outputs=args.outputs)
  File "/root/mlcommons/recommendation/dlrm_v2/pytorch/python/backend_pytorch_native.py", line 137, in load
    snapshot.restore(app_state={"model": self.model})
  File "/root/.local/lib/python3.7/site-packages/torchsnapshot/snapshot.py", line 411, in restore
    event_loop=event_loop,
  File "/root/.local/lib/python3.7/site-packages/torchsnapshot/snapshot.py", line 662, in _load_stateful
    event_loop=event_loop,
  File "/root/.local/lib/python3.7/site-packages/torchsnapshot/scheduler.py", line 459, in sync_execute_read_reqs
    rank=rank,
  File "/opt/anaconda3/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/root/.local/lib/python3.7/site-packages/torchsnapshot/scheduler.py", line 437, in execute_read_reqs
    read_pipeline: _ReadPipeline = d.result()
  File "/root/.local/lib/python3.7/site-packages/torchsnapshot/scheduler.py", line 377, in consume_buffer
    await self.read_req.buffer_consumer.consume_buffer(self.buf, executor)
  File "/root/.local/lib/python3.7/site-packages/torchsnapshot/io_preparers/sharded_tensor.py", line 300, in consume_buffer
    buf=buf, entry=self.entry
  File "/root/.local/lib/python3.7/site-packages/torchsnapshot/io_preparers/tensor.py", line 324, in deserialize_tensor
    memoryview(buf), dtype=dtype, shape=entry.shape
  File "/root/.local/lib/python3.7/site-packages/torchsnapshot/serialization.py", line 244, in tensor_from_memoryview
    return torch.reshape(torch.frombuffer(mv, dtype=dtype), shape)
RuntimeError: shape '[1048576, 128]' is invalid for input of size 65536000
```

I assume there is something wrong with the `.snapshot_metadata` file, but I am not sure what the correct dimensions should be/how to fix it. I would appreciate it if someone could throw some light on the issue.
Repository: mlcommons/inference
Issue: Request NVIDIA to make container BASE IMAGE links the publicly available links
Root cause: Documentation Deficiency

For reproducibility we would request NVIDIA to switch the `BASE_IMAGE` repo links from internal Gitlab links to publicly available repo links. In the `Makefile.docker`, the `BASE_IMAGE` URL is from an internal NVIDIA link. Here is the link to the `Makefile.docker` from the public MLPerf Inference v3.1 results repo:

https://github.com/mlcommons/inference_results_v3.1/blob/main/closed/NVIDIA/Makefile.docker

Hi, https://github.com/mlcommons/inference_results_v3.1/blob/main/closed/NVIDIA/Makefile.docker#L142 BASE_IMAGE is public, please check if you can download it via docker pull.
Repository: mlcommons/inference
Issue: Open Division Rule Clarification Request

According to https://github.com/mlcommons/inference/blob/master/Submission_Guidelines.md#expected-time-to-do-benchmark-runs 

There is no constraint on the model used also except that the model must be trained on the dataset used in the corresponding MLPerf inference task.

By "the dataset used in the corresponding MLPerf inference task", does it mean the validation set used for accuracy testing in the closed division's same workload? It does not quite make sense to enforce this so that retraining of the open division model can be overfitted to the validation dataset.

If it means the original dataset used to train the closed division model, what would the submitters use if the original dataset is not publicly available? In this case this rule needs to be relaxed, such as the submitters can choose their own training dataset except the validation dataset and have to publish the dataset they used for training as a complementary material for open division submission.

@mrmhodak We need to discuss this ASAP in WGM.

>  except that the model must be trained 

I think "trained" should be replaced with "validated".

> such as the submitters can choose their own training dataset except the validation dataset and have to publish the dataset they used for training as a complementary material for open division submission.

Requiring to publish the training dataset may be a step too far?
Repository: mlcommons/inference
Issue: v3.1 results in tableau - no bert-{99,99.9} division
Root cause: Unimplemented Feature Gap

![image](https://github.com/mlcommons/inference/assets/6597818/decbd611-9b44-4413-bb02-895b2fd8aef9)


Network is proper:

![image](https://github.com/mlcommons/inference/assets/6597818/c476f532-cc91-4347-9fc8-0acff453eebd)


Some more issues I have noticed

1. I think all the columns of the old results table must be there in Tableau -  important information like Software/framework, open division model names, hardware/software notes, open division model accuracy etc. are not visible.
2. gptj-99.9 missing in the edge category
3. I think the old spreadsheet link can still be given in the webpage as it has more data and is useful for those working on spreadsheets

This issue is being fixed, will be updated soon.

@arjunsuresh in regards of the collumns, we can add them in the details of each line as hovers too, lets dicuss what is the best option.

Thank you @peladodigital. Displaying as hovers looks fine to me. 

Closing, the issue is fixed.
Repository: mlcommons/inference
Issue: Open division rule on validation dataset is confusing

https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#412-relaxed-constraints-for-the-open-division

Rule 2 states that: 

> The accuracy dataset must be the same as used in an existing Closed benchmark, or must be pre-approved and added to the following list: ImageNet 2012 validation dataset for Image Classification; COCO 2017 validation dataset for Object Detection. From v3.0, if a submitter provides any results with any models trained on a pre-approved dataset, the submitter must also provide at least one result with the corresponding Closed model trained (or finetuned) on the same pre-approved dataset, and instructions to reproduce the training (or finetuning) process.

There are some problems with this rule:

- The sentences mix up the validation dataset with the training dataset. It is not clear in some places how or why validation/training dataset would be applicable in that sentence.
- Probably the rule introduced for v3.0 could be simplified to be generic across all benchmarks that will be in 4.0.

I am requesting @psyhtest and @rnaidu02 to help in formulating a simpler rule 2 for Open.

Based on the WGM discussion, @psyhtest will look at removing/simplifying the "v3.0" sentence. I will try to add a new rule to cover the training dataset. Discussion on these changes can be done in next week's WGM.
Repository: mlcommons/inference
Issue: Compliance TEST05 fails for Retinanet Server scenario
Root cause: Validation Gap

Here is the result from `verify_performance.txt` for Retinanet Server run, 
```
Verifying performance.
reference score = 2228.87
test score = 2223.09
TEST FAIL: Test score invalid
```
However, all the runs for compliance tests are `VALID` runs. The `verify_performnace.py` does not account for `Server` scenario [here](https://github.com/mlcommons/inference/blob/0f71999f0b72cc02d667efd5f009521540fd14be/compliance/nvidia/TEST05/verify_performance.py#L142).
Repository: mlcommons/inference
Issue: The size of `.git` directory is too large
Root cause: Architectural Constraint

The whole code file is about 2M, but the `.git` subdirectory is over 400M.

Unfortunately that's a known issue. While checking out we use `--depth 1` to reduce the download size. I'm not sure if we can rewrite the git history to reduce the size. 

> Unfortunately that's a known issue. While checking out we use `--depth 1` to reduce the download size. I'm not sure if we can rewrite the git history to reduce the size.

Thanks!
Repository: mlcommons/inference
Issue: need to run this command from the toplevel of the working tree.
Root cause: Validation Gap

[root@poc4 bert]# make setup
make[1]: Entering directory '/home/sunyu/benchmark/MLPerf/inference/language/bert'
You need to run this command from the toplevel of the working tree.
make[1]: *** [init_submodule] Error 1
make[1]: Leaving directory '/home/sunyu/benchmark/MLPerf/inference/language/bert'
make: *** [setup] Error 2

--------------------------------------
Why did I get this error when I ran `make setup` on node 68, but it worked on node 69? The 68-node configuration is identical to the 69-node configuration.
Although this error has been fixed, the solution:
1.`rm -rf DeepLearningExamples`
2.`git submodule add < repository_url> DeepLearningExamples`
3.`make setup`

So my question is why did it go wrong at 68-node, and why did my approach work?
Repository: mlcommons/inference
Issue: [llama] FirstTokenComplete is required in the Offline scenario
Root cause: Validation Gap

@pgmpablo157321  Second issue discussed on the today taskforce meeting.

Reproducer: [first_token_in_offline_issue.py.txt](https://github.com/mlcommons/inference/files/13821867/first_token_in_offline_issue.py.txt)

Output:
```
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 459.194
Tokens per second: 0
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes

...

2750 ERRORS encountered. See detailed log.

detailed logs:
:::MLLOG {"key": "error_runtime", "value": "Attempted to record a sample latency before it's first token latency", "time_ms": 0.168189, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 407, "pid": 918835, "tid": 918835}}
```

Expected output:
```
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 459.194
Tokens per second: 2755.16
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
```

Why?

Measuring first token latency in the Offline scenario does not make sense => we have one query and we measure its processing time. First token latency only introduces unnecessary overhead and confusion here.
Repository: mlcommons/inference
Issue: nvidia submission diff from officail require
Root cause: Documentation Deficiency

I see the official website resnet50 guidelines, require QSL Size=1024 and Quality = 99% of FP32 (76.46%)，from nvidia v2.1 result, it use QSL Size=2048 and precision = "int8", so is it meet the requirements? Thanks!

It is. For ResNet50, minimum QSL size is 1024. Any higher value is permitted. But in most systems both 1024 and 2048 should give the same performance.

Any precision is also allowed as long as the accuracy is within 99% of the fp32 accuracy value.
Repository: mlcommons/inference
Issue: issue while inference/recommendation/dlrm/pytorch/ running with CPU docker.
Root cause: Validation Gap

Hi,

I was running dlrm pytorch with CPU docker by using fake data. seeing below error.

/root/mlcommons/recommendation/dlrm/pytorch/python/dlrm_data_pytorch.py:328: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/torch/csrc/utils/tensor_new.cpp:230.)
  X_int = torch.log(torch.tensor(transposed_data[0], dtype=torch.float) + 1)
Traceback (most recent call last):
  File "python/main.py", line 619, in <module>
    main()
  File "python/main.py", line 535, in main
    _ = backend.predict(batch_dense_X, batch_lS_o, batch_lS_i)
  File "/root/mlcommons/recommendation/dlrm/pytorch/python/backend_pytorch_native.py", line 125, in predict
    output = self.model(dense_x=batch_dense_X, lS_o=batch_lS_o, lS_i=batch_lS_i)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/mlcommons/recommendation/dlrm/pytorch/python/dlrm_s_pytorch.py", line 528, in forward
    return self.sequential_forward(dense_x, lS_o, lS_i)
  File "/root/mlcommons/recommendation/dlrm/pytorch/python/dlrm_s_pytorch.py", line 600, in sequential_forward
    ly = self.apply_emb(lS_o, lS_i, self.emb_l, self.v_W_l)
  File "/root/mlcommons/recommendation/dlrm/pytorch/python/dlrm_s_pytorch.py", line 459, in apply_emb
    per_sample_weights=per_sample_weights,
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 391, in forward
    self.padding_idx)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2393, in embedding_bag
    weight, input, offsets, scale_grad_by_freq, mode_enum, sparse, per_sample_weights, include_last_offset, padding_idx
**RuntimeError: Index 0 is out of bounds: 11395, range 0 to 11156**

Command i ran using CPU docker: ./run_local.sh terabyte cpu --max-ind-range=10000000

Machine used: x86_64 GNU/Linux

**Steps followed: **
cd $HOME/mlcommons/inference/recommendation/dlrm/pytorch/docker_cpu
./build_docker_cpu.sh
cd $HOME/mlcommons/inference/recommendation/dlrm/pytorch/docker_cpu
./run_docker_cpu.sh
cd mlcommons/recommendation/dlrm/pytorch
./run_local.sh terabyte cpu --max-ind-range=10000000

please provide the solution to this.

Thanks,
Siva
Repository: mlcommons/inference
Issue: Does the tensorflow version of DLRM support cpus?

Does the tensorflow version of DLRM support cpus?

Actually, the only active version of dlrm is [dlrm_v2](https://github.com/mlcommons/inference/tree/master/recommendation/dlrm_v2/pytorch) and it has only pytorch backend in the reference implementation. AFAIK even the old DLRM TF supports only google TPU.