zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: [Multimodal][vl2l] Offline Scenario hangs at the end of the benchmark
Root cause: Algorithmic Error

The command:
```mlperf-inf-mm-vl2l --settings.scenario offline --settings.mode performance_only```
might hang if the expected qps is too high.

You can try:
```mlperf-inf-mm-vl2l --settings.scenario offline --settings.mode performance_only --settings.offline_expected_qps 10```
and check if you can reproduce.

@wangshangsam 

If we give expected qps as 1, loadgen will generate a minimum of 660*1 samples. If qps is given as 10, this will be 660*10 samples and so on.
Repository: mlcommons/inference
Issue: ResNet50 5.1-dev/5.0
Root cause: Validation Gap

Hi, I recently tried to run ResNet50 and a couple of other networks on my system.

But I ran into an error.

``` bash
AttributeError: 'ResNet50EngineBuilder' object has no attribute 'calibrator'. Did you mean: 'get_calibrator'?
[2025-11-26 13:51:10,841 generate_engines.py:171 INFO] Building engines for resnet50 benchmark in Offline scenario...
[11/26/2025-13:51:11] [TRT] [I] [MemUsageChange] Init CUDA: CPU +23, GPU +0, now: CPU 48, GPU 870 (MiB)
[11/26/2025-13:51:13] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +2771, GPU +446, now: CPU 3020, GPU 1316 (MiB)
```

[error resnet50.txt](https://github.com/user-attachments/files/23772864/error.resnet50.txt)

Run from https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/#__tabbed_120_1

Command 
``` bash
mlcr run-mlperf,inference,_find-performance,_full,_r5.1-dev \
   --model=resnet50 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=5000 --rerun
```

Can you retry the command now? 

If inside the docker container, please do `mlc pull repo` and rerun the mlperf command.
Repository: mlcommons/inference
Issue: Using enforce_eager=True in llama-3.1-8B
Root cause: Documentation Deficiency

Hi,
How can I set the "enforced eager" mode in Llama-3.1-8B command line? For this command:

```
python -u main.py \
  --scenario Offline \
  --model-path $CHECKPOINT_PATH \
  --batch-size $BATCH_SIZE \
  --dtype bfloat16 \
  --user-conf user.conf \
  --total-sample-count 1 \
  --dataset-path $DATASET_PATH \
  --output-log-dir output \
  --tensor-parallel-size $GPU_COUNT \
  --vllm
```
I see this message:
```
INFO 12-08 09:22:13 gpu_executor.py:122] # GPU blocks: 28190, # CPU blocks: 2048
INFO 12-08 09:22:13 gpu_executor.py:126] Maximum concurrency for 131072 tokens per request: 3.44x
INFO 12-08 09:22:16 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 12-08 09:22:16 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.

```
I tried using `-vllm --enforce-eager` in the python command but apparently there is no such option for that.

I also tried setting that option manually in main.py as below:
```
   if args.vllm:
        sut = sut_cls(
            model_path=args.model_path,
            dtype=args.dtype,
            batch_size=args.batch_size,
            dataset_path=args.dataset_path,
            total_sample_count=args.total_sample_count,
            workers=args.num_workers,
            tensor_parallel_size=args.tensor_parallel_size,
            enforce_eager=True               <=======
        )
```
But I get this error:
```
  File "/mnt/users/m/inference/language/llama3.1-8b/main.py", line 173, in main
    sut = sut_cls(
          ^^^^^^^^
TypeError: SUT.__init__() got an unexpected keyword argument 'enforce_eager'
```

Hi,
so enforce eager can be changed by going into SUT_VLLM.py

there you will see the function def

```
    def load_model(self):
        log.info("Loading model...")
        self.model = LLM(
            self.model_path,
            dtype=self.dtype,
            tensor_parallel_size=self.tensor_parallel_size,
        )
        log.info("Loaded model")
```

Here you can add enforce eager separately like so

```
    def load_model(self):
        log.info("Loading model...")
        self.model = LLM(
            self.model_path,
            dtype=self.dtype,
            tensor_parallel_size=self.tensor_parallel_size,
            enforce_eager=True
        )
        log.info("Loaded model")

```
you wont need to add a command line arg this way.
Repository: mlcommons/inference
Issue: Accuracy measurement from loadgen output
Root cause: Unimplemented Feature Gap

How does a submitter using a custom integration into loadgen measure accuracy? There's a script process_accuracy.py for GNMT - is there (or is there going to be) something similar for the computer vision tasks?

There should be; It's build in the benchmark right now and I need to move it into a script.

Is this completed? @guschmue Thanks!

not yet
Repository: mlcommons/inference
Issue: Issue with benchmarking SSD-ResNet34 model.
Root cause: Validation Gap

I'm not able to get proper mAP for ssd-resnet34 model, could someone help with the script. im using pycocotools to evaluate.

DONE (t=1.28s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.001
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.001
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001


@MK-1307 If I were to guess why the scores are nearly zero, it would be down to incorrect preprocessing.

Please note that SSD-ResNet34 (aka "SSD-Large") was last accepted for submission in the v2.0 round, i.e. almost 4 years ago. It has since been replaced by RetinaNet which is itself on its last legs.
Repository: mlcommons/inference
Issue: GNMT's process_accuracy.py should skip duplicate samples
Root cause: Validation Gap

It's possible that the loadgen will repeat some samples in the multi-stream scenario for padding purposes.

@nvmbreughe : Can you update the script to skip sent_id's that have already been encountered?

Hi @briandersn ,

Would you suggest just dropping ID's that have already been seen? Or caching the answers and sending them to loadgen?

The problem for the offline scenario is that we will never be able to generate sufficient ID's as there are only 3003 unique sentences...

In the accuracy mode, the loadgen attempts to visit each sample once; only visiting a sample more than once if it needs padding for multi-stream scenario.

In performance mode, we would expect to see lots of repeat samples though.

Ideally, I think we'd drop ID's that have already been seen for the accuracy mode; otherwise, the samples scored will vary across different SUTs dependent on the samples-per-query.

For performance mode (if we do log the response data) we probably shouldn't skip anything - to verify all responses are good. There isn't an official plan to log the response in performance mode though.



issued. @nvmbreughe provided a patch. @briandersn please check other models.

@briandersn , do you think we can close this?

Yes. Closing.
Repository: mlcommons/inference
Issue: SSD-mobilenet is missing 300x300 resize
Root cause: Validation Gap

Right now if I print qitem.img.shape before the predict command, I got the following for SSD-mobilenet
image dim ssd-mobilenet (1, 500, 335, 3)
image dim ssd-mobilenet (1, 375, 500, 3)
image dim ssd-mobilenet (1, 640, 426, 3)
image dim ssd-mobilenet (1, 375, 500, 3)
image dim ssd-mobilenet (1, 640, 640, 3)
image dim ssd-mobilenet (1, 480, 640, 3)
image dim ssd-mobilenet (1, 335, 500, 3)
image dim ssd-mobilenet (1, 640, 480, 3)
image dim ssd-mobilenet (1, 480, 640, 3)

Resnet resize is correct
image dim (1, 224, 224, 3)
image dim (1, 224, 224, 3)
image dim (1, 224, 224, 3)
image dim (1, 224, 224, 3)
image dim (1, 224, 224, 3)
image dim (1, 224, 224, 3)
image dim (1, 224, 224, 3)
image dim (1, 224, 224, 3)
image dim (1, 224, 224, 3)


@guschmue  please update the resizing part.
Repository: mlcommons/inference
Issue: Error: mlperf_loadgen.TestSettings object has no attribute 'min_query_count'
Root cause: External Dependency Breakage

When I run the demo-script `python demos/py_demo_single_stream.py` then I get the error:
```
Traceback (most recent call last):
  File "demos/py_demo_single_stream.py", line 84, in <module>
    app.run(main)
  File "/home/ivan/.local/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/ivan/.local/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "demos/py_demo_single_stream.py", line 72, in main
    settings.min_query_count = 100
AttributeError: 'mlperf_loadgen.TestSettings' object has no attribute 'min_query_count'
```
The same error I get when I run script in `v0.5/classification_and_detection`:
```
$ ./run_local.sh tf ssd-mobilenet cpu

INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='coco-300', dataset_list=None, dataset_path='/home/ivan/CK-TOOLS/dataset-coco-2017-val', inputs=['image_tensor:0'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], model='/home/ivan/CK-TOOLS/model-tf-mlperf-ssd-mobilenet/ssd_mobilenet_v1_coco_2018_01_28.pb', output='/home/ivan/src/github.com/mlperf/inference/v0.5/classification_and_detection/output/ssd-mobilenet-tf-cpu/results.json', outputs=['num_detections:0', 'detection_boxes:0', 'detection_scores:0', 'detection_classes:0'], profile='ssd-mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream], threads=4, time=None)
INFO:coco:loaded 201 images, cache=0, took=0.5sec
Traceback (most recent call last):
  File "python/main.py", line 526, in <module>
    main()
  File "python/main.py", line 469, in main
    settings.min_query_count = args.queries_single
AttributeError: 'mlperf_loadgen.TestSettings' object has no attribute 'min_query_count'

```

Hi @bellycat77. Can you make sure the latest version of the loadgen is installed as a python module? I double checked the code and the bindings do have that attribute set up. I can't test it atm, but I did test both of the paths you mention before I submitted the recent changes to test settings.

Hello. I have the latest version.
```
$ git log
commit 5e900d29628d63f3f3d77eac9de459b8c2874f40 (HEAD -> master, origin/master, origin/HEAD)
Merge: 7c6635c 3111e35
Author: christ1ne <41205150+christ1ne@users.noreply.github.com>
Date:   Wed Jul 3 12:28:28 2019 -0700
......
```
The error goes away when I make
`git checkout 45ffce47e1711c4ab088ba783d5baa00c532d3b0`

Sorry, now I'd retested it again. I had decided to delete the local clone of the repository at all and create the new one...
Now the problem has gone. But there is some other problem [issue#250](https://github.com/mlperf/inference/issues/250)
Repository: mlcommons/inference
Issue: Some of demo-scripts don't work
Root cause: Documentation Deficiency

There are demos in  `loadgen/demos`. Some of the demos don't work:
```
$ python demos/py_demo_multi_stream_free.py
Traceback (most recent call last):
  File "demos/py_demo_multi_stream_free.py", line 86, in <module>
    app.run(main)
  File "/home/ivan/.local/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/ivan/.local/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "demos/py_demo_multi_stream_free.py", line 72, in main
    settings.target_latency_ns = 100000000
AttributeError: 'mlperf_loadgen.TestSettings' object has no attribute 'target_latency_ns'
```
and
```
$ python demos/py_demo_multi_stream.py
Traceback (most recent call last):
  File "demos/py_demo_multi_stream.py", line 86, in <module>
    app.run(main)
  File "/home/ivan/.local/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/ivan/.local/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "demos/py_demo_multi_stream.py", line 71, in main
    settings.target_latency_ns = 100000000
AttributeError: 'mlperf_loadgen.TestSettings' object has no attribute 'target_latency_ns'
```


Should it be [multi_stream_target_latency_ns](https://github.com/mlperf/inference/blob/73ca74067aefa2a3bd3b85adcbc6f87ea7a64c66/loadgen/test_settings.h#L104)?

@bellycat77 Can you please manually change to see if this resolves the issue?

If I change `target_latency_ns -> multi_stream_target_latency_ns` then mentioned scripts work

Will be fixed by PR #252

PR #252 landed, so we should be able to close this one. @bellycat77 , can you confirm?

Yes, It works. Thank you
Repository: mlcommons/inference
Issue: Assert for `mode==settings.mode` fails in SubmissionRun test mode
Root cause: Validation Gap

The assertion statement in line 308 of loadgen.cc (`assert mode == settings.mode`) fails in the Submission Run test mode. This assert is in the `GenerateQueries` function, which is called by `IssueQueries`. Both `IssueQueries` and `GenerateQueries` are templatized with the current test mode. When `settings.mode=TestMode::SubmissionRun`, `IssueQueries` is called twice, once with the `mode=TestMode::AccuracyOnly` and a second time with `mode=TestMode::PerformanceOnly`. Thus, it seems like the reason for this issue is that `settings.mode` remains `TestMode::SubmissionRun` but `mode` becomes `TestMode::AccuracyOnly` and `TestMode::PerformanceOnly`, which don't match. 


SubmissionRun needs to be implemented and tested in general, for which we should probably have a separate issue.

However, for this particular issue, we could just delete those asserts. @v4njur, do you happen to have the cycles to put together a PR that does that?

Hi @briandersn. Sorry, but what do you mean by "cycles"? As of now, do I have permission to contribute if I haven't signed any CLAs? 

@v4njur : I checked with @petermattson who says NVIDIA has signed the CLA at a corporate level, so you should be good to go.

Regarding "cycles": I was trying to ask if you have the time.

@briandersn: Okay thanks for checking. I will go ahead and make a PR that has the change of deleting the single assert statement then. 

@briandersn I have added a fix in #257. However, I am getting a notification that I still need to sign a CLA. Do I need to sign an individual CLA as well?

Thanks for trying! You do still need to make sure you're on your NVIDIA's
approved submitter list. Paulius (cc'ed) can likely connect you with
whoever owns the list.
Best,
Peter

On Tue, Jul 9, 2019 at 2:58 PM v4njur <notifications@github.com> wrote:

> I have added a fix in #257 <https://github.com/mlperf/inference/pull/257>.
> However, I am getting a notification that I still need to sign a CLA. Do I
> need to sign an individual CLA as well?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/255?email_source=notifications&email_token=AIIVUHP52U5KXM2J7IWDKKTP6UCYBA5CNFSM4H7INTQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZRVDLY#issuecomment-509825455>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AIIVUHK3GYNFAZHLEBIXETTP6UCYBANCNFSM4H7INTQQ>
> .
>


Ok thanks! I have resubmitted the PR in #262.

Changes have been merged. Thanks!
Repository: mlcommons/inference
Issue: [GNMT] What would be the beam width for MLPerf inference?

What would be the beam width for MLPerf inference?

It is 10 as shown in the config file. 

Yes, the default is set to 10 in config. But it can be overriden here as an argument.
https://github.com/mlperf/inference/blob/master/v0.5/translation/gnmt/tensorflow/nmt/nmt.py#L284
https://github.com/mlperf/inference/blob/master/v0.5/translation/gnmt/tensorflow/nmt/nmt.py#L377

The rule says: A reference implementation is a specific implementation of a benchmark provided by the MLPerf organization. The reference implementation is the canonical implementation of a benchmark. **All valid submissions of a benchmark must be equivalent to the reference implementation.**
Repository: mlcommons/inference
Issue: Fail to the latest version code.
Root cause: External Dependency Breakage

Hi,

I cannot run the latest version inference code. Please help me to check. Thanks.


![image](https://user-images.githubusercontent.com/19197677/61028848-49f65300-a3ec-11e9-9bc2-61facb0ed799.png)


@inspurasc This is similar to the issue we experienced a few days ago, which have been since fixed: https://github.com/mlperf/inference/issues/249

Could you please be more specific about what you mean by "the latest version" (output from `git log -1` would be helpful) and which exactly command you are trying to run?

@psyhtest  
I use the dockerfile build a image and then create a container. After enter the container, I run the test using below command  :
 CUDA_VISIBLE_DEVICES=2  python python/main.py --dataset  imagenet --dataset-path /data/dataset-imagenet-ilsvrc2012-val/  --model  /root/classification_and_detection/models/resnet50_v1.pb --output /root/classification_and_detection/gpu_output/resnet50.json --backend tensorflow --profile resnet50-tf --scenario SingleStream 
This cmd is ok on the previously version: Merge pull request #179 

And the latest version: I use the command "git log -1":
![image](https://user-images.githubusercontent.com/19197677/61094762-1adef080-a483-11e9-999c-aa5924ba9499.png)
Repository: mlcommons/inference
Issue: LoadGen Hangs in Offline Scenario with AccuracyOnly Mode 
Root cause: Algorithmic Error

When running AccuracyOnly test mode in the Offline test scenario with LoadGen, LoadGen hangs at the `GetLatenciesBlocking(expected_latencies)` call at the end of the `IssueQueries` function.  `expected_latencies` is defined as `queries_issued * settings.samples_per_query`, but in the Offline scenario in AccuracyOnly mode, there is supposed to be one query of size `settings.samples_per_query` and a remainder query that has up to `settings.samples_per_query` samples in it. Thus, the problem is that `expected_latencies` calculation is not taking into account that the remainder query in the offline scenario may have less than `settings.samples_per_query` samples in it, and it incorrectly waits for more latency responses than it will receive. 



I'm writing unit tests now and can reproduce this locally.

Thanks for pointing out the root cause. I will put together a fix soon.

#278 Should fix the issue. I also added a unit test that will make sure this doesn't regress.

@briandersn Yep your changes in #278 fix this problem. I will wait until your changes are merged into master, and then I'll close this issue. Thanks!

#278 was merged, so I am now closing this issue.
Repository: mlcommons/inference
Issue: Too Many Samples Processed in Offline Scenario + AccuracyOnly Mode if PerformanceSampleCount < settings.samples_per_query 
Root cause: Algorithmic Error

When LoadGen is run in the Offline scenario with AccuracyOnly test mode, if the `PerformanceSampleCount` reported by QSL is less than `settings.samples_per_query`, then it sends too many samples to the SUT to issue. My understanding is that in AccuracyOnly test mode, the total number of responses logged via the `QuerySamplesComplete` function should be equal to the total number of samples we wish to perform inference on, correct? 

If so, then what is happening now is that for each query it sends, LoadGen is sending `settings.samples_per_query` samples in each query, even though only `PerformanceSampleCount` samples are actually loaded into RAM. This must mean that samples that are loaded into RAM are repeated multiple times in a query, which in turn means they are issued to the SUT multiple times in a single query. 

Moreover, in AccuracyOnly mode, `IssueQueries` is called for every loadable set. So this same incorrect behavior is repeated for all loadable sets, which means the accuracy json file quickly becomes very very large compared to the total number of samples that we have. 

Considering that some of the input image sets are very large (for example, Imagenet validation image set has 50,000 images), it is usually not feasible to have `PerformanceSampleCount` be the same as `TotalSampleCount`. But it seems like unless this is the case, the Offline scenario will not actually be coalescing all samples into a single query + remainder query correct? 



I think #278 should also fix this issue. The unit test covers this as well.

@briandersn Your changes in #278 fix this problem as well. I will wait until your changes are merged into master, and then I'll close this issue. Thanks!

#278 was merged, so I am now closing this issue.
Repository: mlcommons/inference
Issue: Clarification regarding qps shown in Summary
Root cause: Algorithmic Error

As per https://github.com/mlperf/inference/blob/master/loadgen/loadgen.cc#L891 (and next few lines), the calculation for QPS without loadgen overhead is measured with latency_min. 
I have couple of clarifications regarding this.
1) Does the latency measured also include the time spent residing in loadgen ? 
2) The QPS w/o loadgen overhead is measured from the query with least latency (1 over latency_min). This assumes that all queries take the same amount of time to get serviced and that the variability is purely because of loadgen. That might not always be case with all models. (RNN models perhaps ?). Is my understanding of this assumption, correct ?



Re 1: The latency measured does not include time spent residing in loadgen.

Re 2: There's a bug with qps_wo_lg :  latency_min should be latency_mean. Sorry about that.

Actual latency report is correct given to 90th percentile for SingleStream. QPS reporting is wrong in SS. not an urgent item.

Fixed in https://github.com/mlperf/inference/commit/5ca30b662536f2fb0a1c055120ae81622381310a .