zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Update loadgen version number to 2.0
Root cause: Configuration Error

https://github.com/mlcommons/inference/blob/966e5592976a8687081335fd914fe062ab47e855/loadgen/version_generator.py#L96

@guschmue FYI

Addressed in PR 1072
https://github.com/mlcommons/inference/pull/1072
Repository: mlcommons/inference
Issue: Can anyone help explain what qps means? I haven't quite understood, thank you.


usually means queries per second
like in a search engine


On Tue, May 7, 2019 at 2:37 AM huihuima <notifications@github.com> wrote:

> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/107>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACCS4T7SZZ7HINXFYPSB3G3PUFEUNANCNFSM4HLHC5ZA>
> .
>


@David-Levinthal You are right. Thank you very much.
Repository: mlcommons/inference
Issue: Trained DLRM models are not useful for measuring accuracy
Root cause: Unimplemented Feature Gap

The trained DLRM models require the corresponding feature dictionaries if they are used to measure accuracy. These dictionaries are small, but take a long time and much memory to prepare with the data preparation scripts in their current form.
To clarify, the dictionaries should be made available along with the trained models since they are needed to translate from the categorical data values to the values needed for inference.

@diyessi Even if the dictionaries are provided, data preprocessing for the 24 days cannot be skipped for inference right?

It has been quite a while but I think with the dictionaries and a trained model you can do inference.
The restart logic in the dictionary code is broken, so if something goes wrong while building the dictionaries you either need to start over or fix the code. The code itself can also be sped up some.

Thank you @diyessi for your reply. That's exactly what I also felt. But the reference implementation itself seems broken (see [issue](https://github.com/mlcommons/inference/issues/1303) and so I could not make further progress. 

outdated
Repository: mlcommons/inference
Issue: Wrong data type for query_count in logs causing submission checker failure for singlestream/multistream power results
Root cause: Interface Contract Mismatch

This line is causing the below error in r2.0 branch for submission checker
https://github.com/mlcommons/inference/blob/r2.0/tools/submission/submission-checker.py#L1085
Error
```
 caused exception in check_power_dir: unsupported operand type(s) for /: 'float' and 'str'
```
Log entry is in str type -- probably needs to be in int type or needs a type change in the operation

```
./ranging/mlperf_log_detail.txt::::MLLOG {"key": "result_query_count", "value": "1755608"
```

I see. @pgmpablo157321  We should remove `std::to_string()` around the size_t numbers: https://github.com/mlcommons/inference/blob/r2.0/loadgen/loadgen.cc#L1021

Actually, let me file a quick fix for this.

Hi @nvpohanh, are you working on this? I have quick fix for this problem

@pgmpablo157321 Well, you beat me on the speed! :D thanks for fixing this!

Since the fix changes the log output is it required to rerun the experiments or can we use the old log entries with an "int" typecast?

I think we should patch the submission checker so that either the size_t output or the string output works.

@arjunsuresh This patch will fix the issue: https://github.com/mlcommons/inference/pull/1078/files

Thanks a lot @nvpohanh @pgmpablo157321
Repository: mlcommons/inference
Issue: Equal Issue mode only implemented for Offline
Root cause: Unimplemented Feature Gap

We have an equal issue mode implementation for Offline scenario, through PR1032 (https://github.com/mlcommons/inference/pull/1032), and this works flawlessly for 3D-UNet Offline runs.

Now for 3D-UNet SingleStream scenario, we are missing the equal issue mode support from LoadGen, and it is problematic as below:

* 3D-UNet KiTS19 input sets has 42 samples, with 15 different shapes; total voxel count of each sample ranges from 7.8 millions to 64 millions.
* First 1050 samples using RNG seed for v2.0 submission produces sequence of samples whose total voxel count to be 33,341,833,216.
* With equal issue mode, first 1050 samples give total voxel count of 34,000,076,800.
* If one produces logs satisfying the min_query_count=1024 for SingleStream, the work is about 2% less than it should be, and the performance metric (90% latency) would be optimistic as such.
* Next round, with different RNG seed, the measurement may be flipped into pessimistic side.
* Overall, unless equal issue mode is introduced, there will be opposing 'official results' round by round, even with the same system (HW&SW) running exact the same 3D-UNet SingleStream.
* Without equal issue mode, early stopping backbone (statistical model) is broken, limiting the usefulness of early stopping model.

It is important for LoadGen to support the equal issue mode, for scenarios other than Offline. 
For v2.0, we probably want to add the support to 3D-UNet SingleStream specifically, and after the submission we can make the implementation to be more generic and unified.
![Screenshot 2022-02-08 121340](https://user-images.githubusercontent.com/83969361/153050046-72bef74f-4bb1-4586-83cd-15bc539b162d.png)



@ashwin @pgmpablo157321 @psyhtest for visibility
Repository: mlcommons/inference
Issue: compile on windows have error " loadgen.cc(191): error C2039: 'iota': is not a member of  'std' 
Root cause: Validation Gap

https://github.com/mlcommons/inference/blob/0a4a8c0f942f44de2465aa5c8e8272782208d067/loadgen/loadgen.cc#L191

according to this [thread](https://stackoverflow.com/a/51135593/3079247), add  
```
#include <numeric> 
```
will fix this error

@nv-jinhosuh ^^^

should be a quick fix.

Thanks pushing it now. 

Thank  you @nv-jinhosuh for the fix.
Repository: mlcommons/inference
Issue: ssd-mobilenet 300x300 accuracy target
Root cause: Documentation Deficiency

The tensorflow entry for ssd-mobilenet 300x300 is listed as 0.23 mAP for fp32 in the supported models table here - https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection
But I see that the submission checker uses a reference value of 22 which is then multiplied by 0.99 to get the accuracy_target - 
https://github.com/mlcommons/inference/blob/master/tools/submission/submission-checker.py#L242
Looking through the documentation and the other GitHub issues, I couldn't figure out the reason for this mismatch between 22 and 23. Can someone shed some light on this?

The accuracy target is reduced to .22 mAP to allow quantized models to meet the minimum required target accuracy - 99% of .22 mAP
Repository: mlcommons/inference
Issue: MLCommons CLA link is broken

I want to make a pull request, However the [MLCommons CLA](https://github.com/mlcommons/systems/blob/main/mlcommons_cla.txt) link response by bot is broken. Where could I sign MLCommons CLA

@jenhaoyang If your organization is already a member of mlcommons and signed CLA already, please ask you company rep to add your git id to the CLA as described in step#7 of https://mlcommons.org/en/get-involved/.
Repository: mlcommons/inference
Issue: Loadgen offline mode not implement
Root cause: Unimplemented Feature Gap

Hi, I find there are four scenarios in loadgen including SingleStream, MultiStream, Sever and Offline. However, the loadgen code now just supports the first three modes while for the Offline mode it just provides a API without function implementation. Do you plan to update the Offline model in loadgen? Or it should be implemented by the submitter? 

@jgong5 @delock

Offline mode is in now. Please check.
Repository: mlcommons/inference
Issue: TEST04 should have reasonable runtime
Root cause: Algorithmic Error

Though the reasoning for TEST04 is very good the methodology of testing looks quite bad. As I understand the maximum input sample one can use is the dataset size which for ResNet50 is 50000. So on a high performing system where QPS is say 300,000 or above the test will be running for 1/6 of a second or lower which is really too low to measure any reasonable performance. Below are two results from 1.1 submission round by Nvidia where the TEST04 performance is an order of magnitude lower than the actual performance reported by TEST05. So, the only time TEST04 really passes is when the system noise is in your favor is my guess. I think we should either get rid of this test (less noise is always better :) or introduce an iteration count over the dataset size so that the test runs over a reasonable amount of time. 


- [A100 PCIe x 8 TEST04](https://github.com/mlcommons/inference_results_v1.1/blob/main/closed/NVIDIA/compliance/A100-PCIe-80GBx8_TRT/resnet50/Offline/TEST04-A/verify_performance.txt)
- [A100 PCIe x 8 TEST05](https://github.com/mlcommons/inference_results_v1.1/blob/main/closed/NVIDIA/compliance/A100-PCIe-80GBx8_TRT/resnet50/Offline/TEST05/verify_performance.txt)
- [A100 SXM x 8 TEST04] (https://github.com/mlcommons/inference_results_v1.1/blob/main/closed/NVIDIA/compliance/DGX-A100_A100-SXM-80GBx8_TRT/resnet50/Offline/TEST04-A/verify_performance.txt)
- [A100 SXM x 8 TEST05] (https://github.com/mlcommons/inference_results_v1.1/blob/main/closed/NVIDIA/compliance/DGX-A100_A100-SXM-80GBx8_TRT/resnet50/Offline/TEST05/verify_performance.txt)



@georgelyuan FYI

Fixed from r2.1
Repository: mlcommons/inference
Issue: [SSD-ResNet34] Is there any trained model for tensorflow?
Root cause: Documentation Deficiency

We are trying to opt performance of SSD-ResNet34 inference 1200x1200 on tensorflow.
We did some explore and need check the accuracy.
So is there any trained model?
And if not, will there be any in future?

resolved now. please see the latest readme.
Repository: mlcommons/inference
Issue: WG decision on Edge scenario Object detector
GitHub labels: inference v2.1 and backlog

Retinanet-ResNeXt50 with 800x800 will be the candidate for Object Detector in v2.1 for Data Center.
What would be the equivalent for Edge?  (up to v2.0, SSD-Mobilenet 300x300 for Edge and SSD-ResNet34 1200x1200 for Data Center)

Decide on next steps: 
1. Poll with option for Edge?

FuriosaAI suggests model EfficientDet for Object Detector for Edge.

Many models are optimized for NVIDIA GPUs because the current training market is dominated by NVIDIA. This is no exception for the models used in MLPerf. MLPerf therefore still struggles to attract wider participation from accelerator suppliers. A founder of a major AI research firm felt "most companies don't see enough ROI from the massive effort required to run these benchmarks". This is definitely not ideal, in view of the spirit of MLPerf, which should encourage a diverse spectrum of suppliers to participate. 

We do agree that SSD-small is an old model, with little practical application value - so we do not object to removing this model from the benchmark. Instead, in stimulating attempts towards new architectures that are different from existing NVIDIA optimized models, we suggest adding one of EfficientDet-D0/D1/D2 as an Object Detection model. Compared to RetinaNet, EfficientDet requires less computations but shows comparable accuracy levels. This should be appropriate for AI accelerators to work on.

Not relevant anymore, closing issue
Repository: mlcommons/inference
Issue: Large language model inference
GitHub labels: inference v2.1 and backlog

Discussion on large language model inference.
Training WG had a [roadmap](https://docs.google.com/document/d/12gKY2UJJKgxmE3kIQEmSoFlxYzaoSZcqITrZNj4ttgA/edit?resourcekey=0-5hJkegEr6XtOGPXUmKUjtQ#heading=h.gm13caqq38tg) and there is a discussion on the Large language model. 

Wait until the training WG make a proposal and then discuss
Repository: mlcommons/inference
Issue: Have issue using run_and_time.sh
Root cause: Environment Incompatibility

OS: Ubuntu 18.04 
Kernal: 5.13.0-30
CUDA:11.1.1
python:3.8.10
numpy:1.22.2


I had run vision/classification_and_detection/run_local.sh success,
however when I try to use the run_and_time.sh tf resnet50 gpu 
it will fail with RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd.
Is there any solution about this?

the error looks like this:
![image](https://user-images.githubusercontent.com/38207944/156308906-80d760fe-b4b7-412b-836a-ec28980e8367.png)


@georgelyuan Can you help with this issue?

@JackHsuan Could you try installing a newer numpy version with `sudo python3 -m pip install numpy==<VERSION>`?
Repository: mlcommons/inference
Issue: Spurious submission checker error and info
Root cause: Validation Gap

When a directory called `assets` was introduced into the v2.0 submission tree, the submission checker started giving this `ERROR`:
```
ERROR   invalid division in input dir assets
```

Another slightly vexing issue are `INFO` messages like:
```
INFO    Target latency: None, Latency: 654487586684, Scenario: Offline
```

Maybe we need a whitelist of top-level dir names. Alternatively, the checker seems to ignore the `.github` dir, so can rename `assets` to `.assets`?

> Another slightly vexing issue are `INFO` messages like:
> 
> ```
> INFO    Target latency: None, Latency: 654487586684, Scenario: Offline
> ```
What is vexing about this message? That there should be a value for the Target latency? That there should not compare this values for the Offline scenario? Or is it something else?


It's sort of irrelevant. I don't see messages of similar kind for other scenarios (where they might be more appropriate).

This is because this info is not being log when using early stopping (that means for the other scenarios). We can log this info for early stopping as well
Repository: mlcommons/inference
Issue: when C++ demo will update?

Hi, @jveejay 
        When C++ demo will update？ And will QSL be provided in loadgen?

BR


Hi @kubowuhao , which demo do you mean specifically? And what's QSL?

> Hi @kubowuhao , which demo do you mean specifically? And what's QSL?

Hi, @psyhtest  , I mean code in https://github.com/mlperf/inference/tree/master/loadgen/demos

QSL : Query Sample Library

I think you can run the reference code with LoadGen without issues now. Please open an issue if you have trouble using LoadGen. @kubowuhao
Repository: mlcommons/inference
Issue: about multiple GPUs on run_local.sh
Root cause: Unimplemented Feature Gap

Hi,
I would like to run the vision/classification_and_detection/run_local.sh to try inference benchmark with multiple NVIDIA GPUs,
However, I had install two GPUs but I only see the first GPU is running in the run_local.sh.
Is there any method or setting can help me to try benchmark on Multiple GPUs?

![image](https://user-images.githubusercontent.com/38207944/158762465-49fa386c-3995-464d-8aa7-ddd8a2b6da9f.png)
![image](https://user-images.githubusercontent.com/38207944/158762561-bfd1040e-2283-4eaf-96f6-f53121ceeb1c.png)

Thanks for your help!

@nvpohanh Can you help answer the question?

The reference implementation doesn't support multi-GPU.
Repository: mlcommons/inference
Issue: [FEATURE]Benchmark for AI system
Root cause: Unimplemented Feature Gap

Is it possible to have  some benchmark for end to end AI system(decoding and multiple models included)? For examples, the [face recognition system](https://github.com/deepinsight/insightface), smart tracfic system with tens of models, and [ocr-system](https://github.com/PaddlePaddle/PaddleOCR). A complete comparison  for the throughout and Latency across different  hardware platforms and  also libraries is necessary. Usability might also be  considered.  This is both beneficial to hardware manufactures,  and also serving infrastructure, for example, triton server, [Paddle Serving](https://github.com/PaddlePaddle/Serving), openvinoserver, TORCHSERVE, [MegFlow](https://github.com/MegEngine/MegFlow), AI-Express, [videoflow](https://openreview.net/pdf?id=Rq31tXaqXq) , CNStream, nnstreamer, mediapipe, and TengineInferPipe, etc.
Repository: mlcommons/inference
Issue: CK ERROR:[package] This package has a known problem
Root cause: Unimplemented Feature Gap

Hi,

I run this command : ck install package --tags=image-classification,dataset,imagenet,val,original,full
But I got this error:
![1649410495637](https://user-images.githubusercontent.com/33252913/162628274-3083df18-edd8-463e-94a5-4636ff649e8e.png)
Could you give me some solutions for the error?
Thanks

Sincerely,
Ling

Sorry @LingYu-Chu, this problem is [2.5 years old](https://github.com/mlcommons/inference/issues/542), and there is no solution.
Repository: mlcommons/inference
Issue: Criteo Dataset Host Offline
Root cause: External Dependency Breakage

Unable to download the Criteo 1TB Click Logs dataset for the DLRM benchmark because the links to get the data are down.

Specifically:
http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_9.gz (or any other day) does not work. Looking for a resolution or an alternate solution in the meantime. 

> Unable to download the Criteo 1TB Click Logs dataset for the DLRM benchmark because the links to get the data are down.
> 
> Specifically: http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_9.gz (or any other day) does not work. Looking for a resolution or an alternate solution in the meantime.

I am also facing this issue. Please provide a resolution.

I am facing the same problem. azuremlsampleexperiments.blob.core.windows.net is resolvable. CC: @nv-jinhosuh (saw you had a recent commit). Any pointer is very welcome. Thanks. 

Seems like link is not valid since the last one/two month

Criteo updated the link for the dataset (hosted on Google storage). Please go thru https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf/ to access the dataset.
Repository: mlcommons/inference
Issue: Update audit policy

Opening this for discussion of audit policy before we start crafting a PR.

My impression is that the proposal is to add to the rules the following:

* if an accelerator has been selected for audit in the previous round, it will not be chosen for random audit in this round
* a "No Audit" option will be added to the committee-selected audit, for cases where the committee does not see significant benefit to MLCommons in auditing any of the systems.

@psyhtest anything else we want to include here?

[moving this to inference_policies, where it should've been in the first place.]
Repository: mlcommons/inference
Issue: ssd-resnet34(1200x1200 pytorch) 'code result' <-> 'document' mAP difference
Root cause: Documentation Deficiency

I just downloaded param ssd-resnet34 1200x1200 pytorch at [table](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection) and ran it.

And result is like this. mAP:19.618%

![ssd-resnet34_1200_pytorch_result](https://user-images.githubusercontent.com/24352190/167070829-953adf05-b833-46c6-9b20-0cd432b7c3ac.png)

The code mAP result is different from mAP accuracy value at table(mAP:0.20) https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection

Is the table result(mAP 0.20) right?

It's been validated by many submitters, so seems right. At any rate, this benchmark has been deprecated (v2.0 was the last round where it appeared), so I would not spend time on making it run.

> It's been validated by many submitters, so seems right. At any rate, this benchmark has been deprecated (v2.0 was the last round where it appeared), so I would not spend time on making it run.

Thank you for your reply! I couldn't find any of the next round model information. Could I ask where can I find the next round model information?
Repository: mlcommons/inference
Issue: 3D-UNet dataset in the v2.0 results table is listed as BraTS

The text below the table correctly mentions KITS.

https://mlcommons.org/en/inference-datacenter-20/
https://mlcommons.org/en/inference-edge-20/

@petermattson @TheKanter Can you update the 3D Unet dataset mention with "KiTS19"?

I think it's fixed. Can you check, confirm, and then close?

Thanks David. Datacenter - Closed - Power still mentions BraTS. The rest is fine.

Check again? We should be good this time!

On Tue, Jun 7, 2022 at 12:49 AM Anton Lokhmotov ***@***.***>
wrote:

> Thanks David. Datacenter - Closed - Power still mentions BraTS. The rest
> is fine.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/mlcommons/inference/issues/1154#issuecomment-1148320111>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AJXLOK562PNTRFRXLJSN4I3VN35JRANCNFSM5XD37S2A>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


It is fixed.
Repository: mlcommons/inference
Issue:  Will Weight files be provided in demos? 
Root cause: Documentation Deficiency

 Will Weight files be provided in demos? 

Yes, see the table here: https://github.com/mlperf/inference/tree/master/cloud/image_classification.
If you can't use onnx, tensorflow or pytorch you can easily get the raw weights from the onnx file. 

There is some simple tool that does something similar here https://github.com/onnx/tensorflow-onnx/blob/master/tools/quantitize_weights.py#L93 and line 93 is where it has the weights in hand and makes it numpy. You could take that code, remove most of it and convert it to what is needed.

I think the issue has been addressed. Please re-open the issue if needed @kubowuhao
Repository: mlcommons/inference
Issue: Two different pytorch models/weights for Retinanet
Root cause: Documentation Deficiency

In the table of models, there seems to be one pytorch model derived from the weights of another. Ref here -

https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection

The two variants->

1) https://zenodo.org/record/6605272
2) https://zenodo.org/record/6617981

Is this as designed?

Can either of the Pytorch variants be used as the base for 2.1 submission?

Thanks.

Both can be used for v2.1 submission.
Repository: mlcommons/inference
Issue: Open Images download scripts should use python3
Root cause: Environment Incompatibility

It appears that [`fiftyone`](https://pypi.org/project/fiftyone/) only works under Python 3. Therefore, one installs as e.g. `python3 -m pip install fiftyone`. Unfortunately, on some machines the default version of Python is still Python 2, so the [following code](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/tools/openimages_calibration_mlperf.sh#L53) in Open Imagees downloaders fails:
```
python openimages_calibration.py \
    --dataset-dir=${DATASET_PATH} \
    --output-labels="openimages-mlperf.json" \
    --classes "${MLPERF_CLASSES[@]}"
```
An easy fix is to replace this with:
```
python3 openimages_calibration.py \
    --dataset-dir=${DATASET_PATH} \
    --output-labels="openimages-mlperf.json" \
    --classes "${MLPERF_CLASSES[@]}"
```
(Or to parameterise the Python interpreter via an environment variable defaulting to `python3`.)

Thank you for fixing this quickly @pgmpablo157321 !
Repository: mlcommons/inference
Issue: How can I obtain the labels.txt?
Root cause: Documentation Deficiency

In run.sh of shufflenet network, it's described as:
```shell
# The downloaded images should be in the following directory structure
# ${IMAGENET_DIR}/labels.txt
# ${IMAGENET_DIR}/val/n*
```
 I cannot find the labels.txt. Although, I find a seem-like text file named ILSVRC2012_validation_ground_truth.txt, but the format is not as requested in imagenet_test_map.py. 
 Can you provide the path to downloads labels.txt?

@sf-wind 

@newstzpz , how did you get the labels.txt on the imagenet dataset?

@sf-wind  how can you run successfully shufflenet without labels.txt?

Thanks. I checked internally, the labels.txt is actually generated in house. That file is added to the FAI-PEP directory. Please patch https://github.com/mlperf/inference/pull/125 and run it again.  

Thank for your reply! Excuse me, which version of FAI-PEP did you use when you run the script of run.sh?

Use master. do you see any issue?

Is this still a problem @junna2016?

close due to lack of activity