zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: MLPerf build problem
Root cause: Validation Gap

It seems that when cloning the latest MLPerf version the GN is not cloned from GN git DB and those fails the mlperf build.

I ran into the same problem. And I cloned GN manually to third_party folder, but there was still build failure below with 'make mlperf_loadgen'.

third_party/gn/gn gen out/MakefileGnProj
ERROR at //loadgen/demos/BUILD.gn:6:15: Only source, header, and object files belong in the sources of a source_set. //loadgen/demos/py_demo_single_stream.py is not one of the valid types.
              "py_demo_single_stream.py" ]
              ^-------------------------
See //BUILD.gn:7:5: which caused the file to be included.
    "loadgen/demos:loadgen_demos_python",
    ^-----------------------------------
make: *** [Makefile:22: mlperf_loadgen] Error 1
Repository: mlcommons/inference
Issue: Can not run inference benchmark as Docker container
Root cause: External Dependency Breakage


While using Docker method to run inference benchmark using the following command

`./run_and_time.sh tf resnet50 cpu`

The error message shows up.

> 
> Traceback (most recent call last):
>   File "python/main.py", line 537, in <module>
>     main()
>   File "python/main.py", line 430, in main
>     model = backend.load(args.model, inputs=args.inputs, outputs=args.outputs)
>   File "/mlperf/python/backend_tf.py", line 41, in load
>     self.sess = tf.Session(graph=g)
> **AttributeError: module 'tensorflow' has no attribute 'Session'**

I have tried 
(1) uninstall tensorflow 1.14.0 or install tensorflow 1.14.0 again.
(2) rebuild docker image
After these steps, still stick to the same error.

After assign tensorflow version in Dockerfile.cpu, the issue is solved.

original:
RUN conda install pytorch-cpu torchvision-cpu -c pytorch
RUN pip install future pillow onnx opencv-python-headless tensorflow onnxruntime
RUN pip install Cython && pip install pycocotools

modified:
RUN conda install pytorch-cpu torchvision-cpu -c pytorch
RUN pip install tensorflow==1.14.0
RUN pip install future pillow onnx opencv-python-headless onnxruntime


I will send a PR for with the change.
Repository: mlcommons/inference
Issue: LoadGen Issues to address after v0.5

Tracking things in a document for now so everyone can add and edit issues.
https://docs.google.com/document/d/1QdRgS-fdAK1UIiyYa1FaJ5Xn9yHR-pX7YpeukjAit7w/edit?usp=sharing


Related issues: #495 #493 

#318  as well?
Repository: mlcommons/inference
Issue: Backbone file for SSD-Resnet34 training
Root cause: Documentation Deficiency

The training script for SSD-Resnet34 train_ssd_large.py (line 97) at
https://github.com/lji72/inference/tree/tf_ssd_resent34_align_onnx/others/cloud/single_stage_detector/tensorflow/train
asks for path to checkpoint to fine-tune from. @lji72 could you please confirm which checkpoint file is being used for that and a link to that will be helpful.

Thanks

 @Shama022  https://zenodo.org/record/3712664#.XoGvkogzY2x
Repository: mlcommons/inference
Issue: Broken link to resnet50-to-onnx.sh
Root cause: Documentation Deficiency

The resnet50 entry in the [supported-models](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection#supported-models) table has a link to [resnet50-to-onnx.sh](https://github.com/mlperf/inference/blob/master/cloud/image_classification/tools/resnet50-to-onnx.sh) which appears to be broken...

I think broken links are a common issue and a radical solution to this problem is automatically checking and reporting them. Luckily the new GitHub actions and  workflows help with that. I recently wrote an action that just does that. [URLs-checker](https://github.com/marketplace/actions/urls-checker) will check and report broken links, which is useful to spot similar issues to this one.
Repository: mlcommons/inference
Issue: RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version 
Root cause: Environment Incompatibility

Hi all, it looks like the sample inference/cloud/single_stage_detector was changed, I ran it before without issues but now it is throwing the below error :

```
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=34 error=35 : CUDA driver version is insufficient for CUDA runtime version
Traceback (most recent call last):
  File "infer.py", line 156, in <module>
Using seed = 1
    main()
  File "infer.py", line 151, in main
    torch.cuda.set_device(args.device)
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 264, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at /pytorch/torch/csrc/cuda/Module.cpp:34

real    0m0.765s
user    0m1.120s
sys     0m2.000s
```


I used the previous code to run it
Repository: mlcommons/inference
Issue: [classification_and_detection] How is accuracy evaluated in the inference benchmark?
Root cause: Documentation Deficiency

How is accuracy evaluated in the inference benchmark? I am currently running the benchmark for resnet50v1.5 onnx model on gpu for fake imagenet data.

We use scripts to calculate accuracy from LoadGen output. The official reference code already includes such calculation. How are you running the model? 

It will be very helpful if I am pointed to the code that evaluates the accuracy, so I can understand how predicted labels and scores are being evaluated.

I am running the benchmark for resnet50-v1.5 onnx model, using fake_imagenet data, with the dockerized workflow:

$ ./run_and_time.sh onnxruntime resnet50 gpu --accuracy



[accuracy-imagenet.py](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py) and [accuracy-coco.py](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-coco.py) are for parsing LoadGen output for image classification and object detection, respectively.
Repository: mlcommons/inference
Issue: ImageNet 2012 validation set is no longer available
Root cause: Environment Incompatibility

Originally reported [here](https://groups.google.com/forum/m/#!topic/mlperf/3I0bh20uCNA).

Resorting to [academic torrents](http://academictorrents.com/browse.php?search=imagenet) to download this dataset would be akin to sourcing rare goods from a black market (similar to this ["Looking for Transistors"](https://www.youtube.com/watch?v=c4pCeEBNiUA) scene from a popular Soviet movie).

A related [PyTorch issue](https://github.com/pytorch/vision/issues/1453).

For now I added an error message in our CK-related packages (https://github.com/ctuning/ck-mlperf/issues/43#issuecomment-580726772) . If someone finds a solution to get a public access to ImageNet, I will update our CK packages ... 

will try to resolve in v0.8. Please email the mailing list if you are blocked by this issue.

Still an issue.

Yep this is still an issue !

And it will forever be...

> Yep this is still an issue !

@vineetgarc - you can download ImageNet validation set via Academic Torrents. Please check this [Readme](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-imagenet-val). 

Feel free to join [this workgroup](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md) where we discuss these issues...


@gfursin The readme link is now moved to [here](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/get-dataset-imagenet-val/README-extra.md)

Thank you @arjunsuresh for the update!
Repository: mlcommons/inference
Issue: [classification_and_detection] Where can we find details on post-processing for inference for imagenet data with resnet50v1.5 model?
Root cause: Documentation Deficiency

Where can we find code and details on post-processing for inference for imagenet data with resnet50v1.5 model?

@ahsabali what's wrong with my answer [here](https://github.com/mlperf/inference/issues/541#issuecomment-559069670)? :)) 

@psyhtest thank you for answering the PRE-processing question, this new question is for POST-processing. 

My bad, shouldn't have rushed answering from a phone. How about [accuracy-imagenet](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py)? It's a standalone piece of code but I think an equivalent piece of code is somewhere inside the reference app too.

To find the post processing for resnet look at the profile for resnet:
https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/main.py#L78
It says it uses a dataset 'imagenet'  which is defined here:
https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/main.py#L38
And that definition says we use PostProcessCommon which is defined here:
https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L81

The output of this is handed back to loadgen and written into a json file. As @psyhtest said, this tool https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py reads the json and comes up with the official accuracy.

There is some code path in the reference app that also calculate the accuracy but its a left over from before loadgen and it will most likely be removed soon.


I need some help in understanding this code.

Could you explain:

1. What exactly is this [PostProcessCommon](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L81) code doing?

2. What exactly is this [accuracy-imagent.py](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py) code doing?
Repository: mlcommons/inference
Issue: [classification_and_detection] ./run_local.sh: command not found
Root cause: Environment Incompatibility

I use [sudo ./run_local.sh tflite mobilenet gpu] will show [sudo: ./run_local.sh: command not found]

use model: mobilenet_v1_1.0_224_quant_edgetpu.tflite
use dataframe: dataset-imagenet-ilsvrc2012-val-min (500 photos)
Use hardware: coral Dev Board

How to run[Run local]

@LINYOUWEI0804 I seriously doubt it will be possible to run the official reference code on Coral. I believe it has it's own runtime, which is not integrated into the reference code.

Having said that, I assume you are trying to run `run_local.sh` from in the same directory where it is located?

Hello, I am executing in the classification_and_detection directory

Try to specify the full absolute path.
Repository: mlcommons/inference
Issue: [classification_and_detection] Predicted labels have a constant offset of 2 compared to actual labels
Root cause: Validation Gap

When I run the inference benchmark for onnnx resnet50_v1.5 using fake imagenet data set, via dockerized workflow, I get 75% accuracy, i.e., 6 out of 8 images are predicted correctly. However, when I write a python script to predict labels for each of the image, using the pre-processing code used in mlperf inference code and the onnx resnet50_v1.5, in my code the predicted labels of 6 out of 8 images (75%) have a constant offset of 2 compared to actual labels, i.e., actual_label = predicted_label - 2. Should I subtract 2 from all predicted labels? If yes, any ideas why?
Repository: mlcommons/inference
Issue: Improve accuracy of documentation on TestSettings.offline_expected_qps

https://github.com/mlperf/inference/blob/876c6e2e390b188d69675a59a71360ab39007bde/loadgen/test_settings.h#L219

Perhaps a minor issue, but it confuses me: Doesn't loadgen generates 10% more _samples_ , not _queries_ (which doesn't seem to make sense in the offline mode), than it thinks it needs to meet the minimum test duration?


On that thought, perhaps the `offline_expected_qps` should be renamed as `offline_expected_throughput`, since it's really specify expected no. _samples_ per second, not _queries_ per second?

Another similar "typo" seems also on the following line:
https://github.com/mlperf/inference/blob/876c6e2e390b188d69675a59a71360ab39007bde/loadgen/test_settings.h#L274

Am I understanding it correctly that specifying `performance_issue_unique = true` shall cause QSL to load unique _samples_ , not just unique _queries_ ? In principle, 2 queries with overlapping but not the same sets of samples are still unique to each other. From how loadgen.c uses this flag, it seems in fact unique samples will be used?
Repository: mlcommons/inference
Issue: [classification_and_detection] Why is there an offset of '-1' in post processing for imagenet dataset?
Root cause: Documentation Deficiency

Why is there an offset of '-1' in post processing for imagenet dataset here:
[https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/main.py#L38](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/main.py#L38)

that is how the models are trained - class 1 is item 0 in the label list. 

The prediction output by the model is the index of max score, e.g., 188, according to your explanation, the offset should be +1 and class label should be 188+1=189, however, this is not the case, the offset is -1 and the class label is evaluated as 188-1=187. Any ideas why this is the case?

For example: 
```ILSVRC2012_val_00000001.JPEG```

In val_map.txt is has
```val/ILSVRC2012_val_00000001.JPEG  65```

65 is a sea snake:
https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a#file-imagenet1000_clsidx_to_labels-txt-L66
Note that the labels start at 0.

If you run this against the mlperf resnet50 reference model the prediction is 66 because resnet50 is trained with 0 being reserved.
Repository: mlcommons/inference
Issue: Unnecessary and confusing type alias used for sampe offset

https://github.com/mlperf/inference/blob/876c6e2e390b188d69675a59a71360ab39007bde/loadgen/loadgen.cc#L366

https://github.com/mlperf/inference/blob/876c6e2e390b188d69675a59a71360ab39007bde/loadgen/loadgen.cc#L370

In both lines above, the type of `same_sample` and `sample_i` should both be simply `uint64_t` (same as `TestSettings.performance_issue_same_index`), instead of the user-defined alias `QuerySampleIndex`.

But perhaps the type of `TestSettings.performance_issue_same_index` should be `size_t` to begin with, given that it's used as an index to `std::vector`. Even so, it would be unnecessary and confusing to use `QuerySampleIndex` for the type of the offset that points to the `QuerySampleIndex` in a vector.

Similarly, the type for `TestSettings.performance_sample_count_override` right after should probably also be `size_t`, which is the type of `QuerySampleLibrary.PerformanceSampleCount()`.
Repository: mlcommons/inference
Issue: Will there be a C++/C version of this?

Seems the MLPerf app is using C++, so if you already have a C++ version of MLPerf inference, Will it go open source? Not an issue, just a question, Many thanks.

sorry guys, I just saw the cc files, will check it out.

@Charleshzhang You could also study the [TFLite program](https://github.com/ctuning/ck-mlperf/tree/master/program/image-classification-tflite-loadgen) [we](http://dividiti.com) used to get over a hundred results for our v0.5 submission.
Repository: mlcommons/inference
Issue: I want to know 'Test Parameters Used' in MLPerf Results Summary

![image](https://user-images.githubusercontent.com/55476509/71069743-a7bd7100-21bc-11ea-980b-c115061d6fbc.png)

1)max_async_queries
2)min_query_count
max_q00uery_count
3)qsl_rng_seed
4)sample_index_rng_seed
5)schedule_rng_seed
6)Min duration
Max duration

I want to know what those parameters  stand for?
Repository: mlcommons/inference
Issue: [classification_and_detection] Why AREA interpolation is used for resizing during preprocessing for imagenet data for inference?
Root cause: Documentation Deficiency

Why AREA interpolation is used for resizing during preprocessing for imagenet data for inference, in [this code](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L172).

Whereas, BILINEAR interpolation is used during training, in [this code](https://github.com/mlperf/training/blob/master/image_classification/tensorflow/official/resnet/imagenet_preprocessing.py#L250).

There is some discussion about this here: 
https://github.com/mlperf/inference/issues/201

If I remember this correctly the tensorflow implementation is different that bilinear in pil or opencv which showed in us not meeting the published accuracy. With the current reference implementation accuracy is exactly what it is suppose to be.


AREA is only used for ResNet. You can use BILINEAR for ResNet too with a small loss of accuracy.
Repository: mlcommons/inference
Issue: Can update the TF ssd-mobilenet PB file
Root cause: Unimplemented Feature Gap

Hi,

The TF ssd-mobilenet PB file in MLPerf used the PB file in tensorflow/models, but this file this too old, the postprocessing part use a large number of operation to do the multi-batch multi-class NMS. Recently Google had release a new core operator named CombinedNonMaxSuppression and tensorflow/model has add this implement in 
https://github.com/tensorflow/models/blob/master/research/object_detection/core/post_processing.py#L981

I try to re-generate the PB using the original checkpoint and the new model implement, get the new PB with CombinedNonMaxSuppression, this PB improvement the FP32 accuracy from 23.1mAP to 26.2mAP.

Can community consider to update this PB file? 


@guschmue  any feedback?

I do not think we want to change the reference model.

I agree. It would be nice to get better accuracy, but the whole reason for keeping this model for v0.7 seems to allow comparisons between v0.5 and v0.7 data.

It's an intriguing idea for the Open division though...

We will update the models per the vision advisory board output.
Repository: mlcommons/inference
Issue: Random loadgen logging issues on windows: Empty mlperf_log_summary.txt after run
Root cause: Resource Mishandling

In majority of runs on windows the logs are not completely written.
The "mlperf_log_summary.txt" is mostly empty, and in those cases, the "mlperf_log_detail.txt" appears incomplete: The performance time usually found at the end do not get logged.

Another related issue is that in Offline scenario, even after completion, the "mlperf_log_summar.txt" reports the wrong 'Samples per second' (Inf). I printed out the [max_latency used here](https://github.com/mlperf/inference/blob/master/loadgen/loadgen.cc#L1038) and saw that it's **zero** even though additional stats in the log_summary indicates non-zero latencies.

Example log_summary (and associated log_detail) is attached
[mlperf_log_summary.txt](https://github.com/mlperf/inference/files/4215885/mlperf_log_summary.txt)

[mlperf_log_detail.txt](https://github.com/mlperf/inference/files/4215899/mlperf_log_detail.txt)

**ps**: loadgen and backend are built with visual studio 2017

closing this issue due to lack of activity. please re-open if needed.
Repository: mlcommons/inference
Issue: I want to know metric of single stream in v0.5 results 
Root cause: Documentation Deficiency


![image](https://user-images.githubusercontent.com/55476509/75311373-e432da80-5899-11ea-94ee-db6fdb1d10ee.png)
In log summary, I can see that it is nano second of 90 percentile latency.

![image](https://user-images.githubusercontent.com/55476509/75311435-147a7900-589a-11ea-9fee-07b2da5257ba.png)

Screen shot is closed Division performance report.


Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz

single stream value is 3.85.

!)I wonder it is 3.85 second or nano seconds~


Thank you!
Kwan




Hi Kwan, LoadGen measures **nano**seconds. The table converts this to **milli**seconds for readibility.

You are actually referring here to SSD-ResNet on Habana's Goya. Here is the LoadGen log:
https://github.com/mlperf/inference_results_v0.5/blob/master/closed/Habana/results/Goya_1/ssd-large/SingleStream/performance/run_1/mlperf_log_summary.txt#L7

It's actually 3.86 ms, not 3.85 ms - it's an artifact of manual results compilation. If you go to http://cknowledge.org/dashboard/mlperf.inference, select "MLPerf Inference v0.5 - Object Detection - SingleStream (milliseconds per sample) from the dropdown menu and then filter the results by submitter "Habana" from the settings panel, you should see:
![Screenshot from 2020-02-26 09-29-29](https://user-images.githubusercontent.com/6597818/75331438-9bb7f480-587a-11ea-8f66-a102fd43a4c7.png)

```
#16
Division: Closed
Submitter: Habana
System: HL-102-Goya PCI-board
Processor: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Processor #: 2
Accelerator: Habana Goya
Accelerator #: 1
Benchmark: ssd-large (1200x1200)
Object Detection performance: 3.86
Object Detection accuracy (mAP, %): 19.810
```

> Hi Kwan, LoadGen measures **nano**seconds. The table converts this to **milli**seconds for readibility.
> 
> You are actually referring here to SSD-ResNet on Habana's Goya. Here is the LoadGen log:
> https://github.com/mlperf/inference_results_v0.5/blob/master/closed/Habana/results/Goya_1/ssd-large/SingleStream/performance/run_1/mlperf_log_summary.txt#L7
> 
> It's actually 3.86 ms, not 3.85 ms - it's an artifact of manual results compilation. If you go to http://cknowledge.org/dashboard/mlperf.inference, select "MLPerf Inference v0.5 - Object Detection - SingleStream (milliseconds per sample) from the dropdown menu and then filter the results by submitter "Habana" from the settings panel, you should see:
> ![Screenshot from 2020-02-26 09-29-29](https://user-images.githubusercontent.com/6597818/75331438-9bb7f480-587a-11ea-8f66-a102fd43a4c7.png)
> 
> ```
> #16
> Division: Closed
> Submitter: Habana
> System: HL-102-Goya PCI-board
> Processor: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
> Processor #: 2
> Accelerator: Habana Goya
> Accelerator #: 1
> Benchmark: ssd-large (1200x1200)
> Object Detection performance: 3.86
> Object Detection accuracy (mAP, %): 19.810
> ```

Oh! Thank you!

Thank you! Anton!

@guschmue Please close this one.