zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: ERROR : Loadgen built with uncommitted changes!
Root cause: Validation Gap

In version 0.5,  this error was included in the "ignore_error" list  inside tools/submission/submission-checker.py .  However; in ver0.7;  this error is *not* part of "ignore_error" list.  We get this error when running the compliance tests. 

I wanted to ask was the omission  from  "ignore_error"  list by design? 

I can off course get around the error by doing a "git commit"  and  then building loadgen. 

@jaydevshelat Did you modify any codes after you `git clone ...` the repo? The LoadGen checks if you modifies anything, which is to ensure that everyone uses the same LoadGen for fairness. Doing `git commit` is not good since you will be in the incorrect commit hash.

Could you try re-cloning the repo and build LoadGen **without** modifying anything in the repo? Or if you have to modify non-LoadGen part, please clone a specifically repo specifically for building LoadGen. Thanks!

Hi,  there is no modification to the loadgen.  However, there is modification made to shell and python files (and couple extra shell scripts added) in classification_and_detection folder.  Should loadgen complaint even when that directory is touched?

please elaborate on "please clone a specifically repo specifically for building LoadGen"  ..... do you mean clone the repo twice -- one for just building loadgen and another one to modify the python / shell files? 

thank you in advance for responding!

loadgen takes that info during its own build.
It would not care about things in the app/python directory.


@guschmue IIRC, the LoadGen builder script did check `git status` and make sure there is no uncommitted change in the repo, not just the `loadgen/` directory. https://github.com/mlperf/inference/blob/master/loadgen/version_generator.py#L45 Or did I misunderstand this line?

yeah, I think you are correct. I never hit this - guess I build loadgen once directly after clone.


@jaydevshelat  please re-open if needed
Repository: mlcommons/inference
Issue: SSD-MobileNet just support batch_size=1 now.
Root cause: Algorithmic Error

When we enable the pytorch SSD-MobileNet defined in "**inference/edge/object_detection/ssd_mobilenet/pytorch/ssd_mobilenet_v1.py**", we find out it just supports the batch_size=1 now.

In "ssd_mobilenet_v1.py" line153 (box_regression = box_regression.squeeze(0)), the batch_size information is removed by squeeze(0), and the successive "decode_boxes" function is performed on box_regression without batch_size information. Finally, line 167 (boxes = boxes[None]) expand the dims with batch_size=1.

If we want to get throughput, we need to revise the code to enable this model work well for batch_size>1, right?

![image](https://user-images.githubusercontent.com/35086899/56102710-4c6a1d80-5f61-11e9-8b09-22d404dc1c56.png)


@jgong5

@delock

@fmassa 

Do we want to add support for batch mode? From my understanding this was just supposed to be used during inference, in which case a batch size of 1 is generally standard.

If batch > 1 is truly required, it can be done in several ways (the simplest one being a for loop on the batch dimension), but this can be far from being optimal.

@fmassa IIRC, the input image is fixed at 1200x1200 so the batched inference should work fine? How about we support batched inference with the simplest algorithm? At least, the backbone can be run with batches. Submitters should be able to optimize the remaining parts...

@jgong5 I can work on adding some basic batch support to the PyTorch version of MobileNet SSD for inference. But that would be after mid May, so if anyone wants to pick it up first it would probably be quicker.

@jgong5 The input size is 300x300 for SSD-MobileNet. It is 1200x1200 for SSD-ResNet. 

@wuhuikx can you check the lates code/model at https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection? Is the issue still present? Thanks!

please check otherwise I will close this issue next week.

This issue has been fixed. Thanks.
Repository: mlcommons/inference
Issue: Link for pytorch/onnx 8 bit mobilenet ssd model points to tensorflow model
Root cause: Documentation Deficiency


The link in the table entry for the pytorch/onnx 8 bit version of ssd-mobilnet:

![image](https://user-images.githubusercontent.com/55827703/93826548-51360500-fc1c-11ea-8831-d434bf89bf45.png)

on this page:

https://github.com/mlperf/inference/tree/master/vision/classification_and_detection 

appears to point to the same tarball as the tensorflow link and contains no ONNX nor PyTorch models:

https://zenodo.org/record/3252084/files/mobilenet_v1_ssd_8bit_finetuned.tar.gz

```
$ tar tf mobilenet_v1_ssd_8bit_finetuned.tar.gz 
mobilenet_v1_ssd_finetuned/
mobilenet_v1_ssd_finetuned/checkpoint
mobilenet_v1_ssd_finetuned/model.ckpt-97016.meta
mobilenet_v1_ssd_finetuned/pipeline.config
mobilenet_v1_ssd_finetuned/graph.pb
mobilenet_v1_ssd_finetuned/model.ckpt-97016.data-00000-of-00001
mobilenet_v1_ssd_finetuned/model.ckpt-97016.index
```




@guschmue  do we need to clean up this?

hm, let me check. 

Any update on this issue? It still points to tensorflow model.
Repository: mlcommons/inference
Issue: DLRM inference Pytorch python scripts are broken
Root cause: Documentation Deficiency

I am tryin to run MLPerf DLRM CPU Inference. The Python Script criteo.py is trying to import modules that no longer exist. 
import dlrm_data_pytorch as dp
import data_loader_terabyte

None of the above modules are present. I could replace dlrm_data_pytorch with dataset.

What is new file for data_loader_terabyte? 

Has anyone tested running inference post moving dir structures?

Regards
Nitin

These are both in https://github.com/facebookresearch/dlrm/tree/master available as a module in https://github.com/mlperf/training/tree/master/recommendation.

It gets imported when you git clone recursively and
export DLRM_DIR=$HOME/mlperf/training/recommendation/dlrm as per https://github.com/mlperf/inference/tree/master/recommendation/dlrm/pytorch#prepare-the-code-and-dataset.

Can I close this @Nitinmlp?


Are these scripts fixed now? Has any one tested?
________________________________
From: christ1ne <notifications@github.com>
Sent: Tuesday, November 3, 2020 12:20 AM
To: mlperf/inference <inference@noreply.github.com>
Cc: Nitinmlp <nitinin@outlook.com>; Mention <mention@noreply.github.com>
Subject: Re: [mlperf/inference] DLRM inference Pytorch python scripts are broken (#749)


Can I close this @Nitinmlp<https://github.com/Nitinmlp>?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/mlperf/inference/issues/749#issuecomment-720658613>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARERJPIN4B57SPTF66QXBLDSN35QVANCNFSM4RZ2AAXQ>.


I have, they work fine, you need to recursively clone both training and inference repos, the data_loader_terabyte module is inside the training repo.

Ok. Please close the ticket. I will try later few wels down the line

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: Cesar Avalos <notifications@github.com>
Sent: Friday, November 6, 2020 4:31:09 AM
To: mlperf/inference <inference@noreply.github.com>
Cc: Nitinmlp <nitinin@outlook.com>; Mention <mention@noreply.github.com>
Subject: Re: [mlperf/inference] DLRM inference Pytorch python scripts are broken (#749)


I have, they work fine, you need to recursively clone both training and inference repos, the data_loader_terabyte module is inside the training repo.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/mlperf/inference/issues/749#issuecomment-722694364>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARERJPI5AQHFDYQTO3FLBG3SOMVDLANCNFSM4RZ2AAXQ>.
Repository: mlcommons/inference
Issue: Server performance metric: completed vs. scheduled samples per second
Root cause: Documentation Deficiency

For the Server scenario, we have observed a slight difference between the current performance metric ("Scheduled samples per second") and the actual performance metric ("Completed samples per second" reported under "Additional Stats") e.g.:

```
================================================
MLPerf Results Summary
================================================
SUT name : SUT
Scenario : Server
Mode     : Performance
Scheduled samples per second : 1497.65
Result is : VALID
  Performance constraints satisfied : Yes
  Min duration satisfied : Yes
  Min queries satisfied : Yes

================================================
Additional Stats
================================================
Completed samples per second    : 1497.60
```

The difference seems to become more pronounced on systems with higher target QPS.

This issue also affects compliance testing (which is how we discovered it in the first place).

I would suggest to swap the two metrics.

@georgelyuan  please take a look.

WG: the proposal is to use "Completed samples per second" for reporting the server performance, rather than the "Scheduled samples per second". Anton will provide more data points. 

WG: we will keep the  "Scheduled samples per second" as the official metric. It is by design that "Completed samples per second" will be slightly different.
Repository: mlcommons/inference
Issue: Dead link to Readme on the datasets
Root cause: Documentation Deficiency

The link to the readme  in Step 2 gives me a 404.
URL: https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection#datasets

Last line here:

```
Step 2 - download the dataset. For this tutorial we create a small, fake dataset that pretends to be imagenet.¶

Normally you'd need to download imagenet2012/valiation for image classification or coco2017/valiation for object detections.

Links and instructions how to download the datasets can be found in the README
```


can you check the latest mater https://github.com/mlperf/inference/tree/master/vision/classification_and_detection?

please re-open if needed.
Repository: mlcommons/inference
Issue: total number of inference benchmarks allowed

WG: We target maximum 8 benchmarks for datacenter inference, 8 benchmarks for edge inference and 8 for mobile. 

All benchmarks should match the training models unless there is a compelling reason not to do so. 

updated the roadmap doc accordingly. https://docs.google.com/document/d/15SJJqVBuLLC6k8Tks1TiyPWFwEGCzFmBhgJPU8ph_7s/edit
Repository: mlcommons/inference
Issue: PyTorch ssd_resnet34 has lower accuracy on coco dataset
Root cause: Environment Incompatibility

I am using the source code [single_stage_detector/pytorch](https://github.com/mlperf/inference/tree/master/others/cloud/single_stage_detector/pytorch)  to test PyTorch ssd_resnet34 accuracy on CPU device, but get a lower accuracy, the mAP is **0.09001**, it is too bad compared the target.
```
Accumulating evaluation results...
DONE (t=10.31s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.090
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.181
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.075
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.015
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.078
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.118
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.115
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.179
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.189
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.024
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.126
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.248
Current AP: 0.09001 AP goal: 0.20000
```
How to produce this result(https://github.com/mlperf/inference/tree/master/others/cloud/single_stage_detector#2-directions)?

1. ```cd others/cloud/single_stage_detector && bash download_dataset.sh && bash download_model.sh```

2.  run [run_and_time.sh](https://github.com/mlperf/inference/blob/master/others/cloud/single_stage_detector/pytorch/run_and_time.sh) script.


Pytorch's version: 1.7.0.
Repository: mlcommons/inference
Issue: Remove Python2 dependencies from LoadGen
Root cause: Unimplemented Feature Gap

Remove Python2 dependencies from LoadGen

Currently, LoadGen still depends on Python2 AFAIK. We should remove these dependencies and use Python3 only.

WG: sounds good. please raise a PR. @nvpohanh 

Will do. ETA by late November.

Is this done @nvpohanh ?

No this is not. Let's push this to v1.1

@christ1ne Please change the labels back if WG thinks this is necessary in v1.0

It is fine to be in v1.1
Repository: mlcommons/inference
Issue: Accuracy Check for DLRM
Root cause: External Dependency Breakage

I am trying to check the accuracy for a convert pytorch to onnx model; however, I receive the following issue:

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from /home/ubuntu/dlrm/dlrm_terabyte.onnxruntime failed:Node (Concat_740) Op (Concat) [ShapeInferenceError] All inputs to Concat must have same rank

The command used:
/run_local.sh onnxruntime dlrm terabyte cpu --scenario Offline --max-ind-range=10000000 --data-sub-sample-rate=0.875 --samples-to-aggregate-quantile-file=./tools/dist_quantile.txt --max-batchsize=2048 --samples-per-query-offline=1 --accuracy 2>&1 | tee temp_debugging.log

Any suggestions on how to resolve ?


onnxruntime was working around the time of submission. @mnaumovfb do you know the latest status on the ONNX models?

There was a recent change in the way the model is saved in ONNX and loaded by the MLPerf DLRM runtime. This change  enabled the use of variable batch size during inference, which was an important requirement for MLPerf. In this change the input data was loaded as tensors, rather than list of tensors. It also required the latest versions of PyTorch and ONNX to work.

Please make sure that
1. You have the latest version of PyTorch and ONNX
2. You save and load the model with the same version of the code (ONNX models saved with prior versions of the code are not compatible with the latest change).

Let us know if you still see the above errors after you have done these suggestions.

please re-open if needed @jemitche1 

I meet the same problem.  I have tried torch1.7.1 , onnx 11  and torch1.7.1 , onnx 12 , but the two environment are all not ok and appeared the same problem.
Repository: mlcommons/inference
Issue: load_gen build failed.
Root cause: Environment Incompatibility

root@ubuntu1:/home/data/mlperf_inference/mlperf_inference# make mlperf_loadgen
cd third_party/ninja && python configure.py --bootstrap
bootstrapping ninja...
wrote build.ninja.
bootstrap complete.  rebuilding...
[25/25] LINK ninja
\# Generate gn's ninja build file
python third_party/gn/build/gen.py
\# Build gn
third_party/ninja/ninja -C third_party/gn/out
ninja: Entering directory `third_party/gn/out'
ninja: no work to do.
\# Copy gn to third_party/gn, where depot_tools expects it.
cp third_party/gn/out/gn* third_party/gn/.
third_party/gn/gn gen out/MakefileGnProj
ERROR at //build/config/linux/pkg_config.gni:103:17: Script returned non-zero exit code.
    pkgresult = exec_script(pkg_config_script, args, "value")
                ^----------
Current dir: /home/data/mlperf_inference/mlperf_inference/out/MakefileGnProj/
Command: python /home/data/mlperf_inference/mlperf_inference/build/config/linux/pkg-config.py glib-2.0 gmodule-2.0 gobject-2.0 gthread-2.0
Returned 1.
stderr:

  File "/home/data/mlperf_inference/mlperf_inference/build/config/linux/pkg-config.py", line 57
    print "You must specify an architecture via -a if using a sysroot."
                                                                      ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("You must specify an architecture via -a if using a sysroot.")?

See //build/config/linux/BUILD.gn:89:3: whence it was called.
  pkg_config("glib") {
  ^-------------------
See //build/config/compiler/BUILD.gn:218:18: which caused the file to be included.
    configs += [ "//build/config/linux:compiler" ]
                 ^------------------------------
Makefile:22: recipe for target 'mlperf_loadgen' failed
make: *** [mlperf_loadgen] Error 1


I had the same problem. Running the build command with sudo solved the problem 

> I had the same problem. Running the build command with sudo solved the problem

Thanks， it is solved.

Just wondering why a build command would require `sudo`...
Repository: mlcommons/inference
Issue: Why is there no resnet50 pytorch model?
Root cause: Unimplemented Feature Gap

there is no resnet50 pytorch model. Only resnet50_v1.onnx is availble. 
I tried to make a resnet50.pytorch model , but failed. Does anybody know how to build a resnet50.pytorch model, Thanks!

When we wrote the reference app long time ago pytorch was still on the onnx route so we used onnx -> pytorch/caffe2. 
This might still work but its not really what pytorch wants.
So we'd need to find a resnet50v1.5 model in pytorch, make sure its equivalent, jit/script it so we don't need to include the code in the reference app and add a backend to the reference app to run the scripted pytorch model.

I will put this item in the backlog for now. We need a volunteer for this.
Repository: mlcommons/inference
Issue: big gap between tensorflow/onnxruntime/pytorch in offline test result
Root cause: Algorithmic Error

in offline test case, 364(tf) to 247(onnxruntime) to 53(pytorch),  Who know the reason, why is there so big gap between the 3 stacks?
./run_local.sh **tf** ssd-mobilenet cpu --max-batchsize 4 --scenario Offline --count 640 --threads 32
TestScenario.Offline qps=**364.32**, mean=1.6841, time=1.757, queries=640

./run_local.sh **onnxruntime** ssd-mobilenet cpu --max-batchsize 4 --scenario Offline --count 640 --threads 32
TestScenario.Offline qps=**247.34**, mean=2.2958, time=2.588, queries=640

./run_local.sh **pytorch** ssd-mobilenet cpu --max-batchsize 4 --scenario Offline --count 640 --threads 32
TestScenario.Offline qps=**53.04**, mean=7.4860, time=12.066, queries=640


Reference implementation is meant to serve as a reference for the submitters to follow, but is not meant to be the most optimal implementation, so it is expected that their performance might be sub-optimal. Could you try the implementations in the actual submissions and see if the same issue exists? https://github.com/mlperf/inference_results_v0.7

The pytorch implementation is here:
https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/python/models/ssd_mobilenet_v1.py
but it looks that it is less than optimal (guess it has python code in the active path).

Perfect would be to use pytorch with scripting mode and add support for pytorch/scripting to the reference app.

There is no requirement on the reference code sleep so the performance differences are expected, but not desirable. I'm closing this issue. Please re-open if needed.
Repository: mlcommons/inference
Issue: Is there any computation overhead of using LoadGen ?

Is there any computation overhead of using LoadGen to generate queries according to the scenario (single-stream, multi-stream, server, offline) for **inference performance mode run**?(check figure-no. 3 in the page 8 of this [Paper](https://arxiv.org/pdf/1911.02549.pdf)).

If there is no overhead of using LoadGen for **inference performance mode run**, please explain the reason.

For MLPerf Inference 0.5, Google measured the Loadgen overhead and found that it was insignificant for results less than about 1.5 MPQS. NVIDIA made some improvements to the Loadgen in 0.7, so it should be possible to accurately measure performance up to about 5 MQPS. I think the Loadgen overhead is sufficiently low for practical applications.

https://github.com/mlcommons/inference/tree/master/loadgen/benchmark

closing this issue. please re-open if needed.
Repository: mlcommons/inference
Issue: resizing ImageNet data
Root cause: Documentation Deficiency

What is the easiest way to resize the  ImageNet data to 224x224? Is there any dedicated script for the task?

Yes, see the following functions:

- https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/python/dataset.py#L168
- https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/python/dataset.py#L187

This he final python script that I used to resize images
```
import cv2
import os
import numpy
import fnmatch
from shutil import copyfile


size = tuple([240,240])
inFolder = '/home/bt0034/CK-TOOLS/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_img_val'
outFolder = '/home/bt0034/mlperf_inference/vision/classification_and_detection/data/imagenet-224'
mapFile= '/home/bt0034/CK-TOOLS/dataset-imagenet-ilsvrc2012-aux/val.txt'
inputFormat = '*.JPEG'

copyfile(mapFile, outFolder+'val_map.txt')
count = 0;
for filename in os.listdir(inFolder):
    if not fnmatch.fnmatch(filename,inputFormat):
        continue;
    print('total processed: ' + str(count) + '; Now processing: '+ filename + ' '*10, end= '\r', flush=True)
    img = cv2.imread(os.path.join(inFolder,filename))
    if len(img.shape) < 3 or img.shape[2] != 3:
        img = cv2.cvtColor(img, cv2.COLOR_GRAY2RGB)
    img = cv2.resize(img, size, interpolation=cv2.INTER_LINEAR)
    cv2.imwrite(os.path.join(outFolder, filename), img)
    count += 1

```

Hi @sabquat 

Since you appear to have used Collective Knowledge to download the images and labels (cool!), you could have simply followed the [resizing instructions](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection/optional_harness_ck/classification#install-the-full-dataset-50000-images). This would give you a choice of preprocessing methods and resulting accuracies.

Hello @psyhtest and @nvpohanh,
Thanks for your suggestion. My problem is solved for now.
Repository: mlcommons/inference
Issue: tflite enablement
Root cause: Documentation Deficiency

Is it possible to run mleprf tool within tflite runtime (tflite interpreter) allowing various backend to execute and benchmark? do you have some yocto recipe for the tool itself?

For [vision modes](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection) tflite is somewhat supported. fp32 modes should work. But if you'd want to use a quantitized model some changes might be needed to prep the model inputs for it.

@guschmue please can you point me to how to use the mlerf?

Maybe start here: https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/GettingStarted.ipynb
This run_local.sh script should work with tflite if you call it with:
```
./run_local.sh tf  mobilenet cpu --backend tflite
```


please re-open if needed

Hi @peter197321, [dividiti](http://dividiti.com) submitted [lots](https://github.com/mlcommons/inference_results_v0.7/tree/master/open/dividiti/results/xavier-tflite-v2.2.0-ruy) of TFLite results to MLPerf Inference v0.5 and v0.7 using automated [Collective Knowledge](http://cknowledge.org) workflows. Please feel free to checkout a tutorial-style [Jupyter notebook](https://raw.githubusercontent.com/krai/ck-mlperf/master/jnotebook/image-classification-tflite-loadgen/image-classification-tflite-loadgen.ipynb) (save to disk, open in Jupyter).

Thank you for your suggestion, @psyhtest is that "Jupyter Notebook" somewhere - it seems the does not exist ...

@psyhtest ?


@peter197321 I'm so sorry, the file has got moved in the meantime. Please check the updated link above.

closing this ticket. please re-open again if needed.