zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: SSD-ResNet34 with onnxruntime and Cuda 10.0 support
Root cause: Environment Incompatibility

Hi all, I am running object detection as `./run_and_time.sh onnxruntime ssd-resnet34 gpu` with Docker, but it looks like the ONNX Runtime version is installed with cuda 9.1 support instead of cuda10.0 support. See below the tracelog:

```
INFO:main:Namespace(accuracy=False, backend='onnxruntime', cache=0, count=None, data_format='NCHW', dataset='coco-1200-onnx', dataset_list=None, dataset_path='/home/CK-TOOLS/dataset-coco-2017-val', inputs=['image'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], model='/home/mlperf_models/resnet34-ssd1200.onnx', output='/output/results_MultiStream_perf.json', outputs=['bboxes', 'labels', 'scores'], profile='ssd-resnet34-onnxruntime', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.MultiStream], threads=128, time=None)
/opt/anaconda3/lib/python3.7/site-packages/onnxruntime/capi/_pybind_state.py:12: UserWarning: Cannot load onnxruntime.capi. Error: 'libcublas.so.9.1: cannot open shared object file: No such file or directory'
  warnings.warn("Cannot load onnxruntime.capi. Error: '{0}'".format(str(e)))
Traceback (most recent call last):
  File "python/main.py", line 526, in <module>
    main()
  File "python/main.py", line 393, in main
    backend = get_backend(args.backend)
  File "python/main.py", line 219, in get_backend
    from backend_onnxruntime import BackendOnnxruntime
  File "/mlperf/python/backend_onnxruntime.py", line 7, in <module>
    import onnxruntime as rt
  File "/opt/anaconda3/lib/python3.7/site-packages/onnxruntime/__init__.py", line 21, in <module>
    from onnxruntime.capi._pybind_state import RunOptions, SessionOptions, get_device, NodeArg, ModelMetadata
ImportError: cannot import name 'RunOptions' from 'onnxruntime.capi._pybind_state' (/opt/anaconda3/lib/python3.7/site-packages/onnxruntime/capi/_pybind_state.py)
```




yes, not really a mlperf issue ... the released version of onnxruntime just didn't come with a cuda10 build and one would need to build from source. 
There is a new onnxruntime release coming soon and that will use cuda10.
In the released version of onnxruntime there are a bunch of warnings logged for ssd* that are also fixed in the next onnxruntime release.
Repository: mlcommons/inference
Issue: Calibration data for GNMT
Root cause: Unimplemented Feature Gap

@guschmue agreed to provide us with calibration dataset for GNMT. 

@guschmue just a note for the two of us that when you push the calibration dataset in, we need to make sure we update the rules [here](https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#81-weight-definition-and-quantization) as it current states:

> MLPerf will provide a calibration data set for all models except GNMT. 

will take me a bit to get to this.

@DilipSequeira will work on this per WG agreement.
Repository: mlcommons/inference
Issue: [Cloud Language modeling] benchmark.py has to be executed with python3 and not python2
Root cause: Environment Incompatibility

in the file https://github.com/mlperf/inference/blob/master/cloud/language_modeling/README it is specified to start benchmark.py with python, but urllib.request does not work. It only exists with python3.

by the way, this patch has to be applied

diff --git a/cloud/language_modeling/benchmark.py b/cloud/language_modeling/benchmark.py
index 814f9f7..022f941 100644
--- a/cloud/language_modeling/benchmark.py
+++ b/cloud/language_modeling/benchmark.py
@@ -229,7 +229,7 @@ def loadValidationDataset(arguments):
     dataset = Dataset(vocab, int(arguments["words_per_sample"]), int(arguments["maximum_samples"]),
         int(arguments["mini_batch_size"]))

-    with open(path, "r") as datasetFile:
+    with open(path, "r" , encoding="utf-8") as datasetFile:
         for line in datasetFile:
             for word in line.split():
                 dataset.addWord(word)


closing & the issue seems to be addressed. Please re-open if the fix is not right.
Repository: mlcommons/inference
Issue: Dockerfile.gpu with TF 1.13.1 instead of TF 1.14.0
Root cause: Configuration Error

Hi all, the main README.md page says the benchmarks have been tested on TF 1.14; however, the Dockerfile.gpu on classification_and_detection folder has the instruction `RUN pip install tensorflow-gpu==1.13.1 onnxruntime-gpu` , should it be updated to tensorflow-gpu==1.14.0?

yes, can update the images to 1.14

Thanks
Repository: mlcommons/inference
Issue: AttributeError: 'mlperf_loadgen.TestSettings' object has no attribute 'min_query_count'
Root cause: External Dependency Breakage

Hi all, I am getting `object has no attribute 'min_query_count'` errors, please see below the code version and the trace log

$ `git log -1`
```
commit 2749668398d63fd9078eaaad8818b4e8442ac51a (HEAD -> master, origin/master, origin/HEAD)
Merge: e37ce8a 3ee4877
Author: jveejay <jvijay@gmail.com>
Date:   Thu Jul 18 18:25:11 2019 -0400

    Merge pull request #274 from briandersn/lg_typos

    Make python and C++ linters happy
dell@r740xd:/home/user1/mlperf_inference_07222019/v0.5$
```
Commands used:
```
./run_and_time.sh pytorch ssd-resnet34 <cpu/gpu>
./run_and_time.sh tf resnet50 <cpu/gpu>
```


Trace log
```
INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format='NHWC', dataset='coco-1200-tf', dataset_list=None, dataset_path='/home/CK-TOOLS/dataset-coco-2017-val', inputs=['image:0'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], model='/home/mlperf_models/ssd_resnet34_mAP_20.2.pb', output='/output/results.json', outputs=['detection_bboxes:0', 'detection_classes:0', 'detection_scores:0'], profile='ssd-resnet34-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream], threads=80, time=None)
INFO:coco:loaded 201 images, cache=0, took=206.3sec
2019-07-22 21:08:13.029987: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-07-22 21:08:13.084203: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500000000 Hz
2019-07-22 21:08:13.091817: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55764883c560 executing computations on platform Host. Devices:
2019-07-22 21:08:13.091874: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
Traceback (most recent call last):
  File "python/main.py", line 531, in <module>
    main()
  File "python/main.py", line 474, in main
    settings.min_query_count = args.queries_single
AttributeError: 'mlperf_loadgen.TestSettings' object has no attribute 'min_query_count'
```


Must be an older loadgen. The way we build the docker images - we build loadgen inside the image and even if some loadgen source code has been update, it might not rebuild that part of the image. You could add a ```--no-cache``` when building the image.
I don't know a good way of ensuring that loadgen rebuilds on change, thinking ...

thanks @guschmue, I have removed the image and rebuilt it
Repository: mlcommons/inference
Issue: changing parameters in the TestSettings structure 
Root cause: Documentation Deficiency

Hi,

How to interpret and set the following fields in the TestSettings structure:
- max_duration_ms 
- min_duration_ms
- max_query_count
- min_query_count

is there any documentation about TestSettings parameters, there influence on the Benchmark and allowed values per scenario?

Thanks,
Reuven


@christ1ne @briandersn can you help?

test_settings.h has comments:

  // The test runs until both min duration and min query count have been met.
  // However, it will exit before that point if either max duration or
  // max query count have been reached.
  uint64_t min_duration_ms = 10000;
  uint64_t max_duration_ms = 0;  // 0: Infinity.
  uint64_t min_query_count = 100;
  uint64_t max_query_count = 0;  // 0: Infinity.

Let me know if further details are needed.



Hi Brain,
so if i am running single stream, and to process 1024 samples it takes me 0.5 second how should i configure the test setting so it will run 60 seconds? does load gen adopts  the timing automatically?
or i need to push more samples?
thanks
Reuven 

Try:

single_stream_expected_latency_ns = (1e9 * 0.5) / 1024;
min_duration_ms = 60000;
max_duration_ms = 0; // 0: Infinity.

The loadgen uses single_stream_expected_latency_ns to determine how many queries to pre-generate so it doesn't need to generate them on the fly.

Brian thanks, i will try setting these parameters as suggested, also to finalize this issue i would like to check the following option:
if i have uploaded 1024 sample, can loadgen keep transmitting these 1024 samples throughout the benchmark run duration in a cyclic transmission  (assuming the benchmark is run for 60 seconded), that way i can lower the expected latency and check the lower bound.


Hi Brain,
Please ignore the last question, i tried your suggestion and it works.
thanks,
Reuven
Repository: mlcommons/inference
Issue: Quantized TF Models
Root cause: Unimplemented Feature Gap

In the [main table](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection#supported-models)  I see the two `int8` models for ONNX but not for TF/TF-Lite. 
Can someone add the rows for the equivalent TF/TF-Lite models and their accuracy (as reported by MLPerf reference flow)?

@parvizp #310 partially addresses your comment: it's only for the quantized MobileNet TF model, and [via CK](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification). I will validate the quantized MobileNet TFLite model shortly.

Good news. I've also found [accuracy results](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/detection/tf-py#ssd-mobilenet-quantized-finetuned) for the quantized finetuned SSD-MobileNet TF model provided by Habana.

When measuring via CK with OpenCV preprocessing, the Top 1 accuracy of the non-quantized TFLite model is the same as that of the non-quantized TF model (71.516%). Potentially it's the same when measured via the official code (71.68%).

The Top 1 accuracy of the quantized TFLite model, however, is higher than that of the quantized TF model (70.776% vs. 70.654%, or 61 more correctly classified images). Interestingly, the Top 5 accuracy is lower (89.198% vs. 89.514%, or 158 fewer correctly classified images).

Thanks for taking the time to run and update the README @psyhtest ! 

So is the `mobilenet-v1 quantized` accuracy of `70.776%` "valid" with respect to the 99% of FP32 threshold? Even using the lower FP32 accuracy from TF-Lite of `71.516*0.99 = 70.800 > 70.776`?

I wasn't aware of the finetuned SSD model from Habana, thanks for pointing that out. Just to be clear you ran it in TesnorFlow with the FakeQuant nodes rather than as a "true" quantized TF-Lite model to get your accuracy score?


> So is the mobilenet-v1 quantized accuracy of 70.776% "valid" with respect to the 99% of FP32 threshold? Even using the lower FP32 accuracy from TF-Lite of 71.516*0.99 = 70.800 > 70.776?

Nope. See my comment [here](https://github.com/mlperf/inference/issues/304#issuecomment-517374475).

> Just to be clear you ran it in TensorFlow with the FakeQuant nodes rather than as a "true" quantized TF-Lite model to get your accuracy score?

Yep. I think it could be converted to TFLite too using the kind of [gymnastics](https://github.com/ctuning/ck-mlperf/blob/master/package/model-tflite-mlperf-ssd-mobilenet/README.md) required for the non-quantized model, but we haven't done this.
Repository: mlcommons/inference
Issue: C++ mlperf benchmark accuracy check
Root cause: Documentation Deficiency

Hi,
i tried to understand how the accuracy check should be implemented in the C++ benchmark, in the python reference example the accuracy test seems to be bypassing the loadgen interface.
is there any documentation of this subject? what is the proper way to implement?
Thanks,
Reuven


Hi Reuven, This is what we are trying to figure out as well (and have made good progress on recently). 

But I think Brian has a plan to provide documentation and examples?

@briandersn

The way it is suppose to work:
the benchmark app passes back the results to loadgen, which will store it in a json file. After the benchmark a script will read the json file and calculates accuracy. We don't have don't have the scripts yet but the code is there in the reference app so its mostly a matter of wrapping the code in a standalone tool.

There is a issue for this already here:
https://github.com/mlperf/inference/issues/241


We have done such wrapping with [CK](https://github.com/ctuning/ck) for image classification models. Please contact me if you think this could be of use.

one question that popped out and connected to this issue,
when reporting results i tried to followup the python example in my c++ code meaning:
1. i converted the results of each sample (image) to floats.
2. build a QuerySampleResponse  object for that sample containing the ID, Pointer to the float results and size in bytes.
3. for multiple sample i generated a buffer of QuerySampleResponce
4. called the QuerySamplesComplete

is this the expected process?
Thanks,
Reuven
Repository: mlcommons/inference
Issue: Do not round up accuracy
Root cause: Documentation Deficiency

The ImageNet 2012 validation set has 50,000 images. When we calculate the top 1 accuracy, we generally get a number with 5 digits after the floating point e.g. 0.71226. Multiplying this number by 50,000 gives us the number of correctly predicted images e.g. 35,613.

However, we currently often represent this number in percent with only two digits after the floating point e.g. 71.22% (if the last digit is discarded) or 71.23% (if the last digit is rounded up) which gives us a range of correctly predicted images from 35,610 to 35,617. This leaves room for doubt whether a particular result is fully reproduced or not.

I suggest to adopt the following rule for documentation and reporting of ImageNet results: report the top 1 accuracy with 5 digits after the floating point, or, if in percent, with 3 digits after the floating point.

While the mAP metric is not related to the number of images, we could also adopt the same rule for COCO results.

With 3 digits after the floating point, the Top 1 accuracy for MobileNet is coming out at 71.676%, or 35,838 images.
```
TestScenario.Offline qps=207.42, mean=5.8869, time=7.714, acc=71.676, queries=1600, tiles=50.0:5.4863,80.0:7.2807,90.0:7.4816,95.0:7.5882,99.0:7.8223,99.9:8.2713
```
Repository: mlcommons/inference
Issue: Should we implement loadSamplesToRam() by ourselves?
Root cause: Unimplemented Feature Gap

Hi all, 
I saw the codes about loadSamplesToRam() [here](https://github.com/mlperf/inference/blob/895453b7a29af5d6925e34c0089c8d803042cb61/loadgen/query_sample_library.h#L50) and [here](https://github.com/mlperf/inference/blob/895453b7a29af5d6925e34c0089c8d803042cb61/loadgen/loadgen.cc#L932) but it actually is "empty" realizations.
So we need to implement it by ourselves?
Thanks.
Repository: mlcommons/inference
Issue: Multistream with N samples per query problem
Root cause: Validation Gap

We have the single stream, multi stream working, but we have one bug in our benchmark when running single stream with N sample per query in the multi-stream (seems to be connected to the why frames are been sent in each query by loadgen.
i would like to verify the following
1. When loadgen issues loadSamplesToRam request, the loaded images has some defined 
   sequence (pseudo random) - the function loads these images to to continues RAM area 
   according to the sequence defined by loadgen.
2. When the loadgen issues a new query in multi-stream scenario for example a query of 4 
   samples, they will reside in the loaded RAM area in continues space and according the the 
    predefined sequence of samples as defined in section 1.  there is no wrap around in indexes or  
    access going beyond  the allocated RAM boundaries? 
3. Taking into consideration item 1 and 2 we would like to verify that following assumption is 
    always true:
   **Taking the pointer to the first sample in the query will point to a continues memory 
   location containing all query samples  and according to the sequence defined by loadgen when loading the images.**
Thanks
Reuven 

i suspect there is a bug in loadgen, when number of samples loaded is not dividable by the number of sample per query it seems that an query is issued  with with less samples that indicated in the query number of samples (i received 10 samples but the actual continuous space included only 8 samples)

i found out that lodgen requests to load more than 3*2^13 samples (24756) as written in the mlpref  documentation while our image buffer allocation is for 3*2^13 images but it seems that loadgen request more than that so our allocation is not enough.
Why does the loadgen request to upload images beyond 24576?


@sheferke can i trouble you to please clarify what the resolution for this is issue was and re-close so that people understand the issue in case they run into it.
Repository: mlcommons/inference
Issue: Not able to run demo scripts
Root cause: Environment Incompatibility

 Hi all,

I am new to MLPerf. I followed the python instructions to install loadgen and see below error when I try to run the demo script. Can somebody please let me know how to address this? I would like to get some familiarity with Loadgen and how to run the benchmark script before the Newcomers meeting today.

python demos/py_demo_single_stream.py

Traceback (most recent call last):
  File "demos/py_demo_single_stream.py", line 26, in <module>
    import mlperf_loadgen
ImportError: /home/susmitha/anaconda3/lib/python3.6/site-packages/mlperf_loadgen.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZTVNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE

Below are the instructions that I followed and see above error at the last step.
pip install absl-py numpy
git clone --recurse-submodules https://github.com/mlperf/inference.git mlperf_inference
cd mlperf_inference/loadgen
CFLAGS="-std=c++14" python setup.py bdist_wheel
pip install --force-reinstall dist/mlperf_loadgen-0.5a0-cp36-cp36m-linux_x86_64.whl
python demos/py_demo_single_stream.py

Thanks
Susmitha


Hi @SusmithaGadagottu. I was browsing the bugs and noticed this never got any attention. Is this still an issue for you?

I haven't seen this specific error before; however, it looks like there might be some kind of missmatch between the standard library the loadgen was compiled against (with setup.py) vs. what's being used at runtime of the demo.




Hi Brian,

Thanks for getting back to me on this issue.
I will get back to you later if I need assistance on this issue.

Thanks
Susmitha

On Thu, Aug 15, 2019 at 2:46 PM Brian C. Anderson <notifications@github.com>
wrote:

> Hi @SusmithaGadagottu <https://github.com/SusmithaGadagottu>. I was
> browsing the bugs and noticed this never got any attention. Is this still
> an issue for you?
>
> I haven't seen this specific error before; however, it looks like there
> might be some kind of missmatch between the standard library the loadgen
> was compiled against (with setup.py) vs. what's being used at runtime of
> the demo.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/297?email_source=notifications&email_token=AMXY4IHQCBGUCEHZHJE4SKLQEXFDFA5CNFSM4IHVNXPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4NCZ5Y#issuecomment-521809143>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AMXY4IGWIV5LZKLACPS7DJDQEXFDFANCNFSM4IHVNXPA>
> .
>
Repository: mlcommons/inference
Issue: Python support/ Invoking from python
Root cause: Unimplemented Feature Gap

I think that additional support for python should be added as the current code require to invoke the run from C. For instance if a part of the processing involves python (e.g. tokenization), why this should be precluded by the fact it has to be invoked from C? 
Also, python support can ease the integration with the future reference benchmarks code.

I think this is a good idea, but I'm not very good at Python. Can you take a crack at porting it?

Sorry for pushing this back on you.

I think this is done - closing.

please re-open if further action is needed @itayhubara 

what is missing ?

I think it is an outdated comment from Nov last year. Just want to clean up the git issues.
Repository: mlcommons/inference
Issue: Source code of quantized models
Root cause: Documentation Deficiency

The model source of the quantized mobilenet and ssd-mobilenet models are missing. Can someone add them?
Also, the mentioned framework of the quantized ssd-mobilenet is onnx, pytorch while the zenodo points to a TF model.

please read the google doc on the mobilenet model to see if the info is detailed enough:https://zenodo.org/record/3353417/#.XUHa3JNKhwc

@itayhubara please share the source code for the model quantization on the ssd-mobilenet int8 model.

Thanks, the info is helpful, but I would appreciate the source code. 
I see that the preprocessing for the int8 model and the fp32 models are different, but the benchmark support only one case, i.e. "imagenet_mobilenet".

@eladcohe can you provide more details on the preprocessing difference between the int8 model and the fp32 model which you observed? Thanks!

@DilipSequeira can you provide instructions for running Mobilenet ONNX int8 model? I am not able to match the given accuracy.

The original trained accuracy was 70.9%. The reference preprocessing code is slightly different from the one used for training, so you should expect 70.81%. We do not have a reference integration into loadgen.
Repository: mlcommons/inference
Issue: MLPerf not working for Resnet50 model from Model Zoo
Root cause: Interface Contract Mismatch

MlPerf works fine for the following resnet50 model -
`Zenodo Resnet50 : https://zenodo.org/record/2541184#.XUDRbeTsZaQ` 

But when I try the following model, it does not work -
`Model Zoo Resnet50 : https://github.com/onnx/models/tree/master/vision/classification/resnet/resnet50`

The reason is caused by the following line in main.py()
`processed_results = self.post_process(results, qitem.content_id, qitem.label, self.result_dict)`

This is returning a blank array.

The command I am using -
`python python\main.py --profile resnet50-onnxruntime --model resnet50.onnx --dataset-path ImageNetImages\First_10 --output results.json --data-format NCHW --max-latency 0.2 --count 10 --accuracy`

 I had to change the profile because of different output -
`"resnet50-onnxruntime": {
        "dataset": "imagenet",
        "outputs": "gpu_0/softmax_1",
        "backend": "onnxruntime",
        "max-latency": LATENCY_RESNET50,
    },`




those are not the same models. For mlperf you MUST use or (derive from) https://zenodo.org/record/2541184#.XUDRbeTsZaQ
Repository: mlcommons/inference
Issue: Pybind Issue
Root cause: Environment Incompatibility

I am trying to run the gettingstarted notebook and I get this issue:

bindings/python_api.cc:179:16: error: expected constructor, destructor, or type conversion before ‘(’ token
 PYBIND11_MODULE(mlperf_loadgen, m) {

It seems the code is not able to build mlperf_loadgen .. Can you please help?

@coder27 It's a duplicate of https://github.com/mlperf/inference/issues/230 (which perhaps should not have been closed until properly resolved). Can you try the solutions I described there?

Thanks. I was able to resolve it using recursive cloning ..