zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Inconsistence between .ckpt and .pb in ssd-resnet34 1200x1200 tensorflow.
Root cause: External Dependency Breakage

Hi all, 

I found that the .pb and .ckpt provided by mlperf in ssd-resnet34 1200x1200 tensorflow may be inconsistent.

I utilized [eval_ssd_large.py](https://github.com/mlperf/inference/blob/master/others/cloud/single_stage_detector/tensorflow/train/eval_ssd_large.py) to test the performance of .ckpt, but average precision (AP) was just 0. 

Then, I used [export_graph.py](https://github.com/mlperf/inference/blob/master/others/cloud/single_stage_detector/tensorflow/train/export_graph.py) to convert downloaded .ckpt to .pb, and then used tensorboard to observe the graph structure. I made the same operation on the downloaded .pb. 
And I found that the these two graphs are different, as shown below.
<img width="369" alt="ssd_pb" src="https://user-images.githubusercontent.com/18712278/75840909-6033a200-5e07-11ea-9fe4-ce4de7af132e.png">
graph for downloaded .pb

<img width="330" alt="ssd_mlperf" src="https://user-images.githubusercontent.com/18712278/75840918-6c1f6400-5e07-11ea-980c-e82b1ad04d96.png">
graph for .pb converted by .ckpt


At the same time I noticed that in ssd-resnet34 tensorflow, two repositories were given. I also tried [export_graph.py](https://github.com/lji72/inference/blob/tf_ssd_resent34_align_onnx/others/cloud/single_stage_detector/tensorflow/train/export_graph.py) in another repository, [lji72,](https://github.com/lji72/inference/tree/tf_ssd_resent34_align_onnx/others/cloud/single_stage_detector/tensorflow) and got another different graph.
<img width="680" alt="ssd_lji72" src="https://user-images.githubusercontent.com/18712278/75840959-878a6f00-5e07-11ea-9307-4f1852cbdee2.png">
graph for .pb converted by .ckpt via code in lji72 repo

In summary, based on the provided .ckpt, I cannot get a correct AP result based on eval_ssd_large.py. Also I cannot get an effective .pb based on export_graph.py.

Both .pb and .ckpt were downloaded from the [official link](https://zenodo.org/record/3345892) 

Please help me. Thank you all.

@ShenTonyM  you can try lateset ckpt and pb model at:https://zenodo.org/record/3712664#.XnB1D8eP63A

Hi @lji72 , many thanks for your help. Your latest .ckpt solves my problem.
Repository: mlcommons/inference
Issue: I got Imagenet Dataset from my friends ~ I want to use~
Root cause: Documentation Deficiency

![image](https://user-images.githubusercontent.com/55476509/75850820-7d7a6780-5e2b-11ea-8630-6bd7d7d6ad3e.png)

It said  No such file or directory: 'imagenet_data/annotations/instances_val2017.json'

![image](https://user-images.githubusercontent.com/55476509/75850986-e82ba300-5e2b-11ea-87f9-9fef814e479c.png)

Here is the directory of imagenet_data which is DATA_DIR

I guess this is because I don't have that .json file.

Can you guys know how I can make or get json file for imagenet to use in MLperf Inference~



Thank you! Always!
Kwan 


Kwan, you must use ImageNet with ResNet or MobileNet, not with SSD-MobileNet or SSD-ResNet. It appears you are trying SSD-MobileNet with ImageNet now.

@psyhtest 

Oh right!

I tried ~ 

Now it wants val_map.txt

Do you know how I can make it? 
I tried to find it with "find" program but it wasn't there

![image](https://user-images.githubusercontent.com/55476509/75866109-e8d23280-5e47-11ea-9ada-993a1d733b1e.png)




![image](https://user-images.githubusercontent.com/55476509/75866143-fa1b3f00-5e47-11ea-96a2-81ecab2a4bb9.png)


Kwan, referring you to the CK instructions [here](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/#using-collective-knowledge-ck):
```bash
$ python -m pip install ck --user
$ ck pull repo:ck-env
$ ck install package --tags=image-classification,dataset,imagenet,aux
$ ck locate env --tags=image-classification,dataset,imagenet,aux
/home/dvdt/CK-TOOLS/dataset-imagenet-ilsvrc2012-aux
$ ls -la `ck locate env --tags=aux`/val.txt
-rw-r--r-- 1 dvdt dvdt 1644500 Feb 25  2014 /home/dvdt/CK-TOOLS/dataset-imagenet-ilsvrc2012-aux/val.txt
```
Copy this file to `imagenet_data/val_map.txt` (note you need to rename it).

@psyhtest 

Thank you !

When I download this

$ ck install package --tags=image-classification,dataset,imagenet,aux

There was val_map with different name ~

I just brought it and change name to val_map and it works! 

Thank you!

@kdk3776 Always happy to help.

@psyhtest Thanks 

please re-open if needed
Repository: mlcommons/inference
Issue: I tried to run MLperf on my lab server gpu
Root cause: Environment Incompatibility

I tried to run MLperf on my lab server gpu

I access to server by ssh 

I cloned MLperf and tried to run ~
![image](https://user-images.githubusercontent.com/55476509/75851455-0e057780-5e2d-11ea-98e5-fe8facccae61.png)

It keep saying that No module named queue.

I didn't have this problem when I using my desktop.

I guess I need to download queue module.

pip install queuelib

I did this. 
![image](https://user-images.githubusercontent.com/55476509/75851607-76ecef80-5e2d-11ea-868c-7b602a5c17bc.png)

But it is still not working~ 

Can you guys help me out~~~?


Thank you!
Kwan.

Maybe you need to repeat this with python3?
```bash
$ python3 -m pip install queuelib
```

![image](https://user-images.githubusercontent.com/55476509/75872858-44a1b900-5e52-11ea-9aab-61a95b6213aa.png)

Still not working...

It said Requirement already satisfied~

Um~ do you have any other idea~?

if you call ```python3 setup.py develop``` in v0.5/classification_and_detection it should install everything you'd need. Never seen an issue with queue.Queue but I only use anaconda.

@guschmue 

When I type python --verstion 

It said python 2

So I changed it 
![image](https://user-images.githubusercontent.com/55476509/76960323-8ce6ce00-695e-11ea-9715-4ef46ce31b87.png)

![image](https://user-images.githubusercontent.com/55476509/76960341-940ddc00-695e-11ea-9469-496a2a2e4a41.png)


Now it said this one 

Python 3 this one too 
![image](https://user-images.githubusercontent.com/55476509/76960357-9d974400-695e-11ea-9108-66d5c5a6812f.png)



ah, don't change /usr/bin ... if might break some apps that depend on python2. Maybe do something like ```alias python=/usr/bin/python3```.
But to fix the original issue:
```
cd loadgen
python3 setup.py develop
```

@guschmue @psyhtest 

git clone --recurse-submodules URL

I solved by this.

After I clone again, I did python3 setup.py develop and it works! 

Thank you so much! 

https://github.com/mlperf/inference/issues/230


closing this issue due to lack of activity. please re-open if needed.
Repository: mlcommons/inference
Issue: How to build loadgen with gprof support
Root cause: Documentation Deficiency

Hi,

I am building mlperf loadgen C++ lib with `make mlperf_loadgen` 
I would like to profile my harness with gprof, for that I need to pass `-pg` command line option both to compiler and linker. I am new to `gn` and `ninja`, I honestly spent few hours trying to figure out how to pass `-pg` with this build system and utterly failed. I am probably missing something obvious, any hints on where should I look into are appreciated!

Thanks,
Max.


cmake works as well if that helps:
```
mkdir build/ && cd build/
cmake ..
make
```
If you want to pass in extra flags:
```
CFLAGS='...' make
```
should work.

@guschmue is right. We [added](https://github.com/mlperf/inference/pull/239) CMake support a while back.

@milakov You can also build LoadGen from scratch using Collective Knowledge:

```bash
$ python3 -m pip install ck --user
$ ck pull repo:ck-mlperf
$ ck install package --tags=mlperf,loadgen,static --env.EXTRA_FLAGS="-pg"
...
-- Using C++ compiler flags:  -std=c++14 -pg -Wno-psabi -O2 -W -Wall
-- Using C++ standard: 14
...
Installation path: /home/anton/CK_TOOLS/lib-mlperf-loadgen-static-gcc-8.3.0-compiler.python-3.6.7-linux-64

Installation time: 27.430544138 sec.

$ ls -la `ck locate env  --tags=mlperf,loadgen,static`/lib/libmlperf_loadgen.a
-rw-r--r-- 1 anton dvdt 2356462 Mar 18 19:39 /home/anton/CK_TOOLS/lib-mlperf-loadgen-static-gcc-8.3.0-compiler.python-3.6.7-linux-64/lib/libmlperf_loadgen.a
```

This automatically downloads the MLPerf Inference repo (this one), detects your compiler, CMake, Python, etc.

@guschmue and @psyhtest cmake works like a charm for me! `-DCMAKE_CXX_FLAGS=-pg -DCMAKE_EXE_LINKER_FLAGS=-pg -DCMAKE_SHARED_LINKER_FLAGS=-pg` command line arguments for the cmake invocation do the job. Thanks a lot!
Repository: mlcommons/inference
Issue: MultiStream Target Latency~
Root cause: Documentation Deficiency

<img width="200" alt="스크린샷 2020-04-07 오후 4 29 12" src="https://user-images.githubusercontent.com/55476509/78642336-85776c80-78ed-11ea-91de-cf2ec1f683cd.png">

Now I tried to run cpu pytorch resnet50 in MultiStream scenario.
Even though I tried to change Samples_per_query as 1.
I still got invalid because my cpu doesn't  meet target latency.
I got valid result  when I run mobilenet. So I thought resnet50 is slower than mobilenet.

Anyway what I want to know is that 1) why target_latency is setting as 100,000,000.
2)100,000,000 is a kind of MLperf standard?
3)Can I change it?
4)How can I change it?
(I saw them in loagen.cc)
<img width="497" alt="스크린샷 2020-04-07 오후 4 41 23" src="https://user-images.githubusercontent.com/55476509/78643077-a8eee700-78ee-11ea-881a-58fa5a1f6bba.png">


Thank you always! 


ResNet has ~7x more GOPS than MobileNet (7.8 vs. 1.1), so yes, it is slower.

The values in `LoadGen.cc` are defaults. They can be overridden by a config file. 

@psyhtest Anton~

Can you let me know the detail about config file~?
Where I can find it and how can I change with it?
 
Thank you!


the config file is here https://github.com/mlperf/inference/blob/master/v0.5/mlperf.conf
The values in there are the once you'd need for a valid submission.

@guschmue 

I clone git hub about last year august.
In my clone directory, there is no mlperf.conf. I mean in v0.5
Should I clone again? or I can just download conf file.



This file came in 9/5/2019, so yes, you need to do a 'git pull'.
Repository: mlcommons/inference
Issue: About latency in MultiStream.

What I know is that MultiStream send multi samples per query.
Ex) 3 samples per one query. 

1) When they return,  three samples return together or 1 samples each?
1-2) If they return together, they will have same latency?
1-3) If they return separately, they will have different latency?  


2)When we see in parameters used, there is min_query_count(=24576).
I tried with different Samples_per_query. It seems fixed.

Which means that if I set SPQ as 3, total samples number will be 24576*3?
 
 24576(query count) * 3(Samples_per_query)  = 73728(Total samples count)

If I set Samples_per_query = 4

 24576(query count) * 4(Samples_per_query)  = 98304(Total samples count)


Can I understand like this?
 

<img width="200" alt="스크린샷 2020-04-07 오후 4 29 12" src="https://user-images.githubusercontent.com/55476509/78644860-8b6f4c80-78f1-11ea-998f-46fc2c550020.png">

min_query_count is independent of the SPQ. It is the total amount of queries needed for a test to generate valid results. please reopen if needed.
Repository: mlcommons/inference
Issue: Is mlperf inference able to run on Windows?
Root cause: Documentation Deficiency

I want to run mlperf on windows.
I have been running mlperf on linux.
I am wondering it is possible to run mlperf on window.

Thank you always!

I want to know mac too! 

yes, loadgen and the python scripts work on windows. The shell scripts like run_local.sh don't so you need to make up the command line to the python script manually.

@guschmue  

Thank you so much! 
I bet mac work too right? 



I remember there was some PR to fix loadgen on osx so I assume somebody used this.

Yes, @ens-lg4 used LoadGen on Mac (but may be not the official app, only CK implementations). 

you can check out past intel core products results e.g. https://github.com/mlperf/inference_results_v0.5/tree/master/closed/Intel/code/ssd-small/openvino-windows
Repository: mlcommons/inference
Issue: Using MobileNetV2 with TensorFlow Lite Benchmarking.
Root cause: Unimplemented Feature Gap

Goal: Compare performance of MATLAB Coder with TFLite on CPU.

I have used instructions in the "Getting Started" guide (https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/GettingStarted.ipynb) to run the TFLite benchmark using the MobileNetV1 model (from here: https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection#supported-models):

./run_and_time.sh tflite mobilenet cpu

I have results using MobileNetV2 for MATLAB Coder. In order to make a fair comparison, how can I use MobileNetV2 with TFLite in the MLPerf workflow? 

Can I use the model in the TensorFlow Lite documentation (https://www.tensorflow.org/lite/guide/hosted_models) in the aforementioned MLPerf workflow? 

Hi @sindhujapr, I'm not sure why you want to use the MLPerf workflow for this purpose. I doubt you need any particular LoadGen scenarios (SingleStream, MultiStream, Server, Offline), as your "MATLAB Coder" most likely doesn't. Also, the official workflow specifically assumes a handful of models. You can extend it for sure, but you may be better off using one of the customizable and extensible [CK workflows for image classification](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification) which work with MobileNet-v2. (See [here](http://cknowledge.org/dashboard/mlperf.inference).)

As an aside, TFLite on x86 is not known to particularly well optimized. I suspect  comparing against MKL-optimized TensorFlow would be fairer.

Thank you for getting back to me @psyhtest . We should be able to mimic LoadGen scenarios through the code generated by MATLAB Coder. I was wondering if you had any suggestions on extending the MLPerf workflow with MobileNet2. I was thinking about extending it with this MobileNetV2 model: https://www.tensorflow.org/lite/guide/hosted_models.

I agree that comparing against MKL-optimized TensorFlow would be fair when comparing TFLite on PC machines. In addition to that, I am trying to look into TFLite performance on embedded devices.
Repository: mlcommons/inference
Issue: Offline and Server scenario problems
Root cause: Resource Mishandling

I use C++ api, but there is something wrong wthin mlperf::c::StartTest() function when finish all test:
pthread_cond_wait@* L185 PC: 0x7fffe897636
in mlperf::loggingLLAsyncLog::GetLatenciesBlocking(unsigned long)（）
in mlperf::logging::Logger:GetLatenciesBlocking(unsigned long)

can help me what's wrong with my test?

is the problem still there?
Repository: mlcommons/inference
Issue: MultiStream scenario problem:ERROR:Attempted to complete a sample twice
Root cause: Validation Gap

when finished test of MultiStream scenario, there ara errors in mlperf_log_detail.txt： ERROR:Attempted to complete a sample twice，what's wrong in the test?

First of all, what is your test? 

do you mean what's the mode i test? i test performance mode.
I found the error is caused by two times call mlperf::QuerySamplesComplete() function.
thank you anyway.

It sounds that the issue was with your implementation. Glad to hear you've solved it. Please resolve the issue then.
Repository: mlcommons/inference
Issue: About Server scenario
Root cause: Documentation Deficiency

Hi guys! Thank you always!

I tried to run server scenario while I was running it. I had some question about the scenario.

1) About Server metric. 

What I know that the metric is queries per second.
![스크린샷_2020-04-15_18-52-16](https://user-images.githubusercontent.com/55476509/79323300-fbf91780-7f48-11ea-8f2c-273c43b915e7.png)

I know that usually metric is on the top of the log_summary.

1-1) On this, Scheduled samples per second? is it correct?
since samples per query is 1. queries per second will be same as samples per second.

1-2) If you see, I tested with different qps. And samples per second is linearly increasing with qps.
I wonder that I could see the result as correct metric since it changed like that.

2) About Server test setting.

![스크린샷_2020-04-15_19-01-20](https://user-images.githubusercontent.com/55476509/79324220-3a430680-7f4a-11ea-93ef-1d32add976b3.png)
First, I tried to test without any command line parameter. but I couldn't get valid result. I got something like "nan".
![스크린샷_2020-04-15_18-58-56](https://user-images.githubusercontent.com/55476509/79323998-e801e580-7f49-11ea-9295-85460fc90f14.png)
I tried with --max-latency 1 and it worked.

2-1) I wonder why this parameter make valid result and what this parameter stands for? 


2-2) I changed value of --max-latency and I could see target latency as multiple of 10*--max-latency.
But I tried this with Multi stream scenario. But in multi stream scenario, it doesn't change.

Do you know why it change and why it isn't change? 
 
![스크린샷_2020-04-15_19-03-53](https://user-images.githubusercontent.com/55476509/79324453-9f96f780-7f4a-11ea-9952-d5786db6e70b.png)

Thank you!
Kwan


 


unfortunately the current MLPerf documentation is not there. Please consult the source code under the loadgen folder
Repository: mlcommons/inference
Issue: Running MultiStream with tflite on mobilenet
Root cause: Validation Gap

![1](https://user-images.githubusercontent.com/55476509/79407851-ef220580-7fd4-11ea-9dc7-faa7ab571d90.jpg)

I tried to run MultiStream with tflite on mobilenet.

When I set samples per query as 1, there is no error sign.
But if I set samples per query more than 1, I can see message on screen.

"Got 2(3,4 depend on samples per query) but expected 1 for dimension 0 of input 88.

Q) I get valid result when I was running SPQ as 1. But I wondering I get valid result with bigger SPQ like 2,3,4. Because it tried several time and it takes really long time. Should I wait or it just not working?

Q) This specific backend(tflite) can not run multistream scenario more than 1 samples per query?
Repository: mlcommons/inference
Issue: Measuring time on gpu vs cpu
Root cause: Documentation Deficiency

Is there a difference in measuring time on cpu and gpu, e.g., waiting for gpu to complete computation before we stop measuring the execution time? Could you point to the code snippets in mlperf where inference on GPU is measured for Tensorflow and PyTorch demonstrating how this is done in mlperf?

No difference.
Both cpu and gpu call the runtime with something like:
```
results = sess->run(...)
```
and since we want to see the results all computation has to be completed before returning.

Thanks for your reply. My concern is, GPU evaluations are asynchronous and GPU execution might not be completed when time is measured. Pytorch has something called torch.cuda.synchronize() to use before stopping the time measurement. Anything thing similar used for pytorch and tensorflow in mlperf while measuring time on gpu?

Since loadgen is used for reporting the final timing where the final result needs to be returned to loadgen, the timing should be device agnostic. please re-open if needed.
Repository: mlcommons/inference
Issue: BERT: accuracy of quantized ONNX model?
Root cause: Documentation Deficiency

Running the quantized BERT model in the accuracy mode:
```
$ python3 run.py --backend=onnxruntime --scenario=Offline --accuracy --quantized
```
[returned](https://github.com/mlperf/inference/issues/592#issuecomment-622378484):  `f1=90.067` (99.11% of the reference), whereas the [table](https://github.com/mlperf/inference/tree/master/v0.7/language/bert#supported-models) states `f1_score=90.482%` (99.57% of the reference).

Can't assign to myself, but I will look into this issue.

Its all yours :)

With the [GPU-accelerated ONNX runtime](https://github.com/mlperf/inference/issues/592#issuecomment-622516250), the accuracy is slightly better (99.16% of the reference):
```bash
$ time python3 run.py --backend=onnxruntime --scenario=Offline --accuracy --quantized
...
{"exact_match": 82.69631031220435, "f1": 90.11218881462509}
Done!

real    16m8.353s
user    12m50.460s
sys     3m32.820s
```

That still doesn't look right. Will have to look into what happened

@nvpohanh Any thoughts?

Ah, sorry I didn't have chance to look into this. Not sure why it doesn't match our internal measurement.

Would it be fine to change the README to the number you got (since it's >99% anyway) for now, before I figure out what's going on? Thanks

That's OK. It would be good for someone else to rerun the accuracy calculation, so at least we know if we have a majority!

@guschmue please verify the accuracy is indeed 99.16%.  If that's the case, @nvpohanh please update the readme.

Thank you!

On cpu:
python3 run.py --backend=onnxruntime --scenario=Offline --accuracy --quantized 
{"exact_match": 82.44087038789026, "f1": 90.12056578204738}
Repository: mlcommons/inference
Issue: BERT: can converting examples to features be done offline?
Root cause: Unimplemented Feature Gap

It appears that converting examples to features happens on every run (even if it's a trivial test run with one sample) and takes around a minute on my workstation:
```
Loading BERT configs...
Loading PyTorch model...
Constructing SUT...
Creating tokenizer...
Reading examples...
WARNING:tensorflow:From /workspace/DeepLearningExamples/TensorFlow/LanguageModeling/BERT/utils/create_squad_data.py:154: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.

Converting examples to features...
Constructing QSL...
Finished constructing QSL.
Finished constructing SUT.
Running Loadgen test...
```

I'm wondering if this part (or some part of this part) can be done offline:
```
Creating tokenizer...
Reading examples...
WARNING:tensorflow:From /workspace/DeepLearningExamples/TensorFlow/LanguageModeling/BERT/utils/create_squad_data.py:154: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.

Converting examples to features...
```
Repository: mlcommons/inference
Issue: I tried to run ./run_local.sh cpu tf ssd-mobilenet
Root cause: Environment Incompatibility

I tried to run ./run_local.sh cpu tf ssd-mobilenet

dongkwan@tux ~/바탕화면/inference-master/tmp/inference/v0.5/classification_and_detection $ ./run_local.sh cpu tf ssd-mobilenet

![image](https://user-images.githubusercontent.com/55476509/81140813-b8248b80-8fa5-11ea-96c8-bee36ce8c0d0.png)



I got this kind of error. Do you know how can I solve this error~?



reads as 'the .pb is not a valid pb', aka something is wrong with the model file.

is the problem still there?

assume this is resolved.