zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Should qsl_idx be part of the SSD model output?
Root cause: Architectural Constraint

Currently, the reference SSD models add image id (actually `qsl_idx`) as part of the output in the post-processing step (see [coco.py#L158](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/coco.py#L158). However, LoadGen already keeps track of `qsl_idx` in the accuracy_summary log.

Does `qsl_idx` have to be part of the model output? Could we add an option to the accuracy script such that it relies on the `qsl_idx` from LoadGen logs instead of the first field of the model output at [accuracy-coco.py#L59](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-coco.py#L59), like what it is done in [accuracy-imagenet.py](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py#L43)?

I think we can change it in v0.6.

Using `qsl_idx` as image_idex in accuracy script in #395. Closing.
Repository: mlcommons/inference
Issue: pytorch native models are missing for resnet and mobilenet classification
Root cause: Unimplemented Feature Gap

Wanted to test inference performance on pytorch-native backend, but currently there is no way to import onnx to pytorch-native.  Instead, the current benchmarks are using caffe2 when specifying pytorch. 
Is there a way to get native pytorch models uploaded as well?

If you want native-python one could do something like this:
https://github.com/mlperf/inference/tree/master/others/edge/object_detection/ssd_mobilenet/pytorch
Repository: mlcommons/inference
Issue: Questions on trace generator
Root cause: Documentation Deficiency

Hi, I have a few questions of using the trace generator:

* Do we assume the inference engine links with the trace generator, or the engine can be a stand-alone process 
* The trace generator runs on the same machine as the model inference, is it a concern that this may skew the result? 
* The query library is saved in memory, with a huge dataset and huge model, is it a concern that this may skew the results? Is it better to load the queries on demand? https://github.com/mlperf/inference/blob/master/trace_generator/trace_generator.h#L75-L76
* Initial target_qps of 512 seems too small. Better specify it as an argument https://github.com/mlperf/inference/blob/master/trace_generator/trace_generator.h#L164
* startup time and warmup time are important. https://github.com/mlperf/inference/blob/master/trace_generator/trace_generator.h#L117-L118
* Accuracy mode is nice to have. https://github.com/mlperf/inference/blob/master/trace_generator/trace_generator.h#L78

Would be good to get some sense on the run-to-run variations. Usually how much is it?


@sf-wind  please update your questions again with the latest LoadGen + reference code again. I think most of the issues you raised have been addressed in the code. 

please read the rules doc regarding where LoadGen should be: 
https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#5-load-generator


close this issue due to lack of activity. please reopen if needed.
Repository: mlcommons/inference
Issue: Loadgen compilation issue for CLANG
Root cause: Environment Incompatibility

With the latest Loadgen code, the compilation is failing for Clang, which is the common compiler used for Android platform.

current loadgen can get compiled with gcc-5.4.0. But the current android-ndk-r17c (Android dev toolset kit) supports only GCC- 4.9.

I am copying the snippet of the error msg.

![image](https://user-images.githubusercontent.com/41215863/64807984-329e2800-d54b-11e9-800a-f6bf8d8413f2.png)


This should be fixed by #413. Please re-open this hasn't actually been fixed.
Repository: mlcommons/inference
Issue: Loadgen offline demo has segfault
Root cause: Algorithmic Error

Just saw this. Needs more investigation.

The demo doesn't crash as recently as 9/5:

b94769ff0467628f0537669e12552cd22e22f3fc : add support for loading mlperf rules parameters from config file (#384)

I don't have time to bisect atm.


Just kidding.

b073bb78b0b1d6b837321a4ad3edf8d56c67c363 Is the patch that causes the offline demo to segfault. Root cause still not determined.

I think it is only for the demo code so the submitters should not be affected. Is that correct?

I suspect submitters might hit this, but I'm not sure. Wasn't able to root cause before I left.

WG not going address it now.
Repository: mlcommons/inference
Issue: uploading images to memory
Root cause: Documentation Deficiency

- When running the multi-stream scenario the number of queries is 270336, but the number of 
  images in the imagenet data set is 50000 so how do the upload to memory should work

1. do i need to upload only 50000 images and loadgen will generate query index with wraparound?
do 2. i need to load 270336 images with replication?

- in the mlperf.conf file the offline scenario don't include min query count how many images should be loaded (assuming i can process 10000 images per second using resnet50?

found the right configuration
Repository: mlcommons/inference
Issue: !./run_local.sh tf resnet50 cpu --scenario Server  - keep getting in valid result
Root cause: Documentation Deficiency

![image](https://user-images.githubusercontent.com/55476509/65113762-5536b200-da1f-11e9-9d6f-d55249d7f4f5.png)

![image](https://user-images.githubusercontent.com/55476509/65113767-6089dd80-da1f-11e9-8610-0051d6ca5705.png)

I don't know why Scheduled samples per second is -nan and other invalidation

If this is the notebook example - it uses a handful of images which is not good enough to pass any checks - just meant to give people an idea. To make any of those checks pass you'd need to use the real dataset.


Dear gushmue what do you mean about "notebook example"? 

I meant GettingStarted.ipynb. The dataset and settings used in there won't get you VALID results. For that you'd need to use the real datasets decriebed in the readme and you'd need to remove the ```--time``` and ```--count``` options.
Repository: mlcommons/inference
Issue: I want to interpret/understand mlperf_log_summary
Root cause: Documentation Deficiency

![image](https://user-images.githubusercontent.com/55476509/65212193-30544480-dadc-11e9-8026-368c5631f6ca.png)

1) I want to understand what each value stands for. Where I can find material explain those result?
or can you guys let me know those result?


2) Is there more detail result since it is summary? 

This is saying it's taking you about 1.6 to 2.1 whole seconds to respond to each inference query - does that seem correct?

If so, the "QPS w/o loadgen overhead" calculation looks significantly bogus.

> min_query_count: 1024

Note that this is not enough work to get a valid submission (you need 24k queries completed for a valid 90th percentile).

> I want to understand what each value stands for.

The best resource I'm aware of is the LoadGen source and documentation?

Don't have time to investigate atm. The logic for QPS w/o..." is here:
https://github.com/mlperf/inference/blob/94e7f492ff936e02a626c49918fd8b17d076fe7e/loadgen/loadgen.cc#L1078

There may be a bug.

After another look, the log says it's taking .16 to 2.1 sesconds (not 1.6). So the QPS w/o loadgen overhead isn't necessarily bogus.

Documentation and source is the best place to understand each of the values:
Link to loadgen guide: https://mlperf.github.io/inference/loadgen/index.html

Whoops, I noticed that x10 difference and then failed in at counting..
Repository: mlcommons/inference
Issue: Setting up parameters for a test run
Root cause: Documentation Deficiency

- For offline case (when running SSD resnet34) what is the correct why to set parameters 
   assuming the following
1. number of images in data set 4952
2. to complete 60 seconds i need to run 15846 images
3. when calling ConstructQSL i send the following parameters:
        1. total_sample_count = 4952
        2. performance_sample_count = 4952
The upload function loads exactly 4952 sample to memory.

- for offline scenario when running for K samples do i need to call QuerySamplesComplete() only when i finish all K samples or i can call QuerySamplesComplete() multiple time each time a finish partial amount of samples. for example each time the benchmark finishes to process M samples where K = M*N where K,M,N are integers.

- similarly when running multi stream which batch 18 do i need to take care of the number of images loaded for example should it be divisible by 18?



> * for offline scenario when running for K samples do i need to call QuerySamplesComplete() only when i finish all K samples or i can call QuerySamplesComplete() multiple time each time a finish partial amount of samples.

yes, this is allowed

> * similarly when running multi stream which batch 18 do i need to take care of the number of images loaded for example should it be divisible by 18?

the loadgen will always send you exactly 18 every time

> similarly when running multi stream which batch 18 do i need to take care of the number of images loaded for example should it be divisible by 18?

Be careful. The loadgen is not aware of the SUT's native batch size, only multi_stream_samples_per_query which may or may not correspond to what you want.

a related question:  

Is the QSL size to be the minimum of total_sample_count? or minimum of performance_sample_count? @briandersn @tjablin 

@rachithayp 

WG: multi-stream always adds padding to make the the multiple of multi_stream_samples_per_query. It also pads differently in performance and the accuracy mode. 

total_sample_count: entire validation dataset count (GNMT:  performance mode ~3M, accuracy mode 3003) 
performance_sample_count: min QSL size specified in the rules.
 
@guschmue will update the mlperf.conf

I've created https://github.com/mlperf/inference/issues/478 about updating `mlperf.conf`.

so the performance_sample_count is now being set in the mlperf.conf with https://github.com/mlperf/inference/pull/489. But total_sample_count still needs to be passed from the application through QSL constructor right?
Repository: mlcommons/inference
Issue: What is the meaning of 'query' in MLperf context? 
Root cause: Documentation Deficiency

![image](https://user-images.githubusercontent.com/55476509/65298223-a8d30800-dba5-11e9-9f58-d00488cc9167.png)

There are a lot of queries in [ MLperf Result Summary ].

Max_async_queries, min and max_query_count , sample per query ....

I know query in data base which is asking data. 

But I don't know what is query in Deep learning and MLperf context.

Can you guys let me know what it is about ? 

 

It is explained in the rules:
https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc

Chris1ne Thank you!
Repository: mlcommons/inference
Issue: Multi-processes in C++,  TlsLogger::~TlsLogger throws exception
Root cause: Resource Mishandling

I invoke loadgen in c++ workspace. My main thread invokes the mlperf::c::ConstructSUT, mlperf::c::ConstructQSL, and mlperf::c::StartTest. And another created thread sends mlperf::c::QuerySamplesComplete to loadgen. Then, there are 3 TlsLogger instances. The 1st is for IOThread, the 2nd is for my main thread, and the 3rd  is for my created thread. When one of my TlsLogger instance is destroyed, an exception will raise. Can you explain the reason. Is it a bug?

If I stop the created thread using interrupting (boost thread), TlsLogger::~TlsLogger throws exception. But if I don't stop the created thread, the issue won't happen.

There's definitely a race condition in ~TlsLogger that we haven't fixed. The suggested workaround is to avoid "daemon" threads and join threads that interact with the loadgen to the main thread before exiting. Loading the loadgen as a python module throws even more complexity into the destruction order, if you happen to be doing that.

It is surprising that not stopping the thread fixes the crash for you though. Hopefully your workaroud is acceptable?

@zrg102030 : Is this blocking a submission for v0.5?

@christ1ne : Tentatively proposing the "WG future version" tag for this bug.
Repository: mlcommons/inference
Issue: Keep getting invalid Result in MultiStream(tried to change min Sample per queries)
Root cause: Configuration Error

![image](https://user-images.githubusercontent.com/55476509/65368412-b0fd7700-dc7b-11e9-97a3-e9170847d4d8.png)

![image](https://user-images.githubusercontent.com/55476509/65368417-bb1f7580-dc7b-11e9-940f-2cf9419dcab9.png)

I keep getting invalid result in Multi Stream. 

In recommendations, it said reduce sample per query to improve latency. 

But samples per query is already 1. 

Do you guys have any idea to fix this situation?



---------------
I tried to change target qps as 20 by --qps.

But seems not working

I tried to find target_qps in main.py 

1)Do you guys know other way to chance target qps? 

Only thing that I found is setting.server_target_qps.
![image](https://user-images.githubusercontent.com/55476509/65368441-fc178a00-dc7b-11e9-8cd8-2592c5419790.png)


![image](https://user-images.githubusercontent.com/55476509/65368431-eace7d80-dc7b-11e9-9df9-a20323baab97.png)

![image](https://user-images.githubusercontent.com/55476509/65368439-f3bf4f00-dc7b-11e9-8376-b34fc90f54c5.png)

-------------------Single Stream result for  resnet50 on onnxruntime or tensorflow. 

![image](https://user-images.githubusercontent.com/55476509/65370166-ccbf4800-dc90-11e9-86aa-7bafd2530f6a.png)

![image](https://user-images.githubusercontent.com/55476509/65370179-f5474200-dc90-11e9-995c-8b31279e7857.png)

Here is requested single Stream result
Repository: mlcommons/inference
Issue: Can I know what is the meaning of min,max duration in MLper Result ? 

![image](https://user-images.githubusercontent.com/55476509/65398675-8f270000-ddf3-11e9-8b43-3372b593db8e.png)

1)Can I know what is the meaning of min,max duration in MLper Result ? 
2) What is qsl_rng_seed ?
3)Sample_index_rng_seed?

If you know let me know~
If you know the URL to explain this stuff, let me have link please.

Thank you.
Dongkwan
Repository: mlcommons/inference
Issue: What is the meaning of these terms in MLperf context?

In MLperf Result Summary,

1) max_async_queries ?
2) min and max duration ?
3) min and max query count ?
4) qsl_rng_seed ? 
5) sample_index_rng_seed? 
6) schedule_rng_seed?


Guys  can you help me understand those term ~? 

Thank you, 
Dongkwan 

https://mlperf.github.io/inference/loadgen/structmlperf_1_1TestSettings.html

@kdk3776 : Closing this one. Let us know if any other clarifications are needed.
Repository: mlcommons/inference
Issue: Does the gap of QPS between offline and server mode make sense?

I observed that the offline and server QPS are not at same level, my understanding is that it's due to the poisson distribution by server scenario, is it correct?

And if yes, how much does the gap make sense?  30%? 40%, etc.

@briandersn @tjablin @guschmue @christ1ne @delock 


@tejasprabhune had a WIP patch in #437 to make the server scenario's effective QPS equal to the requested QPS, but he hasn't been able to compile and test it. The logic of the patch I think is mostly right though, if someone wants to pick that up.

Not sure if that's what this bug is referring to.

Thank you @briandersn for looking at this. The #437 is for scheduled QPS and requested QPS, Mine is about server and offline QPS, e.g.,
With same data set and settings, the offline QPS is 1000, and the server QPS within latency bound is 800, is this 20% gap a reasonable result?


Ah, I see. I'm not sure what a reasonable discrepancy is, but you are right. Server should be slower since some periods of time will have a burst of queries with a much higher "instantaneous QPS".

Unless there's a specific complaint that the discrepancy between server and offline doesn't make sense, I propose closing this for now.
Repository: mlcommons/inference
Issue: Weird server scenario latency increase in SubmissionRun
Root cause: Algorithmic Error

We met a weird case today, the server scenario works normally at PerformanceOnly mode, like below in log,
99.00 percentile latency (ns)   : 20000000

But when changing to SubmissionRun mode and keeping all other settings unchanged, the latency is longer as following, 
99.00 percentile latency (ns)   : 30000000

And no matter how lower the target qps, e.g. cut it half, the latency is still very high. As a comparison, in performance mode, cutting target qps half will make latency much shorter.

I tried changing below code in loadgen.cc to have performance run firstly, the latency becomes normal again (as same as PerformanceOnly mode).
From  
```
  switch (sanitized_settings.mode) {
  case TestMode::SubmissionRun:
      run_funcs.accuracy(sut, qsl, sanitized_settings, &sequence_gen); 
      run_funcs.performance(sut, qsl, sanitized_settings, &sequence_gen);
      break;
```
 To
```
  loadgen::SequenceGen sequence_gen;
  switch (sanitized_settings.mode) {
    case TestMode::SubmissionRun:
        run_funcs.performance(sut, qsl, sanitized_settings, &sequence_gen);
        run_funcs.accuracy(sut, qsl, sanitized_settings, &sequence_gen);      
        break;
```
Is there any clue how to solve it? Is this sth. loadgen need to handle? or does the client code miss sth.?

@briandersn @tjablin @guschmue @christ1ne 


we propose the submitter can choose to use submission run or perf + accuracy run with the same mlperf.conf

Agree with proposal. There are a number of things to fix in the loadgen to make submission mode usable. Allowing manual runs is a lot more flexible, especially for v0.5 where the submission requirements are still in flux.

If we aren't going to rely on the loadgen's submission mode, shall we close this?

OK. I'd like to let others know at the WG. We can close at the WG. 

ok to close as submissionrun is not used in v0.5
Repository: mlcommons/inference
Issue: ERROR:main:/mlperf.conf not found
Root cause: Configuration Error

with Executing the docker job resulting error, however config file exists in cwd as given in the script run_and_time.sh
Any guidance....  (System OS: ubuntu 18.04, docker is 16.04)

./run_and_time.sh tf resnet50 cpu
STARTING RUN AT 2019-09-23 06:00:00 PM
INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, config='../mlperf.conf', count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='~/CK-TOOLS/dataset-imagenet-ilsvrc2012-val', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=32, max_latency=None, model='/home/ubuntu/CK-TOOLS/resnet50_v1.pb', model_name='resnet50', output='/output/results.json', outputs=['ArgMax:0'], profile='resnet50-tf', qps=None, samples_per_query=None, scenario='SingleStream', threads=72, time=None)
INFO:imagenet:loaded 50000 images, cache=0, took=3.3sec
ERROR:main:/mlperf.conf not found


There is a ```--config mlperf.conf``` missing here:
https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/run_and_time.sh#L18

I'll send a PR.

Thx Guenther 
Modifying to following resolved that:
opts="--profile $profile $common_opt --model $model_path --dataset-path $DATA_DIR \
    --output $OUTPUT_DIR $extra_args $EXTRA_OPS --config mlperf.conf "
________________________________________________________________________________________


Resolved
Repository: mlcommons/inference
Issue: FileExistsError: [Errno 17] File exists: '/output/results.json'
Root cause: Validation Gap

with Docker test (run_and_time.sh) resulting error
./run_and_time.sh tf resnet50 [cpu/gpu]

Traceback (most recent call last):
  File "python/main.py", line 541, in <module>
    main()
  File "python/main.py", line 450, in main
    os.makedirs(output_dir, exist_ok=True)
  File "/opt/anaconda3/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/output/results.json'
ENDING RUN AT.......


I have not tested the docker images for a little.
I'll take a look and give it a test.

Thx Guenther.
Appears problem with run_helper.sh line 15
python3 python/main.py $opts --output /output/results.json
may require something like 
 python3 python/main.py $opts --output $OUTPUT_DIR/results.json

should be fixed in master.

Thx Guenther. Verified fixed

Verified Resolved
Repository: mlcommons/inference
Issue: Error with Cloud Inference - Sentiment Analysis Mxnet Implementation
Root cause: Validation Gap

Hi all, I am getting the below error when running Cloud Inference - Sentiment Analysis Mxnet Implementation benchmark, some help?

command: `python eval.py --model cnn --eval pretrained/seq2cnn_model --batch-size 1 --calc_accuracy`

```
test: 39.84 % 0.8604417670682731
test: 39.88 % 0.8603811434302908
test: 39.92 % 0.8603206412825651
test: 39.96 % 0.8603603603603603
Finshed running 10000 batches
test: 40.0 % 0.8604
Traceback (most recent call last):
  File "eval.py", line 104, in <module>
    main()
  File "eval.py", line 74, in main
    batch.data[0]=batch.data[0].as_in_context(xpu)
AttributeError: type object 'StopIteration' has no attribute 'data'
```


solved, it looks like there is an issue catching the StopIteration exception, the for loop should break when it reaches 10000 batches.
Repository: mlcommons/inference
Issue: Loadgen trace for offline and server are incomplete.
Root cause: Unimplemented Feature Gap

Server only traces the latency of each sample, without a way to easily correlate which sample that was.
Offline doesn't trace any per-sample information.
SingleStream and MultiStream trace the sequence ids, sample ids, and a bunch of other timestamps and durations.

For submission and auditing purposes, do folks want server and offline to trace everything that SingleStream and MultiStream do?

It would help folks be able to, for example, graph latency for a particular input over time to detect potential caching effects.

Thanks @georgelyuan. Closing this one.
Repository: mlcommons/inference
Issue: way to change channel order of ssd resnet34
Root cause: Documentation Deficiency

Hi I'm trying to convert ssd resnet 34 pb model to tflite model.
Since tflite only support for channels last data format, channels first ssd resnet 34 should be converted to channels last.
Is there a way to change pb model's channel order? 
Also any comments or instructions for converting ssd pb model to tflite will be helpful to me.

By "pb", do you mean the TensorFlow model? [We](http://dividiti.com) have [looked into this](https://github.com/mlperf/inference/issues/229) but then it ruled out as infeasible.

@psyhtest @deeplearningfromscratch , I update channel_last pb model on  https://zenodo.org/record/3708755#.XmszXqgzY2w. At the same time, I share pb2tflite tools at: https://github.com/lji72/inference/tree/master/others/cloud/single_stage_detector/tensorflow and have given pull request.

@lji72 
Thanks for your help in providing the latest the nhwc.pb

I have problems in converting the old .ckpt to the .pb. The issue page is [here](https://github.com/mlperf/inference/issues/566). 

Would you like to tell which export_graph.py do you use in exporting .ckpt to .pb.  The two different export_graph.py in [mlperf](https://github.com/mlperf/inference/blob/master/others/cloud/single_stage_detector/tensorflow/train/export_graph.py) and [lji72](https://github.com/lji72/inference/blob/tf_ssd_resent34_align_onnx/others/cloud/single_stage_detector/tensorflow/train/export_graph.py) are a bit confusing. 

Or maybe you will upload the latest .ckpt file as well.

Thanks a lot.

@ShenTonyM  update latest ckpt at: https://zenodo.org/record/3712664#.XnB1D8eP63A

@ShenTonyM  I used lji72 export_graph.py to export pb model 

Hi @lji72 , many thanks for your help. My problem is solved.