zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Using a Test Setting to Specify Latency Target
Root cause: Architectural Constraint

Currently, there is a hardcoded setting in LoadGen that specifies that the latency target is 90% (line 694 of [loadgen.cc](https://github.com/mlperf/inference/blob/master/loadgen/loadgen.cc)). 

`PercentileEntry latency_target{.90};`

There is a comment in the line above with a TODO that says to make "0.90" a test setting, which would be very helpful for testing across multiple benchmarks with different latency targets. I am filing this issue to follow up on the status of this fix. 

Also, just for convenience, can we add 0.97 as one of the latency percentiles (found at line 695 of [loadgen.cc](https://github.com/mlperf/inference/blob/master/loadgen/loadgen.cc)) that LoadGen reports about in the "additional stats" section? It would be helpful to automatically report that information, considering that the GNMT target percentile is supposed to be 97%. 

Folks from Alibaba have signed up to implement the TestSetting in this [spreadsheet](https://docs.google.com/spreadsheets/d/1RA-2QC7aXmYRWTWnuRR7fnwBhKdiCVUPkK7NDI8_oaE/edit?usp=sharing).

In the mean time, adding .97 would be great. @v4njur : Do you mind putting together a quick PR for that?


Ok sounds good. I have created PR #321 for adding 0.97 to the list of latency percentiles. Thanks!

@briandersn Would you like me to wait to close this issue until Alibaba is able to add the latency target as a test setting?

Thanks for the patch. Yeah, let's wait for Alibaba to upload their fix.

@frank-wei issued PR #326 , which has landed. We can close this now.

Thanks Frank!
Repository: mlcommons/inference
Issue: no result summary in mlperf_log_summary.txt
Root cause: Configuration Error

After running the python demo scripts, in the mlperf_log_summary.txt, single-stream doesn't show any result summary while the other three scenarios show the results.

@guschmue and @christ1ne you might have some thoughts on this?

I think this is because the demo for single stream uses the AccuracyOnly mode. It's supposed to be PerformanceOnly.

True, I have that too.
I guess the rational is that for accuracy the numbers would be off a little so we don't log them at all ?

@briandersn  I have fixed the test mode there in single stream demo. Please review at #354.

Thanks for addressing this, Frank. Closing this one.
Repository: mlcommons/inference
Issue: Segmentation fault   :  run_local.sh with python3.6
Root cause: Environment Incompatibility

~/mlperf_inference/v0.5/classification_and_detection$ ./run_local.sh tf resnet50 cpu
python3.6: Relink `/lib/x86_64-linux-gnu/libsystemd.so.0' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
python3.6: Relink `/lib/x86_64-linux-gnu/libudev.so.1' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
./run_local.sh: line 14: 18075 Segmentation fault      (core dumped) python3.6 python/main.py --profile $profile $common_opt --model $model_path $dataset --output $OUTPUT_DIR/results.json $EXTRA_OPS $@

________________________________________
$ catchsegv ./run_local.sh tf resnet50 cpu
python3.6: Relink `/lib/x86_64-linux-gnu/libsystemd.so.0' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
python3.6: Relink `/lib/x86_64-linux-gnu/libudev.so.1' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
./run_local.sh: line 14: 27365 Segmentation fault      (core dumped) python3.6 python/main.py --profile $profile $common_opt --model $model_path $dataset --output $OUTPUT_DIR/results.json $EXTRA_OPS $@
*** Segmentation fault
Register dump:

 RAX: 00007f7a602ced70   RBX: 000000000000000e   RCX: 00007f7a602cedd0
 RDX: 00000000018d5750   RSI: 00007ffdcf295318   RDI: 000000000000000a
 RBP: 000000000000000a   R8 : 00007f7b0b56fd80   R9 : 0000000000000000
 R10: 0000000000000000   R11: 0000000000000000   R12: 00007ffdcf295318
 R13: 00000000018d5750   R14: 00007f7a602ced68   R15: 0000000001d02e60
 RSP: 00007ffdcf28e1e8

 RIP: 0000000000002360   EFLAGS: 00010246

 CS: 0033   FS: 0000   GS: 0000

 Trap: 0000000e   Error: 00000014   OldMask: 00000000   CR2: 00002360

 FPUCW: 0000037f   FPUSW: 00000420   TAG: 00007f7b
 RIP: 08dfab18   RDP: 00000000

 ST(0) ffff 8000000000000000   ST(1) 0000 0000000000000000
 ST(2) 0000 0000000000000000   ST(3) ffff f800000000000000
 ST(4) ffff 81ceb32c4b43fcf5   ST(5) ffff 81ceb32c4b43fcf5
 ST(6) ffff ffffffffe219652c   ST(7) 81ce 81ceb32c4b43fcf5
 mxcsr: 1fa0
 XMM0:  000000000000000000000000ffffffff XMM1:  000000000000000000000000ffffffff
 XMM2:  000000000000000000000000ffffffff XMM3:  000000000000000000000000ffffffff
 XMM4:  000000000000000000000000ffffffff XMM5:  000000000000000000000000ffffffff
 XMM6:  000000000000000000000000ffffffff XMM7:  000000000000000000000000ffffffff
 XMM8:  000000000000000000000000ffffffff XMM9:  000000000000000000000000ffffffff
 XMM10: 000000000000000000000000ffffffff XMM11: 000000000000000000000000ffffffff
 XMM12: 000000000000000000000000ffffffff XMM13: 000000000000000000000000ffffffff
 XMM14: 000000000000000000000000ffffffff XMM15: 000000000000000000000000ffffffff

Backtrace:
<Empty nothing listed>
__________________
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic



Maybe related to 
https://github.com/tensorflow/tensorflow/issues/19375
Repository: mlcommons/inference
Issue: What's the difference with batchsize and samples/query

Hi all,
At first, I think the batchsize is the same as samples/query.
However, When I am doing the test with **offline**  scenario, I change the paremeters of batchsize but the output samples/query is always 110
```
samples_per_query : 110
```

So My understands is that in one query, the input samples is 110. However, it will be divided in to several batches for inference.
Is my understand right?

What scenario are you running?

> What scenario are you running?

It's offline scenario.

For the offline scenario, there is exactly one query containing all required samples. It is up to the system under test to divide the query into multiple batches based on the particulars of the system.

@tjablin Thanks for the explanation.
Repository: mlcommons/inference
Issue: Running GNMT Multi Stream Scenario with INVALID result 
Root cause: Configuration Error

When I am trying the reference implementation of [GNMT](https://github.com/mlperf/inference/tree/master/v0.5/translation/gnmt/tensorflow),
```
python loadgen_gnmt.py --store_translation --scenario MultiStream
```
 I get INVALID result,
```
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Multi Stream
Mode     : Performance
Samples per query : 4
Result is : INVALID
  Performance constraints satisfied : NO
  Min duration satisfied : Yes
  Min queries satisfied : Yes
Recommendations:
 * Reduce samples per query to improve latency.
```
The Recommendations tells me to reduce the samples per query.
However, the script didn't provide the argument to set the parameter Samples per query

Follow the Recommendations to reduce the **samples per query** even to 1, I still get the same invalid results.
```
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Multi Stream
Mode     : Performance
Samples per query : 1
Result is : INVALID
  Performance constraints satisfied : NO
  Min duration satisfied : Yes
  Min queries satisfied : Yes
Recommendations:
 * Reduce samples per query to improve latency.
```

I do some debug and find out this code snippet https://github.com/mlperf/inference/blob/fb4fae1dc387da21de37ea97d0d708cb014f95dd/loadgen/loadgen.cc#L821-L829

The variable _settings.target_latency.count()_ is always 100000000 which is smaller than the 90% latency results I get in my SUT.
Is it because we didn't set a appropriate _multi_stream_target_latency_ns_ **target** when we start the loadgen_gnmt.py?

> Is it because we didn't set a appropriate multi_stream_target_latency_ns target when we start the loadgen_gnmt.py?

This might be the issue.

There could also be a bug in the loadgen. One of the open tasks is also to audit the multi-stream logic. The manual and automatic test coverage here is lacking. See "Verify multistream results bookeeping" in the [task spreadsheet](https://docs.google.com/spreadsheets/d/1RA-2QC7aXmYRWTWnuRR7fnwBhKdiCVUPkK7NDI8_oaE/edit?usp=sharing).
Repository: mlcommons/inference
Issue: Porting the MLPerf models to TensorFlow 2.0
Root cause: Unimplemented Feature Gap

We'd like to know whether you have any plans about porting the mlperf models to TensorFlow 2.0.
Please provide us more information if possible.

I don't have any plans to do it personally. Volunteers welcome.

I believe this is something that individual submitters are welcome to do.

I belief there is a high probability that tf-2.0 can deal with the graph of the mlperf models but have not tried. Teaching https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/backend_tf.py to run under tf-2.0 might be a good start.
Repository: mlcommons/inference
Issue: GNMT inference checkpoint missing bias weights
Root cause: Configuration Error

Using TensorFlow 1.14:

```
$ ./download_dataset.sh
$ python run_task.py --run=accuracy
```

This results in the error:

```
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value dynamic_seq2seq/encoder/rnn/basic_lstm_cell/bias
	 [[node dynamic_seq2seq/encoder/rnn/basic_lstm_cell/bias/read (defined at /Projects/mlperf/inference/v0.5/translation/gnmt/tensorflow/nmt/gnmt_model.py:127) ]]
```

The checkpoint provides weights for multi_rnn_cell (as expected for an attention model), not basic_rnn_cell. The hparms correctly specify an attention model, so I'm not sure what's going wrong here.


My bad, another run had corrupted the hparams.
Repository: mlcommons/inference
Issue: Some imagenet networks trained with a labels' offset. I fix this problem. It's pretty simpy to use. Just set --offset flag to True value. I guess you know what kind of architectures must have label offset
Root cause: Configuration Error

https://pastebin.com/APcetBa3

Look at 297-301 and 210-211 lines to review the changes

As this appears to be a nonstandard code submittal, have you signed the CLA?

How can I sign it?


You might want to consider submitting this as a pull request instead, and the bot will give you instructions.
Repository: mlcommons/inference
Issue: loadgen hardening - unique sample issue mode
Root cause: Unimplemented Feature Gap

The purpose of this mode is to facilitate auditors' ability to check that the SUT abides by the rules of MLPerf Inference.
In accuracy mode, the loadable set is shuffled and issued such that the SUT sees only unique samples whereas in performance mode, queries are generated from samples randomly picked from the loadable set with random uniform probability.
Unique sample issue is a mode in performance mode where only unique shuffled samples are issued to the SUT (ala what is done in accuracy mode) and test terminates after all unique samples have been sent once. In conjunction with the ability to modify the performance sample count, this will enable tests which attempt to detect response-caching.

This has landed.
Repository: mlcommons/inference
Issue: loadgen hardening - TestSettings config file parsing
Root cause: Unimplemented Feature Gap

To better facilitate auditing, loadgen should be able to accept a file path string or pointer to a file from which it can parse the necessary TestSettings parameters. This file can then be used by auditors to configure validation experiments to ensure submissions abide by MLPerf Inference rules, without submitters having to pass parameters through their main functions.
There are no strict requirements regarding file format--standard json or csv format should be fine.

Looks like this is being tracked already and covered by Siva (Guenther?) in https://docs.google.com/spreadsheets/d/1RA-2QC7aXmYRWTWnuRR7fnwBhKdiCVUPkK7NDI8_oaE/edit#gid=0 as a P1 task. With how important this feature is for auditing, I think this should deserve being bumped to P0.

https://github.com/mlperf/inference/pull/384 has the change.
We updated the reference implementation to make use of it. 
Going to send out some email how to use it.
Repository: mlcommons/inference
Issue: performance sample count should be parameter in loadgen
Root cause: Architectural Constraint

Hello, currently performance_sample_count is passed as a parameter to the QSL constructor. Based on conversation with Dilip, this should be a parameter in loadgen instead. I assume this should be a TestSettings parameter that loadgen can then pass to QSL through GenerateLoadableSets?

This makes sense. The source of the samples shouldn't care about the a test or system constraint like the performance_sample_count.

Initially this sounds like just code cleanup, but if you think this will help with auditing this might make sense to prioritize. Is that the idea?

Yes, we could leverage this for auditing. 
If you don't have anyone in mind, we can take care of this.

I do not have anyone in mind, so I'll put NVIDIA down. Thank you!

Hi @georgelyuan : You mentioned Ritika would be picking this up. Could you cc them to this issue?

cc += @guschmue, who is looking at adding config file support.

Given that we are within a month of the submission deadline: I wonder if, instead of refactoring QSL's performance_sample_count, we add a TestSetting::performance_sample_count _override that overrides the QSL version if it is not zero.

It'll be a little awkward, but this approach will be backwards compatible while still supporting the auditing requirements.

@nv-rborkar

Yes, i can add a TestSetting as described above which is zero by default and overrides sample_count when non-zero.

Thanks @nv-rborkar for getting this in. Closing the issue.
Repository: mlcommons/inference
Issue: loadgen hardening - accuracy logging in performance mode
Root cause: Unimplemented Feature Gap

Today, there is no accuracy logging in performance mode as query response metadata is ignored.
To validate that the SUT provides legitimate responses in performance mode, accuracy logging should still be performed on a per-sample basis with some probability P, where P can be configured by the submitter/auditor. For submissions, P can be set to zero. During the audit process, auditors may choose to set P to a non-zero value to trigger accuracy logging. If logging causes significant overhead, the auditor may use a range of P and observe performance approaching 'no-logging' performance as P approaches zero. Selection of samples for logging with probability P will be determined by another RNG seed that can be initialized in TestSettings.

Hi George. The feature as proposed sounds very flexible.

I know we talked about just turning on logging 100%, but just doing what you've proposed allows us to defer that decision or choose something anywhere in between. We probably wont know what's best until more submitters are online.

Would you or someone else an nvidia be willing to own the implementation? 

Regarding the RNG, it might make sense to add a log_result flag to SampleMetadata and set it before the test starts.
 

Hi Brian,
Glad you're on board.
Yes, our team is offering to own this implementation.

@georgelyuan : Can we close this one now?

Yes, thanks for your support.
Repository: mlcommons/inference
Issue: Sanitize warmup for GNMT
Root cause: Architectural Constraint

This follows from a PR #322 (which I closed without merging). Currently, GNMT's reference implementation gets warmed up on the LoadSamplesToRam call ([See L236](https://github.com/mlperf/inference/blob/master/v0.5/translation/gnmt/tensorflow/loadgen_gnmt.py#L236)). Instead, we should run warmup right after LoadSamplesToRam is called and right before querries get sent to SUT. Unfortunately, both of these calls happen from within StartTest. This could be solved if we had a loadgen::warmup hook, so loadgen could make that call in between samples being loaded and querries being sent. 

There is an open issue to add this kind of hook, but is low priority right now.
Repository: mlcommons/inference
Issue: Ending Early error due to queries_issued in server scenario
Root cause: Validation Gap

In server scenario, there's an error looks weird as below,

"pid": 319114, "tid": 140443789978112, "ts": 955579ns : POWER_BEGIN: "mode" : Performance, "time" : 08-28-2019 22:00:01.671
"pid": 319114, "tid": 140443789978112, "ts": 488381497ns : **Ending naturally: Minimum query count and test duration met.**
"pid": 319114, "tid": 140443789978112, "ts": 488435298ns : POWER_END: "mode" : Performance, "time" : 08-28-2019 22:00:02.158
"pid": 319114, "tid": 140443789978112, "ts": 488436703ns : ERROR : Ending early: Ran out of generated queries to issue **before the minimum query count and test duration were reached**.
"pid": 319114, "tid": 140443789978112, "ts": 488436703ns : Please update the relevant expected latency or target qps in the TestSettings so they are more accurate.

The 2 bold msg seems conflict each other, the code in loadgen generatea the error is as following,
  if (scenario != TestScenario::Offline && mode == TestMode::PerformanceOnly &&
      **queries_issued >= queries.size()**) {
    LogDetail([](AsyncDetail& detail) {
      detail.Error(
          "Ending early: Ran out of generated queries to issue before the "
          "minimum query count and test duration were reached.");
      detail(
          "Please update the relevant expected latency or target qps in the "
          "TestSettings so they are more accurate.");
    });
  }

Shall it be queries_issued **>** queries.size() ?


> Shall it be queries_issued > queries.size() ?

Hi @xgu3. Thanks for pointing this issue out. We should definitely fix it.

The proposed fix isn't quite right since it will never evaluate to true (we won't issue more queries than we've generated). Instead, we should set a flag when exiting naturally and check for !exited_naturally.

Do you have cycles to put together a PR?

Yes, I’ll try it out.



I ended up fixing this as part of another fix since it was related: https://github.com/mlperf/inference/pull/398

@xgu3 : Can you verify that PR addresses the error you see as well?

@briandersn Just verified and the error is gone. Thank you very much!

Closing.
Repository: mlcommons/inference
Issue: extra qualification error <PerformanceSummary>
Root cause: Validation Gap

try building loadgen as a python module, but encounter this error
loadgen.cc:748:3: error: extra qualification ‘mlperf::loadgen::PerformanceSummary::’ on member ‘PerformanceSummary’ [-fpermissive]
PerformanceSummary::PerformanceSummary(

ups, let me fix it.

I too am getting the same error.

Fixed in PR
Repository: mlcommons/inference
Issue: nice to add --recursive flag when cloning repository
Root cause: Documentation Deficiency

Got into build issue where submodules under third_party were not cloned when I specified 
git clone https://github.com/mlperf/inference
instead I had to use 
git clone --recursive https://github.com/mlperf/inference

@rrawther Thanks for the suggestion. This actually used to be in LoadGen's [README.md](https://github.com/mlperf/inference/blob/master/loadgen/README.md), but is now in [README_BUILD.md](https://github.com/mlperf/inference/blob/master/loadgen/README_BUILD.md).

/cc @briandersn

@rrawther : Which directions were you following?

We should consider updating all clone commands to include --recurse-submodules if it's causing headaches.

Some builds targets don't require the submodules though. If initial setup and authentication for the third_party repositories is a hurdle, we could maybe add a note under the commands where it isn't strictly necessary.

I raised the issue because I got pybind compile error when building loadgen which is a requirement for all. Later on, it was found the third_party was not cloned. I am not sure which builds exactly require third_party. Since it is part of repository, may be it is better to have it cloned by default.

I strongly agree with @rrawther . Yet least we should include it in Dockerfile when running with docker option....
Repository: mlcommons/inference
Issue: Multi GPU processing
Root cause: Architectural Constraint

I am currently trying to benchmark the "classification_and_detection" portion of the MLPerf Inference test suite on various multi-GPU environments (using the code as-is) but regardless of the settings I pass in (threads, batch size, scenarios) the processing is mainly performed on one GPU, whereas tensorflow benchmarks perform the work on all the GPUs. The screencap shows the nvidia-smi output at the same time a test is being run on a 2-GPU system.

Is it actually possible to properly run these benchmarks on multiple GPUs? (again, using the code as-is)

![RunOn1GPU_ssh](https://user-images.githubusercontent.com/54842128/64164702-e0d01080-ce6d-11e9-8897-c09cc1d53c1f.PNG)


Hm, the way the app is right now: it sends all requests into the same session and I'd expect that the session is bound to 1 device. There is no device binding in the models either.

The easiest way to support multiple devices is to create a session for each device and schedule the requests RR into those sessions.
You could add this here: https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/backend_tf.py.

@guschmue Are we allowed to use multi-GPU in the submission?

There's no rule against it, as far as I know.

+1
Repository: mlcommons/inference
Issue: SSD-ResNet34 preprocessing bug - reducing validation set to 4952
Root cause: Algorithmic Error

SSD-R34 setting originally came from MLPerf training, there (also in the original code i.e., pre- loadgen) we remove from the validation set 48 images which don’t include objects from the 80 categories (out of the original 91). Please see MLPerf training git issue #215. During the integration with loadgen, this part disappeared thus the current accuracy of the model is less than 20 (19.98). I opened a PR (#382 ) that fixes this issue and reduce the validation set back to 4952 images. 

fixed with https://github.com/mlperf/inference/pull/382
Repository: mlcommons/inference
Issue: Accuracy mode repeats queries way too many times for GNMT
Root cause: Algorithmic Error

In PR #386 , @nvmbreughe mentions that QSL sizes 3003, 1001 sends queries of 3003 samples over and over again. Whereas 3003, 100 works as expected with queries of 100 samples not repeated.

Something fishy is going on.

Marking this one urgent until we determine it is fixed or doesn't affect submission use cases.

Fixed.
Repository: mlcommons/inference
Issue: SSD-ResNet34 label mapping is done is finalize()
Root cause: Architectural Constraint

Unlike SSD-MobileNet, SSD-ResNet34 predicts the index of the label instead of the label itself, so an additional label mapping is needed (called `use_inv_map` in [coco.py](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/coco.py)). However, currently this additional mapping is done in the `finalize()` and is not part of the performance timing.

Depending on whether this mapping should be timed, we should either:
- Move `use_inv_map` to `__call__()`
OR 
- Add an option to accuracy script [accuracy-coco.py](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-coco.py) to do the mapping in the accuracy script instead.


Added an option to accuracy script accuracy-coco.py to do the mapping in the accuracy script in #395. Closing.