zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Does v0.7/language/bert/ support non-one batch size?
Root cause: Documentation Deficiency

In v0.7/language/bert/, you provide `tf_freeze_bert.py`. But it seems it only accepts 1 input at a time. I try to change `shape=(1, 384)` to `shape=(None, 1, 384)`, but I met some errors. So can I store a frozen model.pb to support non-one batch size? Thanks!

I'm sorry, it seems I can set `shape=(None, 384)` and then the frozen model.pb can support non-one batch size.

can I close this issue? @dengwxn ?
Repository: mlcommons/inference
Issue: submission checker: bert-99 == bert != bert-99.9
Root cause: Validation Gap

https://github.com/mlperf/inference/blob/master/tools/submission/submission-checker.py#L61Are we allowing people to submit the low accuracy variants using the benchmark name without the '-99' suffix?
That might cause some confusion.

I kept bert==bert-99 expecting most would use 'bert' but maybe better to force bert-99 or bert-99.9.


I force -99 and -99.9 now when applicable.
Repository: mlcommons/inference
Issue: Tensorflow DLRM Inference
Root cause: Unimplemented Feature Gap

Currently, I see `0.7/recommendation/dlrm/tf` is using some `tensorflow.python.tpu` dependencies like `tpu_embedding`. Can I run DLRM in a normal GPU? @davidmochen 

We only tried to run the code checked in here on TPU. We did not try running on GPU.

Are you going to support normal GPUs in the future? @christ1ne 

The TF reference for DLRM will not support GPU until next round, as far as I know. @dengwxn Are you planning to submit results to v0.7 using GPUs? PyTorch implementation is one alternative.

Do you know when is the next round? An approximate time will he helpful. @nvpohanh  Yes, I am planning to run inference on our system, but our system is currently focusing on tensorflow models.

I think next round is Q1 2021. Just curious, what's the reason you decided to go with TF instead of other inference frameworks, such as PyT, onnxruntime, TensorRT, etc.?

Thanks! Because our system only supports tensorflow now, and we are considering adding other frameworks.

We may look into providing GPU support for TF1 or TF2 implementation of this model by Q1 2021.

https://github.com/mlperf/inference/tree/master/recommendation/dlrm/tf  please check if this work-in-process is working for you. Please @mnaumovfb if you have further questions. 

@christ1ne @mnaumovfb  greate work for tf1 implementation. For some reason, I can't access google cloud to download checkpoint and data using gsutil. Can you share them using baidudisk or something else?Thanks for help. 





> Are you going to support normal GPUs in the future? @christ1ne

have you tried the implementation of tf in GPU?@dengwxn
Repository: mlcommons/inference
Issue: audit checker TEST01 verify_accuracy fast mode error
Root cause: Validation Gap

The following error message shown when trying to run TEST01 verify_accuracy script with fast mode on resnet result

python $LG_PATH/../v0.7/compliance/nvidia/TEST01/verify_accuracy.py -r ~/mlperf_log_accuracy.json -
t mlperf_log_accuracy.json --dtype byte --fastmode

Verifying accuracy. This might take a while...
Reading accuracy mode results...
Reading performance mode results...
Traceback (most recent call last):
  File "/mnt/ssd/aaronzhong/test/inference/loadgen/../v0.7/compliance/nvidia/TEST01/verify_accuracy.py", line 177, in <module>
    main()
  File "/mnt/ssd/aaronzhong/test/inference/loadgen/../v0.7/compliance/nvidia/TEST01/verify_accuracy.py", line 93, in main
    data = np.frombuffer(bytes.fromhex(sample['data']), dtype_map[args.dtype]) if py33 == True \
UnboundLocalError: local variable 'py33' referenced before assignment

@georgelyuan ^^^

Thanks, Aaron! Will fix this soon.
Have you been able to check if the script works without --fastmode ?

https://github.com/mlperf/inference/pull/690

please fix it, the normal mode is really really really really really really really really slow, though it's working.

Right, it was the only way I could think of to deal with UNet3D's log size. Maybe I should make --fastmode the default.

@georgelyuan any work needed here?

Yes, this has been fixed.
Repository: mlcommons/inference
Issue: [RNN-T] Quantization
Root cause: Unimplemented Feature Gap

Hello,

I’ve been looking into quantizing the various component parts of the RNN-T model and hitting a problem with the LSTMs which would hinder creating a quantized version of the benchmark.

For the first LSTM, when processing the logits within a sample, I’m seeing a steadily growing cell state. For a 15s sample (500 logits), the cell state begins at zero and grows to > +/-200.0.

```
  0 ['max:', '0.994', '', 'min:', '-0.999\n']
  1 ['max:', '1.973', '', 'min:', '-1.970\n']
  2 ['max:', '2.886', '', 'min:', '-2.799\n']
  3 ['max:', '3.807', '', 'min:', '-3.361\n']
  4 ['max:', '4.610', '', 'min:', '-3.572\n']
  ...
494 ['max:', '203.735', '', 'min:', '-268.766\n']
495 ['max:', '204.317', '', 'min:', '-268.591\n']
496 ['max:', '205.116', '', 'min:', '-268.906\n']
497 ['max:', '205.995', '', 'min:', '-269.664\n']
498 ['max:', '206.925', '', 'min:', '-270.510\n']
499 ['max:', '207.837', '', 'min:', '-271.319\n']
```

When quantizing this to i8, the broad range required for the last logit dictates the overall range required, but this is unsuitable for the first digits, and so the output of the LSTM is zeros. I’ve tried using non linear scaling (power of 2) which doesn’t solve the problem.

A workaround I’ve found is to rescale the cell state to +/-8.0 after processing each logit which gets results within 4% of the floating point equivalent, but this is a fudge rather than the solution.

One idea would be to retrain the model with this kind of rescaling in place to prevent the cell state from expanding. Or am I missing something and there is a better solution?


As @galv points out in the [README](https://github.com/dividiti/inference/blob/dvdt-rnnt/v0.7/speech_recognition/rnnt/README.md), no quantization is done. It sounds that we need quantization-aware training, even though the [original paper](https://arxiv.org/pdf/1811.06621.pdf) suggests that symmetric post-training quantization using TF tools works fine. Maybe there utterances were shorter and the error would not accumulate in this way?

Any suggestion for next step @galv?

We would probably need quantization-aware training for this to work, even though the [original RNN-T paper](https://arxiv.org/pdf/1811.06621.pdf) we based this on does not use it. Note that our work on MLCommons's The People's Speech Dataset has trained state-of-the-art-ish CTC models, but not RNN-T models, in tensorflow. So we are bringing up the infra to do that sort of thing, but it's not actively prioritized to do quantization-aware RNN-T training.

I am no expert in quantizing models, to clarify. It's possible that there is a correct solution that someone else who's used to this kind of thing can just point out within an hour.

@nvpohanh @DilipSequeira I didn't see any quantized submission from NVIDIA when I checked the spreadsheet a week ago.Can you confirm whether you had a quantized submission? If so, how did you deal with the above problem?

No, we didn't submit a quantized submission in v0.7.

please re-open if needed.
Repository: mlcommons/inference
Issue: Source of coco 2014 dataset change
Root cause: Documentation Deficiency

Just a friendly reminder that coco 2014 dataset using in the following is removed.
https://github.com/mlperf/inference/tree/master/edge/object_detection/ssd_resnet50

Can try to use this to get the data:
ck install package --tags=dataset,coco,2014

@koc4466677 Apologies for the late reply - I've only just spotted your message!

First of all, [SSD-ResNet50](https://github.com/mlperf/inference/tree/master/edge/object_detection/ssd_resnet50) has been excluded from MLPerf Inference v0.5. 

Second, [SSD-MobileNet](https://github.com/mlperf/inference/tree/master/edge/object_detection/ssd_mobilenet) is still relevant. I have recently updated its instructions (but it seems only after you opened this issue). In particular, I've provided [instructions](https://github.com/mlperf/inference/tree/master/edge/object_detection/ssd_mobilenet#install-the-coco-2017-validation-dataset-5000-images) to download the COCO 2017 validation set:
```
$ ck install package:dataset-coco-2017-val
```

However, the old command still works for me:
```
$ ck install package --tags=dataset,coco,2014
```
Do you have any issues with it?

@koc4466677 Wondering if you've had a chance to check this?

By the way, instead of:
```
$ ck install package:dataset-coco-2017-val
```
you can also use:
```
$ ck install package --tags=dataset,coco,2017
```
which is more stable. (Package names may change, while we try to avoid changing tags.)
Repository: mlcommons/inference
Issue: When I use the GPU run/inference/recommendation, the program USES the CPU
Root cause: Configuration Error

I ran this code：
./run_local.sh pytorch dlrm terabyte gpu --max-in-range=10000000
I got this：

python3 python/main.py --profile dlrm-terabyte-pytorch --config ../mlperf.conf --model dlrm --model-path /root/mlperf/.5/recommendation/model/dlrm_terabyte.pytorch --dataset terabyte --dataset-path /root/mlperf/mlperf_inference/v0.5/recomeo --output /root/mlperf/mlperf_inference/v0.5/recommendation/output/pytorch-gpu/dlrm --use-gpu --max-ind-range=10000000
INFO:main:Namespace(accuracy=False, backend='pytorch-native', cache=0, config='../mlperf.conf', count_queries=None, couna_sub_sample_rate=0.0, dataset='terabyte', dataset_path='/root/mlperf/mlperf_inference/v0.5/recommendation/fake_criteo',d_peak_performance=False, inputs=['continuous and categorical features'], max_batchsize=2048, max_ind_range=10000000, ma_samples_to_aggregate=None, min_samples_to_aggregate=None, mlperf_bin_loader=False, model='dlrm', model_path='/root/mlpe/v0.5/recommendation/model/dlrm_terabyte.pytorch', numpy_rand_seed=123, output='/root/mlperf/mlperf_inference/v0.5/recomtorch-gpu/dlrm', outputs=['probability'], profile='dlrm-terabyte-pytorch', samples_per_query=None, samples_to_aggregate=gregate_quantile_file=None, samples_to_aggregate_trace_file='dlrm_trace_of_aggregated_samples.txt', scenario='SingleStree, test_num_workers=0, threads=8, use_gpu=True)
Using CPU...
Does anybody know why？

@mnaumovfb 

This has been addressed here https://github.com/facebookresearch/dlrm/issues/121

sounds good. please re-open if needed.
Repository: mlcommons/inference
Issue: Using LoadGen library as a stand alone performance benchmark
Root cause: Documentation Deficiency

Hey!
I want to use the LoadGen library to benchmark my model performance on my current infrastructure. 
Is there any way to run the LoadGen library with a custom model and dataset while my model is deployed on a remote server?
If not, I will be glad if someone could give me a reference to a similar tool that can 😃. 

https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/ was intended to have plugable datasets and backends. You could implement a backend that makes the network request by creating something like backend_grpc.py that implements a simple [backend interface](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/backend.py)


Hi @raz-bn,

Let me add to @guschmue's answer. You can indeed extend the reference code to Classification and Detection benchmarks, or other tasks. However, it's typically framework-based (i.e. uses TensorFlow, PyTorch or ONNX), so may not be optimal.

Instead, you may wish to use vendor-specific inference engines e.g. TensorRT from NVIDIA, OpenVINO from Intel and ArmNN from Arm. For this, I'd recommend to look at past submissions (only [v0.5](https://github.com/mlperf/inference_results_v0.5/), at the moment). Still, you may find them hard to reproduce and customize for new datasets, workloads and targets.

We (at [dividiti](http://dividiti.com) and our colleagues from the [cTuning foundation](http://ctuning.org/)) have been working on the open-source [Collective Knowledge](http://cknowledge.org/) technology for several years to enable automated, customizable, portable and reproducible workflows for benchmarking, optimization and co-design. We can easily combine reusable ML, software and hardware components to evaluate complete ML/SW/HW stacks.

We used the CK technology to a great effect for [MLPerf Inference v0.5](https://dividiti.blogspot.com/2019/11/dividiti-accelerate-omni-benchmarking-for-mlperf.html), where we submitted nearly 3 times more results than other dozen submitters combined, and will be using again for v0.7.

For example, we used [ArmNN](https://github.com/ARM-software/armnn-mlperf) and TensorFlow Lite to benchmark MobileNets-v1/v2 on several Arm CPU/GPU systems. Since then, we have also automated [OpenVINO](https://hub.docker.com/r/ctuning/mlperf-inference-v0.5.openvino) and [TensorRT](https://github.com/ctuning/ck-mlperf/tree/master/docker/image-classification-tensorrt-loadgen-py.tensorrt-6) based submissions.

Hope this helps.

https://github.com/mlperf/inference/tree/master/loadgen/benchmark
Repository: mlcommons/inference
Issue: submission checker: CAS error is an accepted error
Root cause: Validation Gap

check https://github.com/mlperf/inference/issues/468

currently submission checker still report it as an error.

seems the code does include the CAS error to the ignore list. Probably the code checking whether the error is CAS or not is not working. 

There is some code that skips over this error but I deactivated that code for v0.7 assuming it was fixed for v0.7.
Guess I was to optimistic and will re-enable this code.
Repository: mlcommons/inference
Issue: BERT calibration dataset proposal
Root cause: Unimplemented Feature Gap

We'd like to propose a calibration dataset since there is none in the reference.  

Here is the 100 randomly selected samples from dev-1.1.json file, which contains 10570 samples in total.

4547  9556  2033  2181 10376  6995  5480  2096  4211   458  8812  4365           
  6025  9277  6287  5469  3426  7483  8462  6584  8969  4659  5045  2875                            
  6314  4506  6466 10086  9957  3453  7249  3947  3855  1500   491  8767                            
  598  7199  2581  2076   735  1418  8000  1400  9433  7771  5483  1574                            
  5430  5166  3718  3232 10247  1417  2018  4838  6570  4923  4400 10121                            
  6256   206   118  9143  9911   788  3320  7383  1250  5784  1436  9861                             
  4930  5387 10414  7388  7624  7641  4775   911  1171  6714   381  3627                            
  8220  7779  5264  1185  6846  9735  4947  2857  1965  3222  2764  7509                            
  561  5789   797 10432

@DilipSequeira  @tjablin 

WG: approved
Repository: mlcommons/inference
Issue: submission checker: Feature request for more checks
Root cause: Unimplemented Feature Gap

Thanks for the great submission-checker!

It would be nice to further add the following checks to the submission checker:

1. Verification of latency constraints for Server & Multi-Stream Scenarios
2. Verification of test duration (minimum query count + 60s or minimum samples/query in the case of Offfline mode)
3. Verification of allowed Loadgen commit hash

I have a PR in works to add (1) and (2). (3) will be straightforward once the committee decides on the loadgen version for use.

Should we also check the min_query_count and min sample count requirements?

Ah, it's probably included in (2)?

Yes, please - more checks are very welcome !

> Should we also check the min_query_count and min sample count requirements?

Right, (2) covers that.

For (3), do we have a list of loadgen commit ids allowed for MLperfv0.7?

I don't think we have that yet.

I was thinking to take a r0.7 release branch once https://github.com/mlperf/inference/pull/702 is merged.
After that we'd expect very few changes.

closing for now since all the work seems complete
Repository: mlcommons/inference
Issue: "Invalid model for closed division" error by submission checker
Root cause: Validation Gap

@guschmue  We are hitting these invalid model names. Could you take a look why these model (benchmark) names are not allowed? Thanks 

```
ERROR:main:closed/NVIDIA/results/our_system/3d-unet-99 has a invalid model (3d-unet-99) for closed division
ERROR:main:closed/NVIDIA/results/our_system/ssd-resnet34 has a invalid model (ssd-resnet34) for closed division
ERROR:main:closed/NVIDIA/results/our_system/dlrm-99.9 has a invalid model (dlrm-99.9) for closed division
ERROR:main:closed/NVIDIA/results/our_system/resnet50 has a invalid model (resnet50) for closed division
ERROR:main:closed/NVIDIA/results/our_system/ssd-mobilenet has a invalid model (ssd-mobilenet) for closed division
```

real silly error - forgot a "," between model names.
added a fix to https://github.com/mlperf/inference/pull/702


@guschmue What about `ssd-mobilenet` `ssd-resnet34` `resnet50` vs `ssd-small` `ssd-large` `resnet`? The mlperf.conf uses the former set...

the official names have been resnet, ssd-small and ssd-large but we have mapping from the unofficial names to the official ones so either way is fine: 
https://github.com/mlperf/inference/blob/master/tools/submission/submission-checker.py#L123

And there is some additional mapping code in https://github.com/mlperf/inference/blob/master/tools/submission/submission-checker.py#L213 for folks in open that use different models for the ssd-* tasks.

but even with that mapping, it still does not allow `closed/NVIDIA/results/our_system/ssd-resnet34`, for example.

You need the fix for #702, I called the mapping func after the initial check.

Ah, got it! I will try that. Thanks


ok to close @nvpohanh ?

Yes, it worked.
Repository: mlcommons/inference
Issue: submission checker: Can it write to submission_checker_log.txt instead of compliance_checker_log.txt?
Root cause: Documentation Deficiency

Can the submission checker script write to submission_checker_log.txt instead of compliance_checker_log.txt? The latter file name is confusing because it refers to compliance checking, which is different.

CC: @guschmue 

submission-checker does not write to a log file.

I think this is more of a submission rule issue: https://github.com/mlperf/policies/blob/master/submission_rules.adoc#inference-1

Should we name it "compliance_checker_log.txt" or "submission_checker_log.txt"?

oh, I see. Yes, submission_checker_log.txt would be correct. Who is doing to PR?

working on that now...

PR: https://github.com/mlperf/policies/pull/55

super!

Unfortunately, we didn't get over this PR at the last meeting but we (dv/dt) will use the new name.

how come we didn't cover this in last WG meeting... 🤦 

@guschmue  @nvpohanh  shall we just remove compliance_checker_log.txt from the requirement since the file is expected to generate with the submission checker?

I don't think submission checker automatically saves its own output to a file. you need to do something like `python3 submission-checker.py > submission_checker_log.txt`.

@guschmue please either update the logger to log at "compliance_checker_log.txt" as shown in the submission rules, or remove that txt line item from the submission rules.
Repository: mlcommons/inference
Issue: submission checker: min_query_count for SingleStream scenario bug
Root cause: Validation Gap

There is a bug here in the submission checker.

The function `check_performance_dir` is responsible for parsing `mlperf_log_summary.txt`.

When it parses for the scenario, it uses: `rt[m.group(1).strip()] = m.group(2).strip()` which converts the string `Single Stream` to `Single`.

`Single` is not a valid scenario, and this key is invalid. However, there are portions of the code that make use of it.
For example: https://github.com/mlperf/inference/blob/master/tools/submission/submission-checker.py#L187

Eventually, `Single` is used as a key instead of `SingleStream` to this dictionary:
https://github.com/mlperf/inference/blob/master/tools/submission/submission-checker.py#L155

The dictionary lookup returns None, and an invalid comparison is made between an integer, and None for min_query_count.

An exception is thrown.

Here is my hack to bypass the issue. I think a proper PR would not use the key "Single" anywhere, so I am not creating a PR.

```
diff --git a/tools/submission/submission-checker.py b/tools/submission/submission-checker.py
index 702f02b..56ced23 100755
--- a/tools/submission/submission-checker.py
+++ b/tools/submission/submission-checker.py
@@ -440,7 +440,8 @@ def check_performance_dir(config, model, path):
                             fname, target_latency, rt['99.00 percentile latency (ns)'])

         # Check Minimum queries were issued to meet test duration
-        min_query_count = config.get_min_query_count(model, scenario)
+        scenario_hack = "SingleStream" if scenario == "Single" else scenario
+        min_query_count = config.get_min_query_count(model, scenario_hack)
         if int(rt['min_query_count']) < min_query_count:
             log.error("%s Required minimum Query Count not met by user config, Expected=%s,
                         fname, min_query_count, rt['min_query_count'])
```

We hit the same issue and checked in a fix (#720) into `r0.7` branch for this. Could you try with `r0.7` branch? Thanks

@nvpohanh the `r0.7` branch works for me. Should I exclusively be using the `r0.7` branch for the upcoming submission?

Yes, submitters should use `r0.7` branch for submission. @guschmue FYI

Thanks for your help @nvpohanh 

sounds like no longer a problem!
Repository: mlcommons/inference
Issue: [DLRM] v0.5/recommendation/python still present in the tree

Most importantly, has the bug [this commit](https://github.com/mlperf/inference/commit/cf0d0b3e71d0889bf24e3e75c8d22c9895e808f0) was intended to fix made its way to `recommendation/dlrm`?

@mnaumovfb 

I think the backend_onnxruntime.py did not move to the new location.

I'll have a PR to fix it today. 

Here is the PR https://github.com/mlperf/inference/pull/728 that fixes this issue and enables support for variable batch size in ONNX.

OK to close @psyhtest ?

Sure, thanks!
Repository: mlcommons/inference
Issue: Problem with Loadgen installation for v0.7
Root cause: Configuration Error

When running `cd loadgen CFLAGS="-std=c++14" python setup.py develop` from the **inference** repo, **r0.7** branch.
I get:
```
Installed /Users/sefi/PycharmProjects/inference/loadgen
Processing dependencies for mlperf-loadgen==0.5a0
Finished processing dependencies for mlperf-loadgen==0.5a0

```
Is this OK or must I get verification messages for 0.7 loadgen?

It is not going to break anything, but yes - we should have changed the version number.