zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Clarifying DLRM aggregation and "samples per second"
Root cause: Documentation Deficiency

I'm looking for some clarity regarding the sample aggregation used in the DLRM benchmark and regarding the reported "samples per second" in the offline results.

- Specifically, what is the intent of the aggregation?
- And, how does the reported "samples per second" correlate to inferences per second?


Per the "Verifying aggregation trace" section in the README:

> In the reference implementation, each sample is mapped to 100-700 user-item pairs following the distribution specified by tools/dist_quantile.txt."


Looking at the code of the submissions it appears that they either use the sample trace file provided or randomly select sample counts from the quantile file.

The net effect appears to be that a sample as reported by "samples per second" really refers to an aggregate sample of between 100 and 700 individual samples with an average around 270.

Some specific questions to clarify:

- If the MLPerf DLRM offline benchmark reports 30K samples per second does that represent 30K aggregate samples per second?
- And, if so, does that translate to approximately 30K * 270 = 6.1M inferences per second?

Clarification on this matter would be greatly apppreciated.

I think it would be helpful to update the README with information regarding this issue.

Thank you in advance!


The motivation is that in real-world scenario, you usually run hundreds of item for a single user, so a "sample" should be hundreds of user-item pairs. 

> If the MLPerf DLRM offline benchmark reports 30K samples per second does that represent 30K aggregate samples per second?
> And, if so, does that translate to approximately 30K * 270 = 6.1M inferences per second?

This is correct. Please let us know how the docs can be improved to make it clearer. Also feel free to make the change and file a Pull Request if you would like to. Thanks.

Thank you @nvpohanh for the clarification. Is this implementation, sending multiple separate samples, intended to provide portability by not relying on the use of EmbeddingBag with multiple bags?

I don't think this decision had anything to do with EmbeddingBag. It's more about how in the real world the recommender users would run the network. Usually a user comes with hundreds of items, and you run inference on a per-user basis. That's why our "sample" has hundreds of user-item pairs.

@mnaumovfb can provide more insight on this topic. 

@nvpohanh summarized the motivation above.

please re-open if needed
Repository: mlcommons/inference
Issue: Question about RNN-T Decoding

Hi, I am quite confusing why here label -= 1, since the blank index is the max label, why it is minus 1 here? 
https://github.com/mlcommons/inference/blob/0377ccd7456cb3856f889e0d4895053f06014e8e/speech_recognition/rnnt/pytorch/decoders.py#L108
```python
    def _pred_step(self, label: int, hidden: Optional[Tuple[torch.Tensor, torch.Tensor]]) -> Tuple[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]:
        if label == self._SOS:
            return self._model.prediction(None, hidden)
        **if label > self._blank_id:
            label -= 1**
        label = torch.tensor([[label]], dtype=torch.int64)
        return self._model.prediction(label, hidden)
```

@ashwin @jdduke @codyaustun @badenh  Thanks a lot!

In our reference implementation, we set `_blank_id` to `len(rnnt_vocab) - 1` (see https://github.com/mlcommons/inference/blob/0377ccd7456cb3856f889e0d4895053f06014e8e/speech_recognition/rnnt/pytorch_SUT.py#L85 ) which is the last label, so `label > self._blank_id` will never be True (i.e. label will never minus 1). 

That part was to handle the case if you set `_blank_id` to something else. For example, if `_blank_id` is set to 0, then all labels need to minus 1 (1 -> 0, 2 -> 1, etc.) to get correct decoder embedding look up values. But this is not used in our reference implementation.

Got it. I forget there should be no blank_id in the LM decoder. So here is to fit "some situation" (RNN-T Encoder & Decoder have same vocab but Encoder just Pretrain with blank ID in somewhere middle), just make the two vocab list compatible. Does I understand right? @nvpohanh
Repository: mlcommons/inference
Issue: is this possible to inference on intel Arria 10 GX with openvino?
Root cause: Documentation Deficiency

is this possible to inference on intel Arria 10 GX with openvino. openvino doesn't have direct support on these model extensions like .pb, .onnx ..etc. one should convert models into IR representation.

The current repo is for reference implementation and we only tested the SW on CPU/GPU, not on FPGA.

However, in theory you can run the models with OpenVINO with the model converted. There are some references from Intel submissions. Here is one example: https://github.com/mlcommons/inference_results_v0.7/tree/master/closed/Intel/code/resnet/resnet-ov

Hope it helps!
 

please re-open if needed
Repository: mlcommons/inference
Issue: DLRM inference python script doesn't work
Root cause: Validation Gap

I am tryin to run MLPerf DLRM CPU Inference. when I inference dataset  Criteo Terabyte with 0.875 sub-sampling (debugging), it can't passed and criteo.py return an error:

AttributeError: module 'dlrm_data_pytorch' has no attribute 'collate_wrapper_criteo'

Does anyone know how to solve this problem?
Thanks !




How did you solve this problem?? I've got a same message.

> How did you solve this problem?? I've got a same message.

I try to git clone the dlrm codes with "mlperf" branch as follows:
       git clone -b mlperf https://github.com/facebookresearch/dlrm.git
and it seems that the problem above has been solved.  However, new error came as follows:
       File "/home/anaconda3/envs/mlperf/lib/python3.7/site-packages/torch/serialization.py", line 764, in _legacy_load
        magic_number = pickle_module.oad(f, **pickle_load_args)
_pickle.UnpicklingError: incalid load key, '\xb9'.




> > How did you solve this problem?? I've got a same message.
> 
> I try to git clone the dlrm codes with "mlperf" branch as follows:
> git clone -b mlperf https://github.com/facebookresearch/dlrm.git
> and it seems that the problem above has been solved. However, new error came as follows:
> File "/home/anaconda3/envs/mlperf/lib/python3.7/site-packages/torch/serialization.py", line 764, in _legacy_load
> magic_number = pickle_module.oad(f, **pickle_load_args)
> _pickle.UnpicklingError: incalid load key, '\xb9'.

I have solved this problem. You can have a try of gitting clone "mlperf" branch.
Repository: mlcommons/inference
Issue: NameError: name 'args' is not defined
Root cause: Validation Gap

I was trying to run the benchmark of dlrm without docker, but when I run the following:
    ./run_local.sh pytorch dlrm terabyte cpu --scenario Offline --max-ind-range=10000000 --data-sub-sample-rate=0.875 --samples- 
    to-aggregate-quantile-file=./tools/dist_quantile.txt --max-batchsize=2048 --mlperf-bin-loader
I got the following error message:
    Traceback (most recent call last):
         File "python/main.py", line 619, in <module>
              main()
         File "python/main.py", line 503, in main
              model = backend.load(args.model_path, inputs=args.inputs, outputs=args.outputs)
         File "/home/mlperf/inference/recommendation/dlrm/pytorch/python/backend_pytorch_native.py", line 58, in load
               md_threshold=None,
          File "/home/mlperf/training/recommendation/dlrm/dlrm_s_pytorch.py", line 372, in __init__
                if args.loss_function == "mse":
       NameError: name 'args' is not defined
Repository: mlcommons/inference
Issue: MLCommons CLA link 404

How can I sign up so that I can contribute?

```
MLCommons CLA bot:
Thank you for your submission, we really appreciate it. We ask that you sign our MLCommons CLA and be a member before we can accept your contribution. If you are interested in membership, please contact membership@mlcommons.org .
```

https://mlcommons.org/en/get-involved/



Thank you. Should the CLA Link go to [get-involved](https://mlcommons.org/en/get-involved/)?

Let me fix the link.
Repository: mlcommons/inference
Issue: 'inference/recommendation/dlrm/pytorch/docker_(g)(c)pu/kickstart.sh' tries to grab from 'github.com/mlperf' and '/v0.5/'  which doesn't exist
Root cause: External Dependency Breakage

```
remote: Repository not found.
fatal: repository 'https://github.com/mlperf/community.git/' not found
```

`-bash: cd: /mlperf/inference/v0.5/recommendation: No such file or directory`
Repository: mlcommons/inference
Issue: First time users don't have a checkpoint to load 'speech_recognition/rnnt'
Root cause: Documentation Deficiency

Commented out the loading of checkpoints for now in `https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch_SUT.py`

Have you read https://github.com/mlcommons/inference/tree/master/speech_recognition/rnnt#2-directions ?

This issue and your PR here https://github.com/mlcommons/inference/pull/797 suggest that you are not following the setup instructions.

Interesting, those are the exact instructions I just followed.
I still had to install a ton of python dependencies after following those instructions.

The tool is running now, I'll have to leave it on overnight to see if everything is ok. 

Did you follow those instructions without issue?

I am the author of this reference benchmark. We had successful submissions for v0.7 using this reference as a guideline.

Your PR suggested you installed pandas and tqdm manually, but those are both listed in conda environment already: https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/environment.yml So my guess is that you did not activate the conda environment appropriately.

Thank you. I'll fix the conda enviroment on my system.

`conda env create -f environment.yml`

Edit:
Successfully benchmarked after setting up the conda environment.
Repository: mlcommons/inference
Issue: ensure INT8 models are available as reference
Root cause: Unimplemented Feature Gap


Hi Christine.

Will this be for both TensorFlow and PyTorch (assuming scripted mode support for PyTorch is enabled).

Thanks,
Crefeda

let me bring this item to the WG meeting next week. Yes we always aspire to cover TF and PyTorch.

Dilip asks: Who owns this responsibility? Model owner? MLC?
Repository: mlcommons/inference
Issue: CK error: [repo] repository update likely failed - exit code 256
Root cause: Environment Incompatibility

Does anyone have this problem？
******************************************************************
Updating repo "ck-env" ...

  Local path: /root/CK/ck-env
  URL:        https://github.com/ctuning/ck-env

  cd /root/CK/ck-env
  git pull

fatal: No remote repository specified.  Please, specify either a URL or a
remote name from which new revisions should be fetched.


 WARNING: repository update likely failed OR IN A DIFFERENT BRANCH/CHECKOUT (git exit code: 256)

How do I solve this problem？？？

@huihuima Sorry about that. I've seen such things induced by Git problems.

Can you please share a few details:
1. How did you install CK? What's the output of:
```
$ ck version
$ git version
```

2. Can you successfully do:
```
$ git clone https://github.com/ctuning/ck-env /tmp/ck-env
$ cd /tmp/ck-env
$ git pull
```

3. If the above steps are successful, try:
```
$ ck rm repo:ck-env --all --force
$ ck pull repo:ck-env
```

I've heard about intermittent issues with GitHub for China. I'm wondering if that's the case here...

I just checked and it works fine for me in France. Never saw this problem before. 

1、$ ck version
      V1.9.7
     $ git version
      git version 2.7.4
2、$ git clone https://github.com/ctuning/ck-env /tmp/ck-env
   error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
It may be a problem with git. Thank you. I will try some solutions of git


@huihuima Thanks for confirming. It is indeed a Git protocol issue.
Repository: mlcommons/inference
Issue: adding more inference scenarios 
Root cause: Unimplemented Feature Gap

More scenarios: multi-tenancy, pipeline, burst throughput


WG: no major complaints on the current set. However, we should revisit after adding new workloads. This is a good question for the advisory boards. 
Multi-tenancy definition: concurrent server scenario like for mutiple models hosted in one model so that we have QPS targets for each model. A model here can be a different architectures, or different weights of the same model.
Repository: mlcommons/inference
Issue: server scenario update for RNN-T 
Root cause: Unimplemented Feature Gap

from @galv 

For automatic speech recognition tasks, I would like LoadGen to have a concept of receiving each sample as a stream, rather than all at once. This is necessary for having an accurate benchmark for SingleStream and Server scenarios for speech recognition. (The current version of LoadGen captures the Offline scenario for speech recognition quite well, meanwhile.) This would probably also necessitate changing the performance metrics as well, which is not great for cohesion with other tasks.
Repository: mlcommons/inference
Issue: Why does the build library link fail？？Why does the gn library link fail？？？
Root cause: Environment Incompatibility


@huihuima Please provide more details on what you are trying to achieve.

Like this link build @ ce8824f ，I can't get the content of this link. I want to compile loadgen and it will fail when linking to the library。

@briandersn 

LoadGen should be building without issues now.
Repository: mlcommons/inference
Issue: why submission checker doesn't check compliance tests for open division?
Root cause: Documentation Deficiency

Compliance tests are required by https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#compliance-testing
and is not included in the relaxed constraints in inference rules, see https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#412-relaxed-constraints-for-the-open-division
So do we need compliance tests for open division? If so, why not check compliance tests for open division in submission checker?

@Chujingjun Thanks for raising this. For v0.7, the WG decided not to request compliance tests for the Open Division. This decision, however, doesn't seem to have made it into the rules though.

WG: suggested text: 
Submitters must run the compliance tests **in the closed division of the Inference benchmarks** to verify that their submission achieves a basic level of compliance with a subset of the MLPerf rules.

see https://github.com/mlcommons/policies/pull/70
Repository: mlcommons/inference
Issue: LoadGen logging warning for the Offline scenario
Root cause: Documentation Deficiency

Since the new logging functionality has been introduced, I've noticed that some of our implementations started producing a warning in the Offline scenario e.g.:

```bash
anton@rpi4coral:~$ cat $(ck find program:image-classification-armnn-tflite-loadgen)/tmp/mlperf_log_detail.txt | grep -i warning_generic_message
:::MLLOG {"key": "warning_generic_message", "value": "Logging allocation detected: tid: 1890651 reserved_entries: 1024 max_entries: 24580", "time_ms": 7780759.263010, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": true, "file": "/home/anton/CK-TOOLS/mlperf-inference-r1.0/inference/loadgen/logging.cc", "line_no": 605, "pid": 1890651, "tid": 1890651}}
```

To reproduce, you can save [image-classification-tflite-loadgen.ipynb](https://raw.githubusercontent.com/krai/ck-mlperf/master/jnotebook/image-classification-tflite-loadgen/image-classification-tflite-loadgen.ipynb) or [image-classification-armnn-tflite-loadgen.ipynb](https://raw.githubusercontent.com/krai/ck-mlperf/master/jnotebook/image-classification-armnn-tflite-loadgen/image-classification-armnn-tflite-loadgen.ipynb) to disk, and open in Jupyter Notebook.

@nvpohanh Does this seem harmless enough to you, or something to worry about?

I won't think we would worry too much about the warning. Also, that warning should have been existed. I only modified the format how it is printed in the detailed log. My understanding is that it indicates that the buffer that LoadGen reserves is not large enough so it does a re-allocation which may hurt performance a little bit, but it does not lead to no-compliant results.

@guschmue Please correct me if I am wrong.

WG: @psyhtest will double check with the v0.7 implementation. Currently we don't anticipate any performance impact from this warning

@psyhtest any feedback?


@psyhtest another reminder. :)


Not considered to be a problem. Re-open if it shows up again.
Repository: mlcommons/inference
Issue: Pybind11 issues in loadgen
Root cause: Environment Incompatibility

Hi all,

Having an issue when trying to build the docker image=mlperf-infer-imgclassify-cpu using either the run_and_time.sh script or just with the docker build command. I have continuously been getting this error: 
![image](https://user-images.githubusercontent.com/71238108/107916607-8d079900-6f34-11eb-9cb8-e624301bf782.png)

Not sure what seems to be the problem or what a potential fix would be here. Could someone please help out? I did see #793 but was not sure what needed to be done to build the docker image

Thank you!


Try `git submodule update --init third_party/pybind`?

Just tried that, and no output onto console after returning, and unfortunately building docker image provided the same errors.

Or try replacing `git clone https://github.com/mlcommons/inference.git` with `git clone --recursive https://github.com/mlcommons/inference.git`?

Just tried, unfortunately the same problem still persists. 

This may be a silly question, but if the run_and_time script is building a docker image (docker build -t mlperf-infer-imgclassify-cpu -f Dockerfile.cpu .) and installing pybind according to the Dockerfile, why does it matter what is cloned? 

@guschmue please help 

IIRC, LoadGen requires the Pybind source files to be located here: https://github.com/mlcommons/inference/tree/master/third_party

Could you check if `third_party/pybind/include/pybind11/functional.h` exists?

Yes, I already had checked - that path and file both exist.
Repository: mlcommons/inference
Issue: dlrm inference problem: NameError: name 'args' is not defined
Root cause: Validation Gap

I'm trying to run dlrm with ~10GB model and following settings:
--max-ind-range=10000000 --data-sub-sample-rate=0.875

Here is the specification of the benchmarking task:

INFO:main:Namespace(accuracy=False, backend='pytorch-native', cache=0, count_queries=None, count_samples=None, data_sub_sample_rate=0.875, dataset='terabyte', dataset_path='/home/hpcclab/mlperf/inference/recommendation/dlrm/criteo', duration=None, find_peak_performance=False, inputs=['continuous and categorical features'], max_batchsize=2048, max_ind_range=10000000, max_latency=None, mlperf_bin_loader=True, mlperf_conf='../../../mlperf.conf', model='dlrm', model_path='/home/hpcclab/mlperf/inference/recommendation/dlrm/model/dlrm_terabyte.pytorch', numpy_rand_seed=123, output='/home/hpcclab/mlperf/inference/recommendation/dlrm/pytorch/output/pytorch-cpu/dlrm', outputs=['probability'], profile='dlrm-terabyte-pytorch', samples_per_query_multistream=None, samples_per_query_offline=1, samples_to_aggregate_fix=None, samples_to_aggregate_max=None, samples_to_aggregate_min=None, samples_to_aggregate_quantile_file='./tools/dist_quantile.txt', samples_to_aggregate_trace_file='dlrm_trace_of_aggregated_samples.txt', scenario='Offline', target_qps=None, test_num_workers=0, threads=64, use_gpu=False, user_conf='user.conf')


However, after reordering the last npz file (day_23_reordered.npz) in the second pass (2nd pass), I get the following error:


Sparse features= 26, Dense features= 13
data file: /home/hpcclab/mlperf/inference/recommendation/dlrm/criteo/terabyte_processed_test.bin number of batches: 1132
Using variable query size: custom distribution (file ./tools/dist_quantile.txt)
Traceback (most recent call last):
  File "python/main.py", line 619, in <module>
    main()
  File "python/main.py", line 503, in main
    model = backend.load(args.model_path, inputs=args.inputs, outputs=args.outputs)
  File "/home/hpcclab/mlperf/inference/recommendation/dlrm/pytorch/python/backend_pytorch_native.py", line 58, in load
    md_threshold=None,
  File "/home/hpcclab/mlperf/training/recommendation/dlrm/dlrm_s_pytorch.py", line 372, in __init__
    if args.loss_function == "mse":
**NameError: name 'args' is not defined**


Any idea is appreciated

@mnaumovfb 

I think that for some reason you are using DLRM model from master rather than mlperf branch. I think this is a bug that was introduced in master code recently. I'll investigate and fix it, but please try using mlperf branch.

The bug itself is that "args" command line parameter was not passed to init call (it used to be outside of it). As a quick workaround you should be able to substitute the entire "if" clause with one of its statements, i.e.
```
self.loss_fn = torch.nn.BCELoss(reduction="mean")
```
Repository: mlcommons/inference
Issue: Difference on reported QPSs between performance and accuracy passes
Root cause: Documentation Deficiency

For a performance only pass the reported QPS is the expected one. No problems there:
```
INFO:main:starting TestScenario.SingleStream
TestScenario.SingleStream qps=x, mean=---, time=---, queries=50000, tiles=---
```
but if we enable the accuracy pass the QPS has a value way higher than expected, y >> x and also seems `y` qps holds a completely unrealistic value:
```
INFO:main:starting TestScenario.SingleStream
TestScenario.SingleStream qps=y, mean=---, time=---, acc=---%, queries=50000, tiles=---
```
On both passes, all the other values, like `mean` and `tiles` reflect the expected values though.

Has anyone experienced something like this?

This qps report pre-dates loadgen. You only want to look at mlperf_log_summary.txt.
I can make a PR to remove this code and take the data from mlperf_log_summay.txt.

@guschmue Thank you.
Glad to know that in that case seems there is no real issue here. I will only take in account the summary file from now on.
Repository: mlcommons/inference
Issue:  can't find path to CK entry "module:program"
Root cause: Environment Incompatibility

@psyhtest 
root@ubuntu:/home/mlperf# ck compile program:image-classification-tf-cpp
CK error: [program] can't find path to CK entry "module:program" (?:032630d041b4fd8a:?)!

How can I fix it? Thanks!

@kubowuhao I believe my advice in #84 should fix this.

If you still have problems after that, please try:
```
$ ck pull repo:ck-autotuning
```

@kubowuhao since #84 is closed now, can this one be closed too? It would be interesting to know what sequence of steps led to this problem, but I appreciate it may be impossible to reproduce now.
Repository: mlcommons/inference
Issue: ssd-mobilenet 300x300 quantized finetuned from Habana

I hope to get some clarification about the ssd-mobilenet 300x300 quantized finetuned (by Habana):

https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection

In the table of the above README, it mentions that the model's mAP is 0.23594 which is better than the non-quantized model (ssd-mobilenet 300x300 whose mAP is 0.23) in the same table.

I have three questions:

[1] Is the quantized model the quantized version of the non-quantized model? (I assume this is the case but just in case)

[2] Was the quantization done post-training?  Or was there a quantization-aware training performed?

[3] What kind of `finetune` processing was done? (This may be related to the question [2])

Thanks for you attention!

@itayhubara 

@itayhubara another friendly reminder :)