zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: main.py: error: argument --profile: expected one argument
Root cause: Validation Gap

root@0824378d443e:/inference/vision/classification_and_detection# ./run_local.sh pytorch ssd-mobilenet cpu
usage: main.py [-h]
               [--dataset {imagenet,imagenet_mobilenet,imagenet_pytorch,coco-300,coco-300-pt,coco-1200,coco-1200-onnx,coco-1200-pt,coco-1200-tf}]
               --dataset-path DATASET_PATH [--dataset-list DATASET_LIST]
               [--data-format {NCHW,NHWC}]
               [--profile {defaults,resnet50-tf,resnet50-pytorch,resnet50-onnxruntime,mobilenet-tf,mobilenet-onnxruntime,ssd-mobilenet-tf,ssd-mobilenet-pytorch,ssd-mobilenet-onnxruntime,ssd-resnet34-tf,ssd-resnet34-pytorch,ssd-resnet34-onnxruntime,ssd-resnet34-onnxruntime-tf}]
               [--scenario SCENARIO] [--max-batchsize MAX_BATCHSIZE] --model
               MODEL [--output OUTPUT] [--inputs INPUTS] [--outputs OUTPUTS]
               [--backend BACKEND] [--model-name MODEL_NAME]
               [--threads THREADS] [--qps QPS] [--cache CACHE] [--accuracy]
               [--find-peak-performance] [--debug] [--mlperf_conf MLPERF_CONF]
               [--user_conf USER_CONF] [--time TIME] [--count COUNT]
               [--max-latency MAX_LATENCY]
               [--samples-per-query SAMPLES_PER_QUERY]
main.py: error: argument --profile: expected one argument
![76c3f19982180b3b3cdb8cbe6b4484e](https://user-images.githubusercontent.com/68749964/130847224-928e89dc-97ad-4854-a027-3a56df980580.png)
![47452d9f8afd3c71c0f27352e556f98](https://user-images.githubusercontent.com/68749964/130847249-27d007cd-9300-4b6a-966b-c556b1d78380.png)


Hello,
I have a similar issue with pytorch and ssd-mobilenet. Did you manage to address this?
Thanks!

outdated
Repository: mlcommons/inference
Issue: update model credits

Can we fill in https://docs.google.com/spreadsheets/d/17q_OtvVI_C5ET0shUTObHwiplUVRj5gFkU1dTkU7AcQ/edit#gid=0 for

RNN-T inference?

to update the website.

I have added RNN-T Inference owner to the above sheet. I have already created this branch with benchmark owners for inference [https://github.com/mlcommons/inference/tree/rnaidu02-patch-author-table#mlperf-inference-master](https://github.com/mlcommons/inference/tree/rnaidu02-patch-author-table#mlperf-inference-master) . Is this something you are looking for?

note that many of our models come from sources like model zoo's .... make sure the credit goes there too, I think I don't see this in the spreadsheet.
 

What we need: Reference implementation owner, integration with loadgen, folks who validated the model for training
@guschmue Can you help with this?

Did we get this finished?
Repository: mlcommons/inference
Issue: Inference with model in NCHW Format
Root cause: Configuration Error

When I try to run inference on with with a NCHW model I get the following error.
```
2019-05-04 13:45:18.523575: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x562c16a7e780 executing computations on platform Host. Devices:
2019-05-04 13:45:18.523609: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
ERROR:main:execute_parallel thread: Cannot feed value of shape (1, 224, 224, 3) for Tensor 'data:0', which has shape '(1, 3, 224, 224)'
ERROR:main:execute_parallel thread: Cannot feed value of shape (1, 224, 224, 3) for Tensor 'data:0', which has shape '(1, 3, 224, 224)'
```

I do understand it is complaining about the NCHW format while it expects the NHWC format. Is changing the python/main.py enough?
```
SUPPORTED_DATASETS = {
    "imagenet":
        (imagenet.Imagenet, dataset.pre_process_vgg, dataset.post_process_offset1,
         {"image_size": [3, 224, 224]}),
    "imagenet_mobilenet":
        (imagenet.Imagenet, dataset.pre_process_mobilenet, dataset.post_process_argmax_offset,
         {"image_size": [224, 224, 3]}),
}
```
 Changing the model could be one option, but I want to make mlperf work with NCHW 
Any help is much appreciated!

For tensorflow so far all models are in NHWC. The NHWC comes as default from the backend:
https://github.com/mlperf/inference/blob/master/cloud/image_classification/python/backend_tf.py#L25

The dataset defined here https://github.com/mlperf/inference/blob/master/cloud/image_classification/python/main.py#L36
is always HWC.

When we create the instance for the dataset like imagenet we pass in what the backend wants and if we need to transpose it to NCHW we'll flag it here:
https://github.com/mlperf/inference/blob/master/cloud/image_classification/python/imagenet.py#L39
and later when calling the pre-processor we'll transpose it if the flag is set, for example here:
https://github.com/mlperf/inference/blob/master/cloud/image_classification/python/dataset.py#L166


is there any more information you need? @syedaffanhamdani 

Nopes! A bundle of thanks!
Repository: mlcommons/inference
Issue: --count argument not working
Root cause: Documentation Deficiency

Although I manually added the --count 500 in the starting shell script I still see in the log file (mlperf_log_detail.txt) GeneratedQueries: "queries" : 60000:
Trace:
```
"pid": 2101, "tid": 140124207314688, "ts": 253977ns : ERROR : SingleStream only supports a samples_per_query of 1.
"pid": 2101, "tid": 140124207314688, "ts": 307040ns : Starting verification mode:
"pid": 2101, "tid": 140124207314688, "ts": 310222ns : ERROR : Unsupported scenario. Only MultiStream supported.
"pid": 2101, "tid": 140124207314688, "ts": 310741ns : Starting performance mode:
"pid": 2101, "tid": 140124207314688, "ts": 381222999ns : GeneratedQueries: "queries" : 60000, "samples per query" : 1, "duration" : 60000000000
```
Don't know why it is behaving like this. :(

--count is the number of images the loadgen samples from. The number of queries needs to meet a minimum time and count and loadgen will just sample from --count items until that is met. 
--count you might use if you don't want to load all 50000 imagenet items. If you want a shorter test time, you can use ```--time seconds``` that will tell loadgen to only get samples for that much time.

i assume this is not an issue.
Repository: mlcommons/inference
Issue: FR: equal distribution option to loadgen
Root cause: Unimplemented Feature Gap

Experimentally we've observed that the range of work required by the smallest and largest entries in the KiTS dataset is 8x. With random sampling, it takes about 5K images to be within 2.5% of the converged offline rate.

We propose to add an option to loadgen to ensure equal sampling, by choosing a sequence of permutations of the dataset determined by the RNG seed, and that this be part of the configuration for 3DUNet.

3DUnet is offline only, shouldn't submitter always able to choose a preferred order for inference?

The randomness comes from the variability in sampling, not the order.

Further clarification based on questions in the WG.

The images in KiTS are variable-sized, and there's up to 8x variation in processing time between the smallest and largest images. While using the same seed guarantees that all submissions use the same trace, different seeds may result in considerable variation in throughput, which would be reflected in different performance across different rounds of MLPerf.

It should be a goal of the offline scenario to converge as closely as possible on the underlying average rate.  Equal sampling promotes this in the presence of variable-sized samples by ensuring each sample is processed the same number of times (or as closely as possible for a specified minimum sample count and runtime.)

The proposed mechanism is to generate the trace by concatenating a sequence of permutations of the dataset, with each permutation chosen using the random seed.

Currently, the parameter performance_count is set by default to 16. This implies that loadgen by default will only generate 16 different sample indices. Is there a reason for this? Or should this be changed to generate indices over the whole dataset?

It should be the whole dataset, which IIRC is only 42 images.
Repository: mlcommons/inference
Issue: [Vision] No option to set performance sample count
Root cause: Unimplemented Feature Gap

In [main.py](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/python/main.py#L536), performance sample count is bundled with the `count` parameter, which has also been used for setting [`max_query_count`](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/python/main.py#L527) 

Propose to add an extra parameters `buffer-size`
```
parser.add_argument("--buffer-size", help="performance sample count")
```

I'd suggest calling this parameter `--performance-sample-count` for better maintainability.

Interestingly, in [line 536](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/python/main.py#L536) the default is 500 (which can be used to set the default value of the new parameter):
```
qsl = lg.ConstructQSL(count, min(count, 500), ds.load_query_samples, ds.unload_query_samples)
```
where as the comment in [line 427](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/python/main.py#L427) says 200:
```
For perf model we always limit count to 200.
```
I'd recommend removing this comment too.
Repository: mlcommons/inference
Issue: Adding minimum system requirement for reproducing the reference implementations
Root cause: Documentation Deficiency

It would be nice to have a "Minimum Hardware Requirement" to reproduce the reference implementations especially for DLRM where the requirements are non-trivial. 

@rnaidu02 find out from intel team on the RAM requirements to run the DLRM benchmark

It uses ~180GB while running DLRM Pytorch reference code for Offline test using pre-processed data.
Repository: mlcommons/inference
Issue: 3D-UNet (KiTS19) Pytorch model not processed by jit.trace
Root cause: Unimplemented Feature Gap

Current [3D-UNet model](https://zenodo.org/record/4771141/files/3dunet_kits19_pytorch.ptc?download=1) (KiTS19) is already processed by jit.trace. Since it is already jitted, it is not possible for performing jit.trace for platform specific optimizations. Intel team is requesting reference implementation owner to share the model not processed by jit.trace (original model weights).  
@delock 


@rnaidu02 please review https://github.com/mlcommons/inference/pull/1033

Thanks, this PR works for me.

Thanks @DilipSequeira
Repository: mlcommons/inference
Issue: Integrate new Object detector with reference code
Root cause: Architectural Constraint

@pgmpablo157321

https://drive.google.com/drive/u/2/folders/10gfOCx7vfeAkWz1q7Y4ykC4lrds46vkJ

I have some changes to integrate the new object detector. However I have some questions:

- Currently I am loading the model from a checkpoint (in pytorch). So this [source code](https://github.com/ahmadki/training/tree/akiswani/ssd_v2/single_stage_detector/ssd/model) is needed for loading the model. Locally, I just have the files in the folder. But how should I include them for the repo? Another option would be to save the entire model and use this file.
- On a related note. Are we going to upload the model to a repository and make it accesible? Like the other models in this [readme](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/README.md)

I think it is not good to include the model source in the benchmark app - it creates a mess over time and we have pretty bad experience with the old ssd.
Better would be to create a torchscript model with the source in some other directory and use that torchscript model in the benchmark app.
I think we do this for the pytorch resnet50 model.

@guschmue Currently the source code of the model is in it's own [directory](https://github.com/pgmpablo157321/inference/tree/master/vision/classification_and_detection/python/models/resnext50), and it includes a [torchscript model](https://github.com/mlcommons/inference/blob/18092a75c9d145033e5e1174bf31679476016f9d/vision/classification_and_detection/python/models/resnext50/retinanet.py#L632) that is used by the [pytorch backend](https://github.com/mlcommons/inference/blob/18092a75c9d145033e5e1174bf31679476016f9d/vision/classification_and_detection/python/backend_pytorch_native.py#L7). Could you take a look at the [PR](https://github.com/mlcommons/inference/pull/1046) and check if there are some changes I should make?

torchscript  models store the torchscript IR representation of the model so it does not need the source code.
We badly want to keep model source out of the benchmark app - it becomes hard to maintain over time.
Unless there is a good reason why torchscript doesn't work here I'd go that route.
Repository: mlcommons/inference
Issue: Error parsing message with resnet-50 using pytorch backend
Root cause: Validation Gap

I get the following error message when trying to load the resnet-50 model for pytorch backend. The model I use is from this [link](https://zenodo.org/record/4588417/files/resnet50-19c8e357.pth)

I am on r1.1 branch using pytorch 1.10

```
./run_local.sh pytorch resnet50 gpu --count 1000 --time 60 --scenario Server --qps 200 --max-latency 0.1
INFO:main:Namespace(accuracy=False, backend='pytorch', cache=0, count=1000, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/home/mchow009/Workspace/datasets/ILSVRC2012', debug=False, find_peak_performance=False, inputs=['image'],
max_batchsize=32, max_latency=0.1, mlperf_conf='../../mlperf.conf', model='/home/mchow009/Workspace/models/resnet50-v1.5/resnet50-19c8e357.pth', model_name='resnet50', output='/home/mchow009/Workspace/mlcommons/inference/vision/classification_and_detection/output/pytorch-gpu/resnet50', outputs=['ArgMax:0'], profile='resnet50-pytorch', qps=200, samples_per_query=None, scenario='Server', threads=64, time=60, user_conf='user.conf')
INFO:root:Failed to import cuda module: No module named 'caffe2.python.caffe2_pybind11_state_gpu'
INFO:root:This caffe2 python run has AMD GPU support!
INFO:imagenet:loaded 1000 images, cache=0, took=0.0sec
Traceback (most recent call last):
  File "python/main.py", line 565, in <module>
    main()
  File "python/main.py", line 443, in main
    model = backend.load(args.model, inputs=args.inputs, outputs=args.outputs)
  File "/home/mchow009/Workspace/mlcommons/inference/vision/classification_and_detection/python/backend_pytorch.py", line 34, in load
    self.model = onnx.load(model_path)
  File "/home/mchow009/.env3/lib/python3.6/site-packages/onnx-1.10.2-py3.6-linux-x86_64.egg/onnx/__init__.py", line 121, in load_model
    model = load_model_from_string(s, format=format)
  File "/home/mchow009/.env3/lib/python3.6/site-packages/onnx-1.10.2-py3.6-linux-x86_64.egg/onnx/__init__.py", line 158, in load_model_from_string
    return _deserialize(s, ModelProto())
  File "/home/mchow009/.env3/lib/python3.6/site-packages/onnx-1.10.2-py3.6-linux-x86_64.egg/onnx/__init__.py", line 99, in _deserialize
    decoded = cast(Optional[int], proto.ParseFromString(s))
google.protobuf.message.DecodeError: Error parsing message

```

@kahmed10 

@marcusnchow from the error message, it looks like you are using an AMD GPU. Can you specify which hardware you are using?
Repository: mlcommons/inference
Issue: Got Error message when running the bert_large model
Root cause: Environment Incompatibility

Hello, I trying to run the bert-large model using pytorch backend. do not use Nvidia-docker.
I want to run the CPU target. But, I met the error message as below.

Traceback (most recent call last):
  File "/opt/app-root/src/inference/language/bert/accuracy-squad.py", line 405, in <module>
    main()
  File "/opt/app-root/src/inference/language/bert/accuracy-squad.py", line 397, in main
    write_predictions(eval_examples, eval_features, results, 20, 30, True, args.out_file, args.max_examples)
  File "/opt/app-root/src/inference/language/bert/accuracy-squad.py", line 272, in write_predictions
    final_text = get_final_text(tok_text, orig_text, do_lower_case)
  File "/opt/app-root/src/inference/language/bert/accuracy-squad.py", line 96, in get_final_text
    tokenizer = tokenization.BasicTokenizer(do_lower_case=do_lower_case)
AttributeError: module 'tokenization' has no attribute 'BasicTokenizer'
Traceback (most recent call last):
  File "run.py", line 110, in <module>
    main()
  File "run.py", line 99, in main
    subprocess.check_call(cmd, shell=True)
  File "/opt/app-root/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'python3 /opt/app-root/src/inference/language/bert/accuracy-squad.py --max_examples=2' returned non-zero exit status 1.

I ran the command as below.
python run.py --backend pytorch --max_examples 2 --accuracy

This is my Question.
1. Is there a way to solve that error? 
2. currently, do not support CPU target? if not, please let me know how to run the bert_model using CPU target.


did you install `tokenization` package correctly?

@nvpohanh Yes, I installed it using a pip package ( pip install tokenization)
I reinstalled the tokenization package and tried again. But, the same error message is still displayed.

tokenization version : 1.0.7

Okay, I found the issue. the `tokenization` here is NOT the pip `tokenization` package. Rather it should be this `tokenization.py` file located in the DeepLearningExamples submodule: https://github.com/NVIDIA/DeepLearningExamples/blob/b03375bd6c2c5233130e61a3be49e26d1a20ac7c/TensorFlow/LanguageModeling/BERT/tokenization.py

Could you uninstall the tokenization package and place this file in the same directory as `accuracy-squad.py`? Thanks

@nvpohanh wow, The problem is solved :) Thank you for your answer.
Repository: mlcommons/inference
Issue: [RNN-T] Only Encoder gets called in class 'RNNT'
Root cause: Algorithmic Error

Hi,

I wanted to try the reference implementation for the RNN-T and I wonder if I encountered a bug in model_separable_rnnt.py.
I think only the encoder gets called?

In line 51 and 52 in class 'RNNT'
```
    def forward(self, x_padded: torch.Tensor, x_lens: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
        return self.encoder(x_padded, x_lens)
```
Repository: mlcommons/inference
Issue: MLCommon CLA link no longer works

The CLA link: https://github.com/mlcommons/systems/blob/main/mlcommons_cla.txt no longer exists and I couldn't sign it for the pull request.

MLCommons CLA bot:
Thank you for your submission, we really appreciate it. We ask that you sign our MLCommons CLA and be a member before we can accept your contribution. If you are interested in membership, please contact membership@mlcommons.org .
0 out of 1 committers have signed the MLCommons CLA.

@nvzhihanj  Can you check the CLA form from here (https://mlcommons.org/en/policies/)

https://drive.google.com/file/d/1SkpAdb3ka8tXepCawgGxAyo6cUH5ZAZg/view ?



@rnaidu02 Thanks for the reply. Seems like the form you pointed grants permission to the "entity" (and its designated employees). Since NVIDIA must have already signed such a form, is there a way to add additional employees without re-signing this form with the full list of employees? Or should I just ask the POC to add one employee at a time to the form?

To be specific we found and filed this form: https://docs.google.com/forms/d/e/1FAIpQLSd4gItNaxbm9okFroxJEUmkmPjAJ6YegNxmHRAq8f6Uv_SDyw/viewform. Is this enough to get the access?

The form works and I have been added. Thanks!
Repository: mlcommons/inference
Issue: while profiling, Is there a way to take inference success rate into consideration 
Root cause: Unimplemented Feature Gap

currently we generate profiling results basing on all inference results. Is there a way to differentiate successful inference results from failed inference results? 

an inference request being successful or not is different from the result being accurate or not. For example, in our case, for every inference request, we need to send a http request to an inference service to get result, so the inference request being successful means the http result status code is 200. And there  could be situations where the http request return back non-200 codes, for example 429, etc.

It would be nice when summarize profiling results, we take account ONLY successful inference results. for example, QPS == # of successful Requests per sec. and latency == duration of successful requests.

Thanks!
Repository: mlcommons/inference
Issue: DLRM server mode, core dumped
Root cause: Resource Mishandling

Hi,

I ran official DLRM code by official command without any modification but got core dumped when running serve mode

**Here is the command:**
./run_local.sh pytorch dlrm terabyte cpu --scenario Server --max-ind-range=40000000 --samples-to-aggregate-quantile-file=./tools/dist_quantile.txt --max-batchsize=2048 --accuracy --mlperf-bin-loader

**Here is the log:**
python/main.py --profile dlrm-terabyte-pytorch --mlperf_conf ../../../mlperf.conf --model dlrm --model-path ./model/dlrm_terabyte.pytorch --datastput /mlcommons_v2.0/inference/recommendation/dlrm/pytorch/output/pytorch-cpu/dlrm --scenario Server --max-ind-range=40000000 --samplile.txt --max-batchsize=2048 --accuracy --mlperf-bin-loader
INFO:main:Namespace(accuracy=True, backend='pytorch-native', cache=0, count_queries=None, count_samples=None, data_sub_sample_rate=0.0, dataset='terabyte', find_peak_performance=False, inputs=['continuous and categorical features'], max_batchsize=2048, max_ind_range=40000000, max_latency=None, mlperf_bin_loodel='dlrm', model_path='./model/dlrm_terabyte.pytorch', numpy_rand_seed=123, output='/mlcommons_v2.0/inference/recommendation/dlrm/pbility'], profile='dlrm-terabyte-pytorch', samples_per_query_multistream=8, samples_per_query_offline=2048, samples_to_aggregate_fix=None, samples_to_aggramples_to_aggregate_quantile_file='./tools/dist_quantile.txt', samples_to_aggregate_trace_file='dlrm_trace_of_aggregated_samples.txt', scenario='Server',  use_gpu=False, user_conf='user.conf')
Using CPU...
Reading pre-processed data=/dataset//terabyte_processed.npz
Sparse features= 26, Dense features= 13
data file: /dataset//terabyte_processed_test.bin number of batches: 89137319
Using variable query size: custom distribution (file ./tools/dist_quantile.txt)
../../../../training/recommendation/dlrm/data_loader_terabyte.py:233: UserWarning: The given NumPy array is not writeable, and PyTorch does not support no the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before coll be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
INFO:main:starting TestScenario.Server

./run_local.sh: line 14:  5102 Segmentation fault      (core dumped) python python/main.py --profile $profile $common_opt --model $model --model-path $modDIR --output $OUTPUT_DIR $EXTRA_OPS $@

Since I got no error when running offline mode, I think the running environment is OK
Anyone got the same problem?

Thanks in advanced
Repository: mlcommons/inference
Issue: build issues with loadgen.cc
Root cause: Validation Gap

```
inference/loadgen/loadgen.cc: In member function ‘void mlperf::loadgen::PerformanceSummary::ProcessLatencies()’:
inference/loadgen/loadgen.cc:617:13: error: ‘struct mlperf::loadgen::PerformanceResult’ has no member named ‘query_intervals’
  617 |   assert(pr.query_intervals.size() == query_count);
      |             ^~~~~~~~~~~~~~~
```

This issue is due to this change. https://github.com/mlcommons/inference/commit/15ae1268b05634caabf54c1fbd5a34449c1410bc#r65695484

@VladislavShipovskiy @arjunsuresh Can't reproduce the error. What branch are you using to build loadgen? Can you share some commands to reproduce

@VladislavShipovskiy @arjunsuresh @rnaidu02 Oh nvm, i found the error. It should be `pr.query_latencies` instead of `pr.query_intervals`. Locally, I have it correctly. Not sure what happened, but i'll make a pr fixing it