zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: ModuleNotFoundError: No module named 'onnx.optimizer'
Root cause: External Dependency Breakage

Hi,
I am executing below command on riscv64-linux.
**$ ./run_local.sh pytorch mobilenet cpu**

get following error,

Traceback (most recent call last):
  File "python/main.py", line 558, in <module>
    main()
  File "python/main.py", line 414, in main
    backend = get_backend(args.backend)
  File "python/main.py", line 243, in get_backend
    from backend_pytorch import BackendPytorch
  File "/home/root/inference/vision/classification_and_detection/python/backend_pytorch.py", line 10, in <module>
    import caffe2.python.onnx.backend
  File "/usr/lib/python3.8/site-packages/caffe2/python/onnx/backend.py", line 28, in <module>
    import onnx.optimizer
**ModuleNotFoundError: No module named 'onnx.optimizer'**

How I resolve it?

Thanks



mobilenet is no longer part of mlperf since v0.7. Still should work. But for pytorch there is a catch: it uses the onnx model via the caffe2 path in pytorch which at that time was still fine in pytorch (it no longer is).
I think you might get it to work by installing an older onnx package, something like pip install onnx==1.6 (I think onnx.optimizer was depreciated some time ago).

You could switch to resnet50 which should be supported by the native python backend in the reference app.


Mobilenet is not supported and switch to ResNet50 to use pytorch backend
Repository: mlcommons/inference
Issue: FileNotFoundError: [Errno 2] No such file or directory: '/home/root/models/mobilenet_v1_1.0_224/val_map.txt'
Root cause: Environment Incompatibility

Hi,
How to fix following error,
**FileNotFoundError: [Errno 2] No such file or directory: '/home/root/models/mobilenet_v1_1.0_224/val_map.txt'**

Thanks

You can download it from e.g. NVIDIA's v1.0 submission:

```
$ wget https://raw.githubusercontent.com/mlcommons/inference_results_v1.0/master/closed/NVIDIA/data_maps/imagenet/val_map.txt
```

Hi @Ankita-020696.

You can also use our [CK automation](https://github.com/ctuning/ck) to install ImageNet aux package that has this file:
```
pip3 install ck
ck pull repo:ck-ml
ck install package --tags=imagenet,aux,from.berkeley
```

You can then find the file val.txt that is the same as above val_map.txt as follows:
```
cd `ck locate env --tags=from.berkeley`
ls -l val.txt
```

Hope it's of any help.
Repository: mlcommons/inference
Issue: Issue downloading DLRM dataset
Root cause: Documentation Deficiency

Hi,

For dlrm, the datasets from the README is out of date for both tf and pytorch.

TF:
```
gsutil cp -r gs://dmchen-data/dlrm_tf1_data/model_dir_0 .
No command was given.

Choose one of -b, -d, -e, or -r to do something.
Try `/usr/bin/gsutil --help' for more information.
```
Pytorch:

The download links given in the README are out of date: https://github.com/mlcommons/inference/tree/master/recommendation/dlrm/pytorch

https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/
https://labs.criteo.com/2013/12/download-terabyte-click-logs/

```
 wget https://s3-eu-west-1.amazonaws.com/kaggle-display-advertising-challenge-dataset/dac.tar.gz
--2021-05-11 16:44:13--  https://s3-eu-west-1.amazonaws.com/kaggle-display-advertising-challenge-dataset/dac.tar.gz
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.108.51
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.108.51|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-05-11 16:44:13 ERROR 404: Not Found.
```

Are there ones that are up to date ?

Regards,

Linking issue https://github.com/mlcommons/inference/issues/917

Dave Chen says:

> For TF files, I think it is still accessible (gs://dmchen-data/dlrm_tf1_data/model_dir_0). For Pytorch, I did not work on that and am not sure who wrote the Pytorch instructions.

oh good. 

Is this freely accessible?
```
gsutil cp -r gs://dmchen-data/dlrm_tf1_data/model_dir_0 .
ServiceException: 401 Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.
```
Related issue https://github.com/mlcommons/inference/issues/918 

Related issue https://github.com/mlcommons/inference/issues/877

The scripts in TF v1 DLRM are for training. Should they really be here for the inference repo?


Criteo dataset for MLPerf users should be downloaded from https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf/

@guschmue  created this PR https://github.com/mlcommons/inference/pull/961 with updated link.

> Is this freely accessible?
> 
> ```
> gsutil cp -r gs://dmchen-data/dlrm_tf1_data/model_dir_0 .
> ServiceException: 401 Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.
> ```
> 
> Related issue #918

This location was stale.  The DLRM inference data has been uploaded at the following location:

[DLRM inference data](https://drive.google.com/drive/folders/1B8n8YzkKSn5Khs2gqiu_Ql16_v-DxF3B?usp=sharing).

Hi,
The Criteo Kaggle dataset that has been linked to earlier, is no longer available. I am referring to this link:
https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/
Do you know why? Can you please look into that?
Repository: mlcommons/inference
Issue: ModuleNotFoundError: No module named 'tensorflow' 
Root cause: Environment Incompatibility

HI,
I am working on riscv aarch for which tensorflow is not ported. But I have tflite installed on it.
when I execute below command:
**$ ./run_local.sh tflite mobilenet cpu**

get the following error,

> INFO:main:Namespace(accuracy=False, backend='tflite', cache=0, count=None, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/home/root/Ankita/inference/vision/classification_and_detection', debug=False, find_peak_performance=False, inputs=['input:0'], max_batchsize=32, max_latency=None, mlperf_conf='../../mlperf.conf', model='/home/root/Ankita/inference/vision/classification_and_detection/mobilenet_v1_1.0_224.tflite', model_name='mobilenet', output='/home/root/Ankita/inference/vision/classification_and_detection/output/tflite-cpu/mobilenet', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-tf', qps=None, samples_per_query=None, scenario='SingleStream', threads=4, time=None, user_conf='user.conf')
> Traceback (most recent call last):
>   File "python/main.py", line 558, in <module>
>     main()
>   File "python/main.py", line 414, in main
>     backend = get_backend(args.backend)
>   File "python/main.py", line 249, in get_backend
>     from backend_tflite import BackendTflite
>   File "/home/root/Ankita/inference/vision/classification_and_detection/python/backend_tflite.py", line 9, in <module>
>     import tensorflow as tf
> ModuleNotFoundError: No module named 'tensorflow'


Is tensorflow module must to execute above module, if not how do I fix this error?

Thanks

we use the tflite that comes prebuilt with tensorflow. 
```
pip install tensorflow
```
will fix it.
Repository: mlcommons/inference
Issue: ERROR:imagenet:no images in image list found
Root cause: Environment Incompatibility

Hi,
How to fix this error, from where can i get the image?

> $ ./run_local.sh tflite mobilenet cpu
> INFO:main:Namespace(accuracy=False, backend='tflite', cache=0, count=None, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/home/root/Ankita/inference/vision/classification_and_detection', debug=False, find_peak_performance=False, inputs=['input:0'], max_batchsize=32, max_latency=None, mlperf_conf='../../mlperf.conf', model='/home/root/Ankita/inference/vision/classification_and_detection/mobilenet_v1_1.0_224.tflite', model_name='mobilenet', output='/home/root/Ankita/inference/vision/classification_and_detection/output/tflite-cpu/mobilenet', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-tf', qps=None, samples_per_query=None, scenario='SingleStream', threads=4, time=None, user_conf='user.conf')
> ERROR:imagenet:no images in image list found
> Traceback (most recent call last):
>   File "python/main.py", line 558, in <module>
>     main()
>   File "python/main.py", line 428, in main
>     ds = wanted_dataset(data_path=args.dataset_path,
>   File "/home/root/Ankita/inference/vision/classification_and_detection/python/imagenet.py", line 77, in __init__
>     raise ValueError("no images in image list found")
> ValueError: no images in image list found
> 

Thanks

you'd need to get a copy of imagenet. For testing you could use
the 320px subset from here https://github.com/fastai/imagenette 
(would not work for submission).


I assume this is resolved.
Repository: mlcommons/inference
Issue: Accuracy issue on ResNet50
Root cause: Documentation Deficiency

Hi,
     I'm reproducing the accuracy result for the resnet50 onnx model (opset-8) using onnxruntime as the backend and running with docker. When running on CPU, it matches the one specified in the document, 76.456%. However, I got 76.460% when running with GPU. Is this expected? 
     
Some specs when running with GPU if this helps, and please let me know if you need more information:
GPU: GeForce RTX 3090
CUDA Version: 11.2
ONNXRuntime-GPU Version: 1.7.0 (Identified from the output when building docker image)

Thanks

Isn't 76.460% slightly better than CPU? Why would that be an issue?

So as long as it is at least 99% of the target top-1 accuracy as specified in the paper, then it is acceptable?
Also, where can I find the quality target for tasks that are not in v0.5? Since the paper only consists of tasks in v0.5 and their quality target.

Accuracy threshold can be found here: https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#411-constraints-for-the-closed-division

And yes, anything above the accuracy threshold is fine.

Thanks for the reply. This is really helpful!
Repository: mlcommons/inference
Issue: _pickle.UnpicklingError: invalid load key, '\x0a'.
Root cause: Environment Incompatibility

Hi,
How to fix following error?

> $ ./run_local.sh  --dataset coco-1200 --profile ssd-resnet34-pytorch
> INFO:main:Namespace(accuracy=False, backend='pytorch-native', cache=0, count=None, data_format=None, dataset='coco-1200', dataset_list=None, dataset_path='/home/root/Ankita/data/coco', debug=False, find_peak_performance=False, inputs=['image'], max_batchsize=32, max_latency=None, mlperf_conf='../../mlperf.conf', model='/home/root/Ankita/model/resnet50_v1.pb', model_name='ssd-resnet34', output='/home/root/Ankita/inference/vision/classification_and_detection/output/tf-cpu/resnet50', outputs=['bboxes', 'labels', 'scores'], profile='ssd-resnet34-pytorch', qps=None, samples_per_query=None, scenario='SingleStream', threads=4, time=None, user_conf='user.conf')
> Error in cpuinfo: processor architecture is not supported in cpuinfo
> INFO:coco:reduced image list, 4999 images not found
> INFO:coco:loaded 1 images, cache=0, took=8.7sec
> /home/root/Ankita/inference/vision/classification_and_detection/python/coco.py:115: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
>   self.label_list = np.array(self.label_list)
> Traceback (most recent call last):
>   File "python/main.py", line 558, in <module>
>     main()
>   File "python/main.py", line 436, in main
>     model = backend.load(args.model, inputs=args.inputs, outputs=args.outputs)
>   File "/home/root/Ankita/inference/vision/classification_and_detection/python/backend_pytorch_native.py", line 26, in load
>     self.model = torch.load(model_path,map_location=lambda storage, loc: storage)
>   File "/usr/lib/python3.8/site-packages/torch/serialization.py", line 593, in load
>     return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
>   File "/usr/lib/python3.8/site-packages/torch/serialization.py", line 762, in _legacy_load
>     magic_number = pickle_module.load(f, **pickle_load_args)
> _pickle.UnpicklingError: invalid load key, '\x0a'.
> 

Thanks

@rnaidu02 @guschmue 
Any suggestion?

Thanks
Repository: mlcommons/inference
Issue: Bad tf dtypes import for vision/classification_and_detection
Root cause: External Dependency Breakage

The tensorflow backend for vision/classification_and_detection (python/backend_tf.py) imports "tensorflow.core.framework.dtypes" which does not exist. Instead it should import "tensorflow.dtypes".

I'm not sure if this is some issue with my local system (I'm using the dockerfile so it shouldn't be). Was this written for an old version of tensorflow that the dockerfile no longer uses?

Reproduce:

- Create Docker container from vision/classification_and_detection/Dockerfile.gpu
- Launch container as: `docker run -it --rm --gpus all mlperf_image`
- Follow the example from vision/classification_and_detection/GettingStarted.ipynb
    - I didn't use ipynb but ran the commands manually (including both cpu and gpu configurations)
    - This actually occurs no matter which model you try to run

Error Message:
```Traceback (most recent call last):
  File "python/main.py", line 558, in <module>
    main()
  File "python/main.py", line 414, in main
    backend = get_backend(args.backend)
  File "python/main.py", line 234, in get_backend
    from backend_tf import BackendTensorflow
  File "/tmp/inference/vision/classification_and_detection/python/backend_tf.py", line 8, in <module>
    from tensorflow.core.framework import dtypes
ImportError: cannot import name 'dtypes' from 'tensorflow.core.framework' (/opt/anaconda3/lib/python3.7/site-packages/tensorflow/core/framework/__init__.py)
```

System Setup:

- CUDA:  NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 11.1
- Python (from dockerfile): Python 3.7.3
- Tensorflow (from dockerfile): 2.5.0

sorry, that must have come in recently - guess it doesn't work for tf2. Let me fix it.

Hi, my solution for this type of issue is
change the line  "from tensorflow.core.framework import dtypes" in backend_tf.py to 
"from tensorflow.python.framework import dtypes", and it worked  for me
Repository: mlcommons/inference
Issue: torch & pickle version required for MLperf
Root cause: Documentation Deficiency

Hi,
what torch & pickle version is required for mlperf/inference?

Thanks

@Ankita-020696 Can we get specific details on the model that requires these details?

> @Ankita-020696 Can we get specific details on the model that requires these details?

I am facing pickle error with inference for pytorch.
https://github.com/mlcommons/inference/issues/932
Repository: mlcommons/inference
Issue: why  resnet50-19c8e357.pth is not used for pytorch backend
Root cause: Configuration Error

I see the mode file from https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection


<!--StartFragment-->
resnet50-v1.5 | pytorch | 76.014% | imagenet2012 validation | from zenodo | from TorchVision | fp32 | NCHW
-- | -- | -- | -- | -- | -- | -- | --


<!--EndFragment-->

but, ' ./run_local.sh pytorch resnet50 cpu' requires resnet50_v1.onnx, not the resnet50-19c8e357.pth file, see more in run_common.sh, thanks.
```
#
# pytorch
#
if [ $name == "resnet50-pytorch" ] ; then
    model_path="$MODEL_DIR/resnet50_v1.onnx"
    profile=resnet50-onnxruntime
    extra_args="$extra_args --backend pytorch"
fi
```

there will be issue if I just change the .onnx file to .pth.

hello, I have the same question too. How do you fix the problem finally?

hi, my understanding is that it is demo for interface, not for the real implementation.
Repository: mlcommons/inference
Issue: the 3d Unet onnx
Root cause: Documentation Deficiency

the exported onnx seems has the input format NCHWD ?   How can I get the onnx with format NCDHW?

the dummy_input may be wrong with dummy_input = torch.rand([1, channels,  height, width, depth]).float().to(device)?   Maybe it should be dummy_input = torch.rand([1, channels, depth, height, width]).float().to(device) in unet_pytorch_to_onnx.py ?

> the exported onnx seems has the input format NCHWD ? How can I get the onnx with format NCDHW?

The model is trained with NCHWD (H=W=224, D=160), but it doesn't really matter. You can view it as NCDHW with D=H=224 and W=160. 

If you really want to make the input become [BS, C, 160, 224, 224], then you will have to transpose all the weights in the ONNX nodes

> > the exported onnx seems has the input format NCHWD ? How can I get the onnx with format NCDHW?
> 
> The model is trained with NCHWD (H=W=224, D=160), but it doesn't really matter. You can view it as NCDHW with D=H=224 and W=160.
> 
> If you really want to make the input become [BS, C, 160, 224, 224], then you will have to transpose all the weights in the ONNX nodes

you mean allthough the input  of torch.nn.con3d is NCDHW, you just let D=224, H=224, W=160 in the input?

> you mean allthough the input of torch.nn.con3d is NCDHW, you just let D=224, H=224, W=160 in the input?

Exactly!

> > you mean allthough the input of torch.nn.con3d is NCDHW, you just let D=224, H=224, W=160 in the input?
> 
> Exactly!
Thank you!
Repository: mlcommons/inference
Issue:  DLRM: How long does a sample run take?

I ran this command (with subsampling):
```shell
./run_local.sh pytorch dlrm terabyte cpu --scenario Server  --max-ind-range=10000000 --data-sub-sample-rate=0.875 --samples-to-aggregate-quantile-file=./tools/dist_quantile.txt --max-batchsize=2048
```
on a 192 core machine, and till now it is about 15 hrs, and there is still no sign to reach an end. I wonder how long it usually takes to run this command?

Well, I don't know why you let the default value of `duration` None, and give sample commands in README.md with it unset...... It is NOT user-friendly at all......
Repository: mlcommons/inference
Issue: BERT doesn't work for PyTorch without a GPU
Root cause: Environment Incompatibility

Related to #808.

If I remove `.cuda()` from the PyTorch SUT, the tests never start, and get stuck after issuing queries. I believe this is due to the model used by the PyTorch SUT contains CUDA specific parts, so the most likely solution is providing a model for BERT that doesn't require a GPU/contain CUDA specifics. 

There isn't much information between starting and finishing, nor on how long it takes, so I did manage to get a finished build with my setup eventually.
Repository: mlcommons/inference
Issue: NameError: name 'interpreter_wrapper' is not defined
Root cause: Validation Gap

# ./run_local.sh tflite --profile ssd-mobilenet-tf
INFO:main:Namespace(accuracy=False, backend='tflite', cache=0, count=None, data_format=None, dataset='coco-300', dataset_list=None, dataset_path='/home/root/Ankita/data/coco', debug=False, find_peak_performance=False, inputs=['image_tensor:0'], max_batchsize=32, max_latency=None, mlperf_conf='../../mlperf.conf', model='/home/root/Ankita/model/resnet50_v1.tflite', model_name='ssd-mobilenet', output='/home/root/Ankita/inference/vision/classification_and_detection/output/tflite-cpu/resnet50', outputs=['num_detections:0', 'detection_boxes:0', 'detection_scores:0', 'detection_classes:0'], profile='ssd-mobilenet-tf', qps=None, samples_per_query=None, scenario='SingleStream', threads=4, time=None, user_conf='user.conf')
INFO:coco:reduced image list, 4997 images not found
INFO:coco:loaded 3 images, cache=0, took=9.0sec
/home/root/Ankita/inference/vision/classification_and_detection/python/coco.py:115: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  self.label_list = np.array(self.label_list)
Traceback (most recent call last):
  File "python/main.py", line 558, in <module>
    main()
  File "python/main.py", line 436, in main
    model = backend.load(args.model, inputs=args.inputs, outputs=args.outputs)
  File "/home/root/Ankita/inference/vision/classification_and_detection/python/backend_tflite.py", line 32, in load
    self.sess = interpreter_wrapper.Interpreter(model_path=model_path)
NameError: name 'interpreter_wrapper' is not defined


dupe of https://github.com/mlcommons/inference/issues/947
Repository: mlcommons/inference
Issue: You don't need the strides in this function, why is it here ?

https://github.com/mlcommons/inference/blob/1f53d40aecd15b412d9eb16e4382520746c1549f/vision/classification_and_detection/python/models/ssd_r34.py#L207

It is dead code. We can remove it :)

@guschmue  to look at this change
Repository: mlcommons/inference
Issue: Yocto/inference benchmarks 

Dears,

I'd like to use the mlperf for inference in yocto environment for benchmarking ARM-based devices. Is it available?  

Hi @onyx22574, We (Krai/dividiti) have successfully used automated Collective Knowledge workflows to benchmark many Arm platforms (see MLPerf Inference v0.5-v1.0). I haven't used Yocto for a while though. (Most of our submissions were Ubuntu-based.) Would you like to try e.g. the [TFLite workflow](https://github.com/krai/ck-mlperf/tree/master/program/image-classification-tflite-loadgen) described in this [Jupyter notebook](https://github.com/krai/ck-mlperf/tree/master/jnotebook/image-classification-tflite-loadgen)?

@onyx22574 Feel free to reopen if you have any further questions or comments.
Repository: mlcommons/inference
Issue: Error with Single State Detector - GPU Support: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
Root cause: Interface Contract Mismatch

hi all, I am running the inference single state detector benchmark with GPU support and I got the below error:

$ ./run_and_time.sh
```
stdbuf: missing operand
Try 'stdbuf --help' for more information.

real    0m0.001s
user    0m0.000s
sys     0m0.000s
/opt/conda/lib/python3.6/site-packages/torch/nn/_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
  warnings.warn(warning.format(ret))
Traceback (most recent call last):
  File "infer.py", line 144, in <module>
Using seed = 1
loading annotations into memory...
Done (t=0.70s)
creating index...
index created!
loading model checkpoint /mlperf/pretrained/resnet34-ssd300.pth
    main()age: 1/4952
  File "infer.py", line 141, in main
    eval_ssd300_mlperf_coco(args)
  File "infer.py", line 127, in eval_ssd300_mlperf_coco
    coco_eval(ssd300, val_coco, cocoGt, encoder, inv_map, args.threshold,args.device)
  File "infer.py", line 67, in coco_eval
    ploc, plabel = model(inp)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mlperf/pytorch/ssd300.py", line 125, in forward
    layers = self.model(data)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mlperf/pytorch/base_model.py", line 71, in forward
    layer1_activation = self.layer1(data)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 320, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
```
It looks like the input tensors and the model tensors are different. Some help?