zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Error for generate_final_report.py with pandas2.0
Root cause: External Dependency Breakage

Today we are facing [this error](https://github.com/mlcommons/ck/actions/runs/4608918363/jobs/8145335876) in CM while generating the final xlsx report after an inference run.

```
Traceback (most recent call last):
  File "/home/runner/CM/repos/local/cache/d8e97e24297d49bf/inference/tools/submission/generate_final_report.py", line 233, in <module>
    sys.exit(main())
  File "/home/runner/CM/repos/local/cache/d8e97e24297d49bf/inference/tools/submission/generate_final_report.py", line 227, in main
    writer.save()
AttributeError: 'XlsxWriter' object has no attribute 'save'
```

This issue happens when we upgrade to pandas 2.0
Repository: mlcommons/inference
Issue: Enhancements to submission checker
Root cause: Validation Gap

We noticed the following enhancements which can be done to the submission checker

1. Not allowing empty README files
2. Checking for mandatory fields in measurements.json and ensuring they are non empty 
3. Checking for all mandatory files – power related mandatory measurement files are not checked as of now
4. Submission checker is counting improper results (they do [get discarded](https://github.com/mlcommons/inference/issues/1342) in the results table). 
5. Submission checker is counting invalid results - for example bert-99 multistream. These are not added to the results table but still can be misleading to the submitters
Repository: mlcommons/inference
Issue: AttributeError: module 'dlrm_data_pytorch' has no attribute 'collate_wrapper_criteo_offset'
Root cause: External Dependency Breakage

I am trying to execute run_local.sh script from mlcommons/inference/recommendation/dlrm/pytorch. Below is the command and error:

Command:
./run_local.sh pytorch dlrm terabyte cpu --max-ind-range 10000000

Output:
+ python python/main.py --profile dlrm-terabyte-pytorch --mlperf_conf ../../../mlperf.conf --model dlrm --model-path ~/mlcommons/inference/recommendation/model//dlrm_terabyte.pytorch --dataset terabyte --dataset-path ./fake_criteo --output ~/mlcommons/inference/recommendation/dlrm/pytorch/output/pytorch-cpu/dlrm --max-ind-range 10000000
INFO:main:Namespace(accuracy=False, backend='pytorch-native', cache=0, count_queries=None, count_samples=None, data_sub_sample_rate=0.0, dataset='terabyte', dataset_path='./fake_criteo', duration=None, find_peak_performance=False, inputs=['continuous and categorical features'], max_batchsize=2048, max_ind_range=10000000, max_latency=None, mlperf_bin_loader=False, mlperf_conf='../../../mlperf.conf', model='dlrm', model_path='~/mlcommons/inference/recommendation/model//dlrm_terabyte.pytorch', numpy_rand_seed=123, output='~/mlcommons/inference/recommendation/dlrm/pytorch/output/pytorch-cpu/dlrm', outputs=['probability'], profile='dlrm-terabyte-pytorch', samples_per_query_multistream=8, samples_per_query_offline=2048, samples_to_aggregate_fix=None, samples_to_aggregate_max=None, samples_to_aggregate_min=None, samples_to_aggregate_quantile_file=None, samples_to_aggregate_trace_file='dlrm_trace_of_aggregated_samples.txt', scenario='SingleStream', target_qps=None, test_num_workers=0, threads=24, use_gpu=False, user_conf='user.conf')
Using CPU...
Reading pre-processed data=./fake_criteo/terabyte_processed.npz
Sparse features= 26, Dense features= 13
Traceback (most recent call last):
  File "python/main.py", line 619, in <module>
    main()
  File "python/main.py", line 501, in main
    **kwargs)
  File "~/mlcommons/inference/recommendation/dlrm/pytorch/python/criteo.py", line 131, in __init__
    collate_fn=dp.collate_wrapper_criteo_offset,
AttributeError: module 'dlrm_data_pytorch' has no attribute 'collate_wrapper_criteo_offset'




It is present in [the latest DLRM code](https://github.com/facebookresearch/dlrm/blob/f06d3deed5a15710fb28e0ede0af31ebd53789cb/dlrm_data_pytorch.py). I guess the README is wrongly pointing to the MLCommons training repository and not the official DLRM repository. 

@arjunsuresh  Could you please provide link to official DLRM repo which should be used instead of git clone --recurse-submodules https://github.com/mlcommons/training.git ?

Actually this PR got deleted by mistake. Changes can be seen here: https://github.com/mlcommons/inference/pull/1265/files

Noted. Thank you @arjunsuresh .
Repository: mlcommons/inference
Issue: torch.cuda.OutOfMemoryError: CUDA out of memory. 
Root cause: Resource Mishandling

when I excude this command:
./run_local.sh pytorch dlrm terabyte gpu --scenario Server  --max-ind-range=40000000 --samples-to-aggregate-quantile-file=./tools/dist_quantile.txt 

then:
Using 8 GPU(s)...
Reading pre-processed data=/home/user01/file/node05/joyecai/nfs_share/dlrm_raw_dataset/terabyte_processed.npz
Sparse features= 26, Dense features= 13
Using variable query size: custom distribution (file ./tools/dist_quantile.txt)
Traceback (most recent call last):
  File "/home/user01/file/node05/joyecai/code/mlcommons/inference/recommendation/dlrm/pytorch/python/main.py", line 619, in <module>    main()
  File "/home/user01/file/node05/joyecai/code/mlcommons/inference/recommendation/dlrm/pytorch/python/main.py", line 503, in main
    model = backend.load(args.model_path, inputs=args.inputs, outputs=args.outputs)
  File "/home/user01/file/node05/joyecai/code/mlcommons/inference/recommendation/dlrm/pytorch/python/backend_pytorch_native.py", line 61, in load
    dlrm = dlrm.to(self.device)  # .cuda()
  File "/home/user01/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1145, in to
    return self._apply(convert)
  File "/home/user01/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "/home/user01/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "/home/user01/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 820, in _apply    param_applied = fn(param)  File "/home/user01/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1143, in convert    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 18.91 GiB (GPU 0; 79.18 GiB total capacity; 70.34 GiB already allocated; 8.00 GiB free; 70.35 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

@ColaDrill What were the RAM and GPUs of the machine you tried to run the benchmark on?

> 1008GB RAM and 80GB x 8 GPU (NVIDIA A800)



> > 1008GB RAM and 80GB x 8 GPU (NVIDIA A800)

did you fix oom issues?
Repository: mlcommons/inference
Issue: Recommendation inference code uses a method that doesn't exist in the training code
Root cause: Interface Contract Mismatch

The code below requires a method dp.collate_wrapper_criteo_offset

https://github.com/mlcommons/inference/blob/f174de72973be7f3c720ef4b5b38c77ef672d0d3/recommendation/dlrm/pytorch/python/criteo.py#L131

The method doesn't exist in the specific DLRM commit included in the training code as submodule

https://github.com/facebookresearch/dlrm/blob/6d75c84d834380a365e2f03d4838bee464157516/dlrm_data_pytorch.py

Is this a known issue?



yes. that submodule path is not valid for inference. Please use [this](https://github.com/mlcommons/inference/pull/1265/files) change. But since DLRM is going to be changed to DLRMv2, the current code is not going to be used for any upcoming submissions (unless DLRMv2 upcoming is not done before the coe freeze date). 

ok. Thanks!

Why did your change get closed without being merged into main?



You're welcome. Probably that was an accidental mistake which went unnoticed until recently. 

Fixed in #1359
Repository: mlcommons/inference
Issue: Power related fields in the system description
Root cause: Documentation Deficiency

We have [these fields](https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L1276) mentioned as required in the submission checker for power submissions. But the descriptions for these are missing in the [submission rules](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc) and [inference policies for power](https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc). I think the below fields should be explained and added in the table [here](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#57-system_desc_idjson-metadata) to make it easier for the submitters. Some fields which are going to [power_settings](https://github.com/mlcommons/inference_results_v3.0/blob/main/closed/NVIDIA/measurements/A100-PCIe-80GBx8_TRT_MaxQ/resnet50/power_settings.adoc) file like `power_management` may also be omitted here.

- power_management
- filesystem
- boot_firmware_version
- management_firmware_version
- other_hardware
- number_of_type_nics_installed
- nics_enabled_firmware
- nics_enabled_os
- nics_enabled_connected
- network_speed_mbit
- power_supply_quantity_and_rating_watts
- power_supply_details
- disk_drives
- disk_controllers



These fields are not made mandatory in the submission checker as of now as they are not in any rules. Hence closing this issue.
Repository: mlcommons/inference
Issue: TensorFlow v1 API warning
Root cause: External Dependency Breakage

When running `vision/classification_and_detection/run_local.py` with TensorFlow as the backend, the following warning is produced...
```shell
WARNING:tensorflow:From /.../inference/vision/classification_and_detection/python/backend_tf.py:47: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
WARNING:tensorflow:From /.../python3-venv/lib/python3.10/site-packages/tensorflow/python/tools/strip_unused_lib.py:84:extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. 
Instructions for updating:
This API was designed for TensorFlow v1. See https://www.tensorflow.org/guide/migrate for instructions on how to migrate your 
code to TensorFlow v2.
WARNING:tensorflow:From /.../python3-venv/lib/python3.10/site-packages/tensorflow/python/tools/optimize_for_inference_lib.py:112: remove_training_nodes (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This API was designed for TensorFlow v1. See https://www.tensorflow.org/guide/migrate for instructions on how to migrate your code to TensorFlow v2.
```
Are there any plans to patch the code if it could soon stop working?

Can you please do a PR for the change?
Repository: mlcommons/inference
Issue: issue on deploying resnet_model to embedded devices
Root cause: Architectural Constraint

Hi，guys:
I got an error  on resnet_model/max_pooling2d/MaxPool when I try to deploy https://zenodo.org/record/4735647/files/resnet50_v1.onnx into embedded devices . 

The device model compiler need the same padding for top and bottom , left and right。

But from graph of resnet50_v1, the padding of MaxPool has an input (0,0,1,1) 
![image](https://github.com/mlcommons/inference/assets/10462414/f8739fb2-b9ad-4ad8-933c-3435aea1b7da)

From source of resnet_model.py，It has the "same" padding，Could I just set “valid” and use same weight to test the device?
![image](https://github.com/mlcommons/inference/assets/10462414/945959d6-c848-4bef-a6e0-96283b4ef73c)

Could anyone give me some advice?
Repository: mlcommons/inference
Issue: GPT-J: missing checkpoint and confusing README
Root cause: Documentation Deficiency

The script [https://github.com/mlcommons/inference/tree/master/language/gpt-j](https://github.com/mlcommons/inference/tree/master/language/gpt-j) refers to [https://github.com/badhri-intel/inference](https://github.com/badhri-intel/inference) - it asks to clone this repository instead of using mlcommons/inference. I think it's a mistake and we shouldn't refer to external github repositories for the code.

Moreover, [download_gptj.py](https://github.com/mlcommons/inference/blob/master/language/gpt-j/download_gptj.py) downloads [EleutherAI/gpt-j-6b](https://huggingface.co/EleutherAI/gpt-j-6b) which _has not been fine-tuned for downstream contexts in which language models are commonly deployed, such as writing genre prose, or commercial chatbots. This means GPT-J-6B will not respond to a given prompt the way a product like ChatGPT does_.

Here is an example:
```
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Summarize the following news article:

### Input:
(CNN)A suicide attacker detonated a car bomb near a police vehicle in the capital of southern Afghanistan's Helmand province on Tuesday, killing seven people and injuring 23 others, the province's deputy governor said. The attack happened at about 6 p.m. in the Bolan area of Lashkar Gah city, said Mohammad Jan Rasoolyar, deputy governor of Helmand. Several children were among the wounded, and the majority of casualties were civilians, Rasoolyar said. Details about the attacker's identity and motive weren't immediately available.

### Response:
(CNN)The suicide bomber detonated a car bomb near a police vehicle in the capital of southern Afghanistan's Helmand province on Tuesday, killing seven people and injuring 23 others, the province's deputy governor said. The attack happened at about 6 p.m. in the Bolan area of Lashkar Gah city, said Mohammad Jan Rasoolyar, deputy governor of Helmand. Several children were among the wounded, and the majority of casualties were civilians, Rasoolyar said. Details about the attacker's identity and motive weren't immediately available.

### Explanation:
The task is to summarize the news article
```

I believe that finetuned checkpoint should be publicly available so that everyone can reproduce results from the benchmark.

Could you publish reference scores in README?

Updated the README in the [PR](https://github.com/mlcommons/inference/pull/1386) - Modified the repo name and added reference model ROUGE scores. 


We are still working on publishing the fine-tuned model publicly. But we have already shared the checkpoint internally with the task force which you can try.
Repository: mlcommons/inference
Issue: GPT-J: Dataset issues
Root cause: Validation Gap

I have analyzed [download_cnndm.py](https://github.com/mlcommons/inference/blob/master/language/gpt-j/download_cnndm.py) and [dataset.py](https://github.com/mlcommons/inference/blob/master/language/gpt-j/dataset.py):

* validation dataset is used, not test. Why?
* there are 13368 examples which is expected ( (https://huggingface.co/datasets/cnn_dailymail#data-splits](https://huggingface.co/datasets/cnn_dailymail#data-splits) )
* Problem A: 399 examples are longer than 1919 tokens. Currently tokenizer trims end which means that in about 3% examples we are losing end `\n\n### Response:`. It may impact final accuracy score.
* Problem B: reference summary of 250 examples is shorter than 30 tokens.
* Problem C: reference summary of 591 examples is longer than 128 tokens.
* Problem D: two summaries are longer than full prompt

In total 1171 (almost 9%) examples do not meet our constraints.

I see that there were some attempts to address this issue. The code [download_cnndm.py lines 38-50](https://github.com/mlcommons/inference/blob/master/language/gpt-j/download_cnndm.py#L38) is dead. Could you remove dead code?

Also, very confusing is default number of `num_examples` which is set to 4869 in [main.py#L29](https://github.com/mlcommons/inference/blob/master/language/gpt-j/main.py#L29). Where does this number come from? Shouldn't we evaluate accuracy on the full dataset?

Dataset histograms:
![image](https://github.com/mlcommons/inference/assets/37601244/4cabb04a-4371-4c44-bbaf-fee582a45828)
![image](https://github.com/mlcommons/inference/assets/37601244/7484b3ae-8a81-4d82-9a21-a2ba0b8d5d1d)

The code used for generating my statistics:
```python
import json
from transformers import AutoTokenizer

def generate_prompt(sample):
    return (
        "Below is an instruction that describes a task, paired with an input that provides further context. "
        "Write a response that appropriately completes the request.\n\n"
        "### Instruction:\n" + sample['instruction'] + "\n\n### Input:\n" + sample['input'] + "\n\n### Response:"
    )

with open("cnn_eval.json") as f:
    dataset = json.load(f)

model_name = "EleutherAI/gpt-j-6B"
tokenizer = AutoTokenizer.from_pretrained(
            model_name,
            model_max_length=2048,
            padding_side="left",
            use_fast=False,)
tokenizer.pad_token = tokenizer.eos_token # WHY !?

for sample in dataset:
    summary_len = len(tokenizer(sample['output'])['input_ids'])
    prompt_len  = len(tokenizer(generate_prompt(sample))['input_ids'])
    print("{}, {}".format(
        prompt_len,
        summary_len
    ))
```

We have always used validation set and not the test set for MLPerf Inference benchmarking. I have removed the redundant code from download_cnndm.py and also updated the max_examples in main.py
Repository: mlcommons/inference
Issue: RNNT reference implement fails with latest librosa
Root cause: External Dependency Breakage

```
Notice:  A new release of pip is available: 23.0.1 -> 23.1.2
Notice:  To update, run: pip install --upgrade pip
/opt/hostedtoolcache/Python/3.9.17/x64/bin/python3 /home/runner/CM/repos/ctuning@mlcommons-ck/cm-mlops/script/get-generic-python-lib/detect-version.py > tmp-ver.out 2> tmp-ver.err
Traceback (most recent call last):
Dataset loaded with 4.36 hours. Filtered 1.02 hours. Number of samples: 2513
  File "/home/runner/CM/repos/local/cache/c7ff93ff161b44f4/inference/speech_recognition/rnnt/run.py", line 91, in <module>
    main()
  File "/home/runner/CM/repos/local/cache/c7ff93ff161b44f4/inference/speech_recognition/rnnt/run.py", line 56, in main
    sut = PytorchSUT(args.pytorch_config_toml, args.pytorch_checkpoint,
  File "/home/runner/CM/repos/local/cache/c7ff93ff161b44f4/inference/speech_recognition/rnnt/pytorch_SUT.py", line 59, in __init__
    self.audio_preprocessor = AudioPreprocessing(**featurizer_config)
  File "/home/runner/CM/repos/local/cache/c7ff93ff161b44f4/inference/speech_recognition/rnnt/pytorch/preprocessing.py", line 32, in __init__
    self.featurizer = FeatureFactory.from_config(kwargs)
  File "/home/runner/CM/repos/local/cache/c7ff93ff161b44f4/inference/speech_recognition/rnnt/pytorch/parts/features.py", line 259, in from_config
    return featurizer.from_config(cfg, log="log" in feat_type)
  File "/home/runner/CM/repos/local/cache/c7ff93ff161b44f4/inference/speech_recognition/rnnt/pytorch/parts/features.py", line 236, in from_config
    return cls(sample_rate=cfg['sample_rate'], window_size=cfg['window_size'],
  File "/home/runner/CM/repos/local/cache/c7ff93ff161b44f4/inference/speech_recognition/rnnt/pytorch/parts/features.py", line 126, in __init__
    librosa.filters.mel(sample_rate, self.n_fft, n_mels=nfilt, fmin=lowfreq,
TypeError: mel() takes 0 positional arguments but 2 positional arguments (and 3 keyword-only arguments) were given
```

Fixed now
Repository: mlcommons/inference
Issue: Run image classification benchmark on real imageNet2012 dataset
Root cause: Documentation Deficiency

Hi, since you haven't provided preprocess steps of `imagenet2012 validation dataset` for `vision/classfication and detection` benchmark. And I hope to run the benchmark on this real dataset. In your `Getting started` tutorial, you use `make_fake_imagenet.sh` script to generate a `val_map.txt` file. I was wondering, if I hope to use the real imagenet2012 validation dateset, do I need to generate a similar `val_map.txt` as well? 

If so, does the second column of this file mean the category of the image? I have downloaded the dataset, however, it seems that it only includes a bunch of images without labels. So I was wondering how should I get the category if the second column does mean the category?

Thank you very much for your help!

Here's the `val_map.txt` generated by `make_fake_imagenet.sh` (Originial includes 8 rows, here I pasted only 2 rows of the file):
```
val/800px-Porsche_991_silver_IAA.jpg 817
val/512px-Cacatua_moluccensis_-Cincinnati_Zoo-8a.jpg 89
```

Is [this](https://www.dropbox.com/s/92n2fyej3lzy3s3/caffe_ilsvrc12.tar.gz?file_subpath=%2Fval.txt) what you're looking at?

For using the full dataset we need to get [imagenex-aux](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-aux) in addition to the [imagenet-val dataset. ](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-val)

Yes, the links are very helpful. Thank you!
Repository: mlcommons/inference
Issue: Update submission checker for v3.1
Root cause: Unimplemented Feature Gap

- [x] Add v3.1 config
- [x] Add DLRMv2 checks
- [x] Add GPT-J checks, with additional check for generation length (multiple metric)
- [x] Update to the last power version
- [x] Update seeds

@pgmpablo157321 The update to the latest power version can happen with [this PR](https://github.com/mlcommons/power-dev/pull/316) unless the power WG updates anything else. 

@pgmpablo157321 We need to add GPT-J details [here](https://github.com/mlcommons/inference/blob/master/mlperf.conf) right?

@arjunsuresh I am not sure, I see there is another conf file [here](https://github.com/mlcommons/inference/blob/master/language/gpt-j/mlperf.conf) that also doesn’t have the gpt-j details.

@badhri-intel @rnaidu02 Do you know if we need to put any values for gpt-j here?
Repository: mlcommons/inference
Issue: Reduce the compliance run times for GPT-J/GPT-3
Root cause: Configuration Error

Since GPT-J is taking too long to run, we are requesting the min_queries of the Offline scenario to be reduced from 24756 to say 1000 for compliance tests. Please review if this is possible. 

@psyhtest @ashwin @pgmpablo157321 
Please review. 

As discussed in the WG meeting on 27/June TEST05 is skipped for GPT-J models for inference 3.1. Will revisit this issue after inference 3.1 submissions.
Repository: mlcommons/inference
Issue: inputs and outputs parameter of vision/classification and detection benchmark
Root cause: Documentation Deficiency

Hi, in the README of `vision/classifcation and detection` benchmark, there are two parameters:

`--inputs INPUTS` comma separated input name list in case the model format does not provide the input names. This is needed for tensorflow since the graph does not specify the inputs.

`--outputs OUTPUTS` comma separated output name list in case the model format does not provide the output names. This is needed for tensorflow since the graph does not specify the outputs.

I'm a little confused about the meaning of `--outputs`, does that mean the outputs of inference? If so, how should I know the output of model inference?
And if I use tensorflow, do I must provide these two parameters? It would be better if you could provide an example of what these two parameters should look like on Imagenet dataset. 

Thank you very much!

Preparation of the required inputs (like models, datasets etc) for the different backends is done in CM. You can find the instructions to run different backends like `onnxruntime` or `tf` [here](https://github.com/mlcommons/ck/blob/master/docs/mlperf/inference/resnet50/README_reference.md). By running them, you can see the exact run commands being generated.

`main.py` is called by [this script](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/run_local.sh) which takes care of the inputs/outputs.