zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Add CK2/CM based workflows for all the MLPerf Inference tasks
Root cause: Unimplemented Feature Gap

This issue is created to keep track of the CK2/CM based workflows for all the MLPerf inference workflows being prepared by the [MLCommons Taskforce on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
- [x] Do end-to-end submission generation of all the MLPerf Inference reference implementations using the latest software stack
  - [x] ResNet50 ([GitHub action added](https://github.com/mlcommons/inference/blob/master/.github/workflows/test-resnet50.yml))
    - [x] Onnxruntime CPU
    - [x] Tensorflow CPU
    - [x] Onnxruntime CUDA
    - [x] Tensorflow CUDA
  - [x] RetinaNet ([GitHub action added](https://github.com/mlcommons/inference/blob/master/.github/workflows/test-retinanet.yml))
    - [x] Onnxruntime CPU
    - [x] Pytorch CPU
    - [x] Onnxruntime CUDA
    - [x] Pytorch CUDA
  - [x] Bert
    - [x] Onnxruntime CPU
    - [x] Pytorch CPU
    - [x] Tensorflow CPU
    - [x] Onnxruntime CPU Quantized
    - [x] Onnxruntime CUDA/ROCm
    - [x] Pytorch CUDA/ROCm
    - [x] Tensorflow CUDA/ROCm
    - [x] Onnxruntime CUDA/ROCm Quantized
  - [x] 3d-Unet
    - [x] Onnxruntime CPU
    - [x] Pytorch CPU
    - [x] Tensorflow CPU
    - [x] Onnxruntime CUDA
    - [x] Pytorch CUDA
    - [x] Tensorflow CUDA
  - [x] Rnnt
    - [x] Pytorch CPU
    - [x] Pytorch CUDA/ROCm
  - [x] DLRMv2
    - [x] Pytorch CPU
    - [ ] Pytorch CUDA
- [x] Demonstrate how to generate `user.conf` files for all the required MLPerf scenarios
- [x] Adding power workflow for all the reference implementations
- [x] Add Nvidia implementation in CM workflow
  - [x] ResNet50
  - [x] RetinaNet
  - [x] Bert
  - [x] 3d-Unet
  - [x] Rnnt
  - [x] DLRMv2
- [x] Add Intel implementation in CM workflow
  - [x] ResNet50
  - [x] RetinaNet
  - [x] Bert
  - [x] 3d-Unet
  - [ ] Rnnt
  - [x] DLRMv2

I close this ticket. The new progress report from the automation and reproducibility TF is available [here](https://github.com/mlcommons/inference/issues/1559)
Repository: mlcommons/inference
Issue: Retinanet reference implementation fails for pytorch 1.13
Root cause: External Dependency Breakage

The current retinanet reference implementation is failing with pytorch 1.13 

```
/home/arjun/.local/lib/python3.10/site-packages/torch/serialization.py:779: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)
  warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
Traceback (most recent call last):
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/vision/classification_and_detection/python/main.py", line 615, in <module>
    main()
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/vision/classification_and_detection/python/main.py", line 494, in main
    model = backend.load(args.model, inputs=args.inputs, outputs=args.outputs)
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/vision/classification_and_detection/python/backend_pytorch_native.py", line 27, in load
    self.model = torch.load(model_path, map_location=lambda storage, loc: storage)
  File "/home/arjun/.local/lib/python3.10/site-packages/torch/serialization.py", line 783, in load
    return torch.jit.load(opened_file, map_location=map_location)
  File "/home/arjun/.local/lib/python3.10/site-packages/torch/jit/_serialization.py", line 156, in load
    map_location = validate_map_location(map_location)
  File "/home/arjun/.local/lib/python3.10/site-packages/torch/jit/_serialization.py", line 176, in validate_map_location
    raise ValueError(
ValueError: map_location should be either None, string or torch.device, but got type: <class 'function'>
```
Repository: mlcommons/inference
Issue: A bug in submission_checker

We have this bug in the submission checker (trying to print the name of empty dirs) when the submission folder is having empty dirs. 
```
python3 /home/arjun/CM/repos/local/cache/c1b19aef683e444c/inference/tools/submission/submission_checker.py --input '/home/arjun/mlperf_submission' --submitter 'octoml'
Traceback (most recent call last):
  File "/home/arjun/CM/repos/local/cache/c1b19aef683e444c/inference/tools/submission/submission_checker.py", line 2479, in <module>
    sys.exit(main())
  File "/home/arjun/CM/repos/local/cache/c1b19aef683e444c/inference/tools/submission/submission_checker.py", line 2451, in main
    results = check_results_dir(config, args.submitter, args.skip_compliance,
  File "/home/arjun/CM/repos/local/cache/c1b19aef683e444c/inference/tools/submission/submission_checker.py", line 1952, in check_results_dir
    "%s has the following empty directories: %s", name, empty_dirs
UnboundLocalError: local variable 'name' referenced before assignment
```
Repository: mlcommons/inference
Issue: Strange workflow behaviour
Root cause: Documentation Deficiency

When I initially opened this [PR](https://github.com/mlcommons/inference/pull/1308), the CM checks failed:
<br>
<img width="500" alt="Screen Shot 2023-01-10 at 10 21 10 AM" src="https://user-images.githubusercontent.com/30699846/211597826-5eb87f75-af87-4b40-bee2-daae7890315f.png">
<br>
And I got an error that had nothing to do with my changes:
<br>
<img width="600" alt="Screen Shot 2023-01-10 at 10 20 30 AM" src="https://user-images.githubusercontent.com/30699846/211598477-d00090df-4cdc-444e-97bb-5c08934d481e.png">
<br>
After trying a couple of fixes, I updated the [main branch of my repo](https://github.com/pgmpablo157321/inference/tree/master), and the all the checks were suddenly successful. But the PR was not made from that branch. All of that means:

1. The PR wanted to merge changes from `pgmpablo157321:fix_submission_checker` to `master`
2. Initially, it failed the CM checks
3. When I updated the branch `pgmpablo157321:master` all the checks were suddenly successful.

@arjunsuresh Do you know what is happening here? Are the workflows taking some information from the master branch of the source repo?



@pgmpablo157321  Yes, the tests by default takes the repo from which the PR is coming for running. So this behaviour is expected.
Repository: mlcommons/inference
Issue: Bug in generate_final_report.py
Root cause: Validation Gap

The following error occurs when the column `#a` is an integer column:
```
Traceback (most recent call last):
  File "/Users/pablogonzalez/Desktop/MLCommons/inference/tools/submission/generate_final_report.py", line 233, in <module>
    sys.exit(main())
  File "/Users/pablogonzalez/Desktop/MLCommons/inference/tools/submission/generate_final_report.py", line 52, in main
    df['a#'] = df['a#'].apply(lambda x: int(x) if x.isnumeric() else x)
  File "/Users/pablogonzalez/opt/anaconda3/envs/mlperf-infra/lib/python3.9/site-packages/pandas/core/series.py", line 4433, in apply
    return SeriesApply(self, func, convert_dtype, args, kwargs).apply()
  File "/Users/pablogonzalez/opt/anaconda3/envs/mlperf-infra/lib/python3.9/site-packages/pandas/core/apply.py", line 1082, in apply
    return self.apply_standard()
  File "/Users/pablogonzalez/opt/anaconda3/envs/mlperf-infra/lib/python3.9/site-packages/pandas/core/apply.py", line 1137, in apply_standard
    mapped = lib.map_infer(
  File "pandas/_libs/lib.pyx", line 2870, in pandas._libs.lib.map_infer
  File "/Users/pablogonzalez/Desktop/MLCommons/inference/tools/submission/generate_final_report.py", line 52, in <lambda>
    df['a#'] = df['a#'].apply(lambda x: int(x) if x.isnumeric() else x)
AttributeError: 'int' object has no attribute 'isnumeric'
```
Repository: mlcommons/inference
Issue: Sample mean in place of population mean used in 3D UNet InstanceNorm3d 

Assume we have a tensor with shape 1×32×128×128×128, 1 is batch and 32 is channels. We need to calculate mean and variance of each channel. Normal method is to add all data in 128×128×128 to calculate population mean and variance. However, the new approach in consideration is to add partial data of 128×128×128 to calculate sample mean and variance that still meets the accuracy requirement.

Is this new approach allowed? 

Since the calculation is not mathematically equivalent, it is not allowed.

This is not allowed in closed division but can be done in open division right?

Yes.
Repository: mlcommons/inference
Issue: How to measure system power locally?
Root cause: Documentation Deficiency

Hi, I am wondering if there are any instructions on how to measure the inference power consumption on local machines. I checked some documents but seems it is quite unclear on this part.

Please see https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc 

Do you already have a power meter?

 Hi, @psyhtest , thanks for the quick reply. According to the link, I see it requires a power meter and an additional PC to log the power data. I am wondering if there are any links that can direct me to the power measure code scripts, for that in the document you refer to, links are invalid in sections 3 and 4 (e.g. Power Measurement BKM, Power Measurement Flow, power automation tool).

> Do you already have a power meter?

Are there any special requirements or recommendation for the power meter?

Hi @YJHMITWEB please see the below link for the power measurement documentation. Power analyzer requirements are also given there. 
https://github.com/octoml/ck/blob/master/docs/tutorials/mlperf-inference-power-measurement.md


Thanks!
Repository: mlcommons/inference
Issue: 3d-unet failing with numpy 1.24
Root cause: External Dependency Breakage

3d-unet reference implementation is failing with numpy 1.24. Works fine with numpy 1.21.
```
Loading file case_00189
Traceback (most recent call last):
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/vision/medical_imaging/3d-unet-kits19/run.py", line 177, in <module>
    main()
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/vision/medical_imaging/3d-unet-kits19/run.py", line 158, in main
    lg.StartTestWithLogSettings(sut.sut, sut.qsl.qsl, settings, log_settings)
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/vision/medical_imaging/3d-unet-kits19/base_SUT.py", line 173, in issue_queries
    final_result, mystr = self.infer_single_query(query, mystr)
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/vision/medical_imaging/3d-unet-kits19/inference_utils.py", line 151, in get_latency
    result, mystr = function(*args, **kw)
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/vision/medical_imaging/3d-unet-kits19/base_SUT.py", line 151, in infer_single_query
    result, norm_map = self.from_tensor(
  File "/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/vision/medical_imaging/3d-unet-kits19/pytorch_SUT.py", line 94, in from_tensor
    return my_tensor.cpu().numpy().astype(np.float)
  File "/home/arjun/.local/lib/python3.10/site-packages/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'float'. Did you mean: 'cfloat'?
/home/arjun/CM/repos/octoml@ck/cm-mlops/script/benchmark-program/run.sh: line 16: 2102208 Segmentation fault      (core dumped) /usr/bin/python3 run.py --backend=pytorch --scenario=Offline --mlperf_conf '/home/arjun/CM/repos/local/cache/a127f06d39034768/inference/mlperf.conf' --model=/home/arjun/CM/repos/local/cache/164337fa3d07427e/3dunet_kits19_pytorch.ptc --preprocessed_data_dir=/home/arjun/CM/repos/local/cache/a5f9082e085849d0 --user_conf '/home/arjun/CM/repos/octoml@ck/cm-mlops/script/app-mlperf-inference/tmp/1bb03286a6b44250873ef1cbe4cf142b.conf'

```
Repository: mlcommons/inference
Issue: Submission checker error with 3d-unet
Root cause: Configuration Error

Got the following error when run submission checker. 

[2023-02-08 05:37:40,235 submission_checker.py:1645 ERROR] closed/Intel/results/1-node-2S-SPR-PyTorch-INT8/3d-unet-99.9/Offline/performance/run_1/mlperf_log_detail.txt performance_sample_count, found 0, needs to be >= 43
[2023-02-08 05:37:40,235 submission_checker.py:1696 INFO] Target latency: None, Latency: 14079230289064, Scenario: Offline
[2023-02-08 05:37:40,235 submission_checker.py:2336 ERROR] closed/Intel/results/1-node-2S-SPR-PyTorch-INT8/3d-unet-99.9/Offline/performance/run_1 has issues

While in mlperf.conf https://github.com/mlcommons/inference/blob/c4a19872d9e7ba2fe2d5b8a4c3d3c02e82233785/mlperf.conf#L14 performance_sample_count_override is 0. What is the correct performance_sample_count_override for 3dunet in 3.0?
# set to 0 to let entire sample set to be performance sample
3d-unet.*.performance_sample_count_override = 0



This error seems very weird, since if you set `performance_sample_count_override`, the `performance_sample_count` should be the size of the dataset.
https://github.com/mlcommons/inference/blob/c4a19872d9e7ba2fe2d5b8a4c3d3c02e82233785/loadgen/test_settings_internal.cc#L117-L119

For 3d-unet this should be 43. You could try to set this value to 43 and see if this solves your issue, but it should work with 0.

Any updates on this issue?


@rnaidu02 Can this be closed?
Repository: mlcommons/inference
Issue: Proposal: Copy the check.py from power-dev repo to inference repo
GitHub labels: Inference v3.1

I'm suggesting to move [this file](https://github.com/mlcommons/power-dev/blob/master/compliance/check.py) from the power-dev repository to the inference repository. If we do this then the submitters can rely on power-dev as a separate repository and submission checker can run without relying on a git submodule - which as we have seen causes a lot of issues and prone to mistakes among the submitters. Moreover this will mean that inference repository code freeze has no more dependency on the power-dev repository - both repositories just need to have a commit hash which every submitter can start using independently. 

I like this idea. But maybe too late for 3.0 since we are in code freeze?

Sure. This proposal is for 3.1 onwards. 

[PR 1347](https://github.com/mlcommons/inference/pull/1347) has effected this change. So closing this issue.
Repository: mlcommons/inference
Issue: Misleading warning in submission checker for power 
Root cause: Validation Gap

The below ERROR is not even a warning -- it is completely expected during the ranging stage. Instead of treating this error as a warning, it must not even be reported. The current message is confusing to the submitters. 

```
[x] Check errors and warnings from PTD logs
	'02-21-2023 14:56:24.328: ERROR: Bad watts reading nan from WT310' in ptd_log.txt during ranging stage. Treated as WARNING

[x] Check PTD configuration
[x] Check debug is disabled on server-side

All checks passed. Warnings encountered, check for audit!
```

This should be fixed in [this PR](https://github.com/mlcommons/power-dev/pull/298)
Repository: mlcommons/inference
Issue: loadgen 3.0  py_demo_offline.py Segmentation Fault
Root cause: Resource Mishandling

Hi Pals,

I have built the mlperf-loadgen v3.0 module and installed it in python 3.9.15.
I executed the py_demo_offline.py example, which ran into a **segmentation fault**.
All the other demo.py (py_demo_single_stream.py, py_demo_multi_stream.py, py_demo_server.py) work perfectly.

Is there any solution?

Sincerely,
Kevin

Hi Kevin,
Can you please point to the py_demo_offline.py which you are referring to?

Any update for this, I have the same segment fault issue for running loadgen/demos/py_demo_offline.py @arjunsuresh 

Fatal Python error: Segmentation fault

Thread 0x00007fe34a258640 (most recent call first):
  File "/workspace/dev/inference/loadgen/demos/py_demo_offline.py", line 40 in process_query_async
  File "/opt/.pyenv/versions/3.9.10/lib/python3.9/threading.py", line 910 in run
  File "/opt/.pyenv/versions/3.9.10/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/opt/.pyenv/versions/3.9.10/lib/python3.9/threading.py", line 930 in _bootstrap

Thread 0x00007fe34c64cb80 (most recent call first):
  File "/workspace/dev/inference/loadgen/demos/py_demo_offline.py", line 71 in main
  File "/opt/.pyenv/versions/3.9.10/lib/python3.9/site-packages/absl/app.py", line 254 in _run_main
  File "/opt/.pyenv/versions/3.9.10/lib/python3.9/site-packages/absl/app.py", line 308 in run
  File "/workspace/dev/inference/loadgen/demos/py_demo_offline.py", line 77 in <module>
Segmentation fault (core dumped)

It might be related with my retinanet offline does not work either

@arjunsuresh  any update for this, i still encounter this bug when run `python loadgen/demos/py_demo_offline.py` :

Fatal Python error: Segmentation fault

Thread 0x00007f115d49e700 (most recent call first):
  File "/lab-mlperf-inference/code/llama2-70b-99.9/test_VllmFp8/inference/loadgen/demos/py_demo_offline.py", line 40 in process_query_async
  File "/opt/conda/envs/py_3.9/lib/python3.9/threading.py", line 917 in run
  File "/opt/conda/envs/py_3.9/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/opt/conda/envs/py_3.9/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007f115f8f24c0 (most recent call first):
  File "/lab-mlperf-inference/code/llama2-70b-99.9/test_VllmFp8/inference/loadgen/demos/py_demo_offline.py", line 72 in main
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/absl/app.py", line 254 in _run_main
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/absl/app.py", line 308 in run
  File "/lab-mlperf-inference/code/llama2-70b-99.9/test_VllmFp8/inference/loadgen/demos/py_demo_offline.py", line 78 in <module>
Segmentation fault (core dumped)

To solve this problem, simply set `settings.offline_expected_qps = 250`
Repository: mlcommons/inference
Issue: Power checker fails when there is >5% time delta gap between ranging and testing
Root cause: Validation Gap

We are having a failure with the power checker when there is a gap in performance between the ranging and testing mode runs. This is happening on a 16 core machine where Bert is run only on CPU and the runs (each mode) are nearly 2 hours long and that should be why the system performance is going down. The behaviour is consistent across runs and so even repeated runs are not helping. The power checker is failing when there is a >=5% time gap between the ranging and testing mode runs. 

PS: We are not using sleep call in between the ranging and testing mode runs. So CPU frequency might be dropping after the ranging mode runs. Also we are not doing any core pinning - so the performance behaviour is not very deterministic. 

## Ranging

```
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 4.54472
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
```

## Testing 
```
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 4.03963
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
```
Repository: mlcommons/inference
Issue: DLRM Inference fails:KeyError
Root cause: Validation Gap

Hi,
I am using the Terabyte dataset and the 12GB pre-trained model you published. I am running inference locally so I used `run_local.sh` script. I first ran the script to pre-process the dataset. It successfully pre-processed 21 days and then failed (I think due to memory limitations). Then I started pre-processing again and since the pre-processed files for the first 21st days were already there, the 3 remaining days were pre-processed. Finally I have all the 24 days pre-processed but now when I use them to run inference I get this error:
`File "$HOME/training/recommendation/dlrm/data_utils.py", line 147, in processCriteoAdData
    X_cat_t[j, k] = convertDicts[j][x]
KeyError: 1637495`
I am running in a conda virtual env with pytorch 1.10 and numpy 1.19 because it has been mentioned in other issues we'd better use those versions. The command I use for doing inference (initially used the same command for pre-processing) is this:
`./run_local.sh pytorch dlrm terabyte cpu --scenario Offline --max-ind-range=10000000 --data-sub-sample-rate=0.875 --samples-to-aggregate-quantile-file=./tools/dist_quantile.txt --max-batchsize=2048`

Can you please help with this issue?

Thanks!

@pgmpablo157321 Could you help?
Repository: mlcommons/inference
Issue: TEST04 update for Inference v3.1
Root cause: Unimplemented Feature Gap

Currently TEST04 consists of running performance mode with only one sample as follows:
- Offline scenario, the same sample is repeated as many times as necessary to fill the query
- Single-Stream/MultiStream/Server scenario test works by sending the same query as many times as necessary to satisfy a minimum duration of 10 minutes.

This creates the issue that the test doesn’t apply for benchmarks with a datasets with very unbalanced sample input sizes or very unbalanced processes times. Currently the test only applies for `ResNet50`.

**Requirements:**
- Be more applicable for benchmarks beyond `ResNet50`. This translates into tackling the input size / processing time variation issue.

**Initial Proposal**
- Select one batch (size ~8 or 16) of samples and run performance mode alternating between the samples in the batch.

**Other considerations**
- Is there a specific way of choosing the samples (E.g: choosing random samples, choosing samples close to the average time)
- Experiment with the test to ensure the it passes reliably.
- It is likely that for some models (particularly 3d-Unet) this issue has no solution, due to the fact that the input length varies a lot and there are few samples in the dataset.

This is **not** the final proposal to fix this test. Please feel free to discuss here other possible ways to solve this issue

I think we can do something like this. I'm considering retinanet as an example here. Here, `performance_sample_count=64` and so take 64 unique inputs.

Repeat inp1 N times, followed by inp2 N times and so on. Let the time taken be t1. (Maximum cache hits)

Then run inp1 followed by inp2 ... inp64 and repeat this N times. Let the time taken be t2. (Minimum cache hits)

Compare t1 and t2 for compliance. 

Minimum value of N should be 2 to ensure t1 is benefitting from caching. In the worst case, the runtime will be 4 times the accuracy run time as there are two runs and each run is doing twice the performance_sample_count number of inputs. Submitters should be free to increase the value of N on faster devices as a short runtime can cause larger variation and test failure. 

This modification can make TEST04 applicable to all the benchmarks. 

Ideally this test can be applicable to: `rnnt`, `bert`, `dlrm`, `retinanet`
Repository: mlcommons/inference
Issue: Inconsistency in the system desc required fields for cloud systems
Root cause: Validation Gap

The [submission rules](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#57-system_desc_idjson-metadata) say that we should use `vcpu_count` for cloud systems but the [submission checker](https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L1260) is making `core_count` as mandatory. I think these should be made consistent.