zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Network Division GPT-j input tensor transfer
Root cause: Documentation Deficiency

Hi,

In v3.0, for BERT, we allowed LoadGen node to send InputTokenIDs tensors only and let SUT to reconstruct SegmentIDs/AttentionMask tensors.

Do we allow the similar thing in GPT-j for v3.1? I can see the same thing could be done.

@liorkhe FYI; also Lior: do you happen to know Gavin's handle from KRAI?

Hi,
The main reason as far as I recall was to allow the test to be similar to the server mode, since the BERT inputs were partitioned at a certain size so when sending the input and partitioning at the server created different inputs/outputs. If we have the same issue with GPT-J it makes sense, I think. 
The issue then was https://github.com/mlcommons/inference/issues/1291. As far as I know from that issue Gavin is https://github.com/G4V

Thanks Lior! Yes I see GPT-j tensors are similarly preprocessed. Would you help bringing this up in WG? I guess we may need to touch up the policy doc (https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#benchmarks-and-qsl-preprocessing)?

yes. but isn't https://github.com/mlcommons/inference_policies/pull/275 already covering it?

Oh I didn't know this policy update. :) Looks like it's already covering it! Thank you @liorkhe !

Closing this.
Repository: mlcommons/inference
Issue: Removed more-power-check flag
Root cause: Documentation Deficiency

Hi,

It seems that we removed more-power-check flag in this version of code?

https://github.com/mlcommons/inference/blob/da9b4d54bd2bd2ae26316f75987d71103122e3ca/tools/submission/README.md?plain=1#L58

Should we update the readme file of it?

Good catch, @xihajun. It looks that other flags have been added without documentation e.g. `--skip-meaningful-fields-emptiness-check`.

@xihajun yes, since `power-check` is mandatory and is automatically part of the submission-ui check this flag is removed now in the submission-checker. I have updated the README [here](https://github.com/mlcommons/inference/pull/1431)

@psyhtest yes, those flags are not useful though except for checking old results. I believe it is too complex to retain the backward compatibility of the submission checker - we should start using different versions of the submission checker for each round -- will make the checker quite simple.
Repository: mlcommons/inference
Issue: TEST04 takes reciprocal of scores for SingleStream/MultiStream
Root cause: Algorithmic Error

It is rather peculiar and confusing that for TEST04 ([only applicable to ResNet50](https://github.com/mlcommons/inference/issues/1350)), the comparisons for [SingleStream](https://github.com/mlcommons/inference/blob/master/compliance/nvidia/TEST04/verify_performance.py#L55) and [MultiStream](https://github.com/mlcommons/inference/blob/master/compliance/nvidia/TEST04/verify_performance.py#L61) are performed on score reciprocals (more precisely, `1e9 / float(score)`) rather than on scores directly like for TEST01 and TEST05 e.g.:

- SingleStream:
```
resnet50/singlestream/TEST04/verify_performance.txt:Verifying performance.
resnet50/singlestream/TEST04/verify_performance.txt-reference score = 1569.1221859754996
resnet50/singlestream/TEST04/verify_performance.txt-test score = 1571.568441019822
resnet50/singlestream/TEST04/verify_performance.txt-TEST PASS
--
resnet50/singlestream/TEST05/verify_performance.txt:Verifying performance.
resnet50/singlestream/TEST05/verify_performance.txt-reference score = 637299
resnet50/singlestream/TEST05/verify_performance.txt-test score = 638180
resnet50/singlestream/TEST05/verify_performance.txt-TEST PASS
--
resnet50/singlestream/TEST01/verify_performance.txt:Verifying performance.
resnet50/singlestream/TEST01/verify_performance.txt-reference score = 637299
resnet50/singlestream/TEST01/verify_performance.txt-test score = 638024
resnet50/singlestream/TEST01/verify_performance.txt-TEST PASS
```

- MultiStream:
```
resnet50/multistream/TEST04/verify_performance.txt:Verifying performance.
resnet50/multistream/TEST04/verify_performance.txt-reference score = 460.6963794332237
resnet50/multistream/TEST04/verify_performance.txt-test score = 469.96500640562306
resnet50/multistream/TEST04/verify_performance.txt-TEST PASS
--
resnet50/multistream/TEST05/verify_performance.txt:Verifying performance.
resnet50/multistream/TEST05/verify_performance.txt-reference score = 2170627
resnet50/multistream/TEST05/verify_performance.txt-test score = 2216725
resnet50/multistream/TEST05/verify_performance.txt-TEST PASS
--
resnet50/multistream/TEST01/verify_performance.txt:Verifying performance.
resnet50/multistream/TEST01/verify_performance.txt-reference score = 2170627
resnet50/multistream/TEST01/verify_performance.txt-test score = 2184797
resnet50/multistream/TEST01/verify_performance.txt-TEST PASS
```
Repository: mlcommons/inference
Issue: GPT-J: MD5 hash value mismatch the provided fine-tuned model
Root cause: Documentation Deficiency

Followed the [README](https://github.com/mlcommons/inference/blob/master/language/gpt-j/README.md) to download the fine-tuned model. But the MD5 hash(3a6050739456a3902d46bab0e5d8952b) value mismatches with the reference one(e677e28aaf03da84584bb3073b7ee315), while the accuracy of downloaded model looks fine. Could anyone double check and confirm on that?
<img width="1890" alt="image" src="https://github.com/mlcommons/inference/assets/43191160/d40247ed-a065-4468-80f3-3cda480e13f2">
<img width="1025" alt="image" src="https://github.com/mlcommons/inference/assets/43191160/8876d57f-e628-4ff4-bdfc-fc6c786b198f">



It seems the MD5 is different for different downloads. I have given [a PR](https://github.com/mlcommons/inference/pull/1431) to remove this in the README.
Repository: mlcommons/inference
Issue: Optional field for closed division marked as required
Root cause: Validation Gap

```
[2023-07-16 01:20:48,197 submission_checker.py:2872 ERROR] closed/NVIDIA/results/DGX-H100_H100-SXM-80GBx1_TRT, field host_memory_configuration requires a meaningful response but is empty
[2023-07-16 01:20:48,197 submission_checker.py:2872 ERROR] closed/NVIDIA/results/DGX-H100_H100-SXM-80GBx1_TRT, field host_networking requires a meaningful response but is empty
[2023-07-16 01:20:48,197 submission_checker.py:2872 ERROR] closed/NVIDIA/results/DGX-H100_H100-SXM-80GBx1_TRT, field host_networking_topology requires a meaningful response but is empty
[2023-07-16 01:20:48,197 submission_checker.py:2872 ERROR] closed/NVIDIA/results/DGX-H100_H100-SXM-80GBx1_TRT, field accelerator_host_interconnect requires a meaningful response but is empty
[2023-07-16 01:20:48,197 submission_checker.py:2872 ERROR] closed/NVIDIA/results/DGX-H100_H100-SXM-80GBx1_TRT, field accelerator_interconnect requires a meaningful response but is empty
[2023-07-16 01:20:48,197 submission_checker.py:2872 ERROR] closed/NVIDIA/results/DGX-H100_H100-SXM-80GBx1_TRT, field cooling requires a meaningful response but is empty
[2023-07-16 01:20:48,198 submission_checker.py:2865 ERROR] closed/NVIDIA/results/DGX-H100_H100-SXM-80GBx1_TRT, field host_networking_card_count is missing
[2023-07-16 01:20:48,198 submission_checker.py:2865 ERROR] closed/NVIDIA/results/DGX-H100_H100-SXM-80GBx1_TRT, field system_type_detail is missing 
```
These field are reported as missing by the submission checker.

@nvyihengz All these fields are mandatory as per the rules [here](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#system_desc_id-json-metadata). Except [system_type_detail](https://github.com/mlcommons/policies/pull/151/files) all others need a meaningful response. But if any field does not make sense on a given system, "N/A" is enough to satisfy the submission checker.
Repository: mlcommons/inference
Issue: ImageNet download by commercial submitter
Root cause: Documentation Deficiency

If a submitter tries to download ImageNet for RN50 manually now, they get the following URL and need to accept the terms of access listed there: https://image-net.org/download.php

The terms of access, for reference:

```
[RESEARCHER_FULLNAME] (the "Researcher") has requested permission to use the ImageNet database (the "Database") at Princeton University and Stanford University. In exchange for such permission, Researcher hereby agrees to the following terms and conditions:

* Researcher shall use the Database only for non-commercial research and educational purposes.
* Princeton University and Stanford University make no representations or warranties regarding the Database, including but not limited to warranties of non-infringement or fitness for a particular purpose.
* Researcher accepts full responsibility for his or her use of the Database and shall defend and indemnify the ImageNet team, Princeton University, and Stanford University, including their employees, Trustees, officers and agents, against any and all claims arising from Researcher's use of the Database, including but not limited to Researcher's use of any copies of copyrighted images that he or she may create from the Database.
* Researcher may provide research associates and colleagues with access to the Database provided that they first agree to be bound by these terms and conditions.
* Princeton University and Stanford University reserve the right to terminate Researcher's access to the Database at any time.
* If Researcher is employed by a for-profit, commercial entity, Researcher's employer shall also be bound by these terms and conditions, and Researcher hereby represents that he or she is fully authorized to enter into this agreement on behalf of such employer.
* The law of the State of New Jersey shall apply to all disputes under this agreement.
```

It seems like these terms were added sometime after March 2021. How does a commercial entity use ImageNet for MLPerf submission with these terms of access?

Hi Matthew from CTI here,

I have preemptively submitted for access of the dataset through the website,  since it could take 5 days to get a response from  ImageNet. We are trying to submit a benchmark for one of our boards and the question by nv-ananjappa is related to our current questions about the terms of access, and whether or not this can still be used to benchmark a commercial product.

Thanks

@mfischerCTI You might want to check with Legal in CTI to determine if your use of the ImageNet dataset for MLPerf benchmarking is compliant with the terms of access. From the info I got, the Legal of other companies have cleared it for use for their own benchmarking.

Okay thanks, I'll look into it
Repository: mlcommons/inference
Issue: Submission checker seems broken for cases where deterministic TEST01 fails
Root cause: Validation Gap

Currently getting the below error from submission checker.

```
[2023-07-27 21:15:01,346 submission_checker.py:3078 INFO] Compliance test accuracy check (deterministic mode) in closed/cTuning/compliance/gcp_nvidia_l4-nvidia_original-gpu-tensorrt-vdefault-default_config/bert-99.9/offline/TEST01 failed
Traceback (most recent call last):
  File "/home/arjun/CM/repos/local/cache/32b44644f91e4836/inference/tools/submission/submission_checker.py", line 3401, in <module>
  File "/home/arjun/CM/repos/local/cache/32b44644f91e4836/inference/tools/submission/submission_checker.py", line 3245, in main
  File "/home/arjun/CM/repos/local/cache/32b44644f91e4836/inference/tools/submission/submission_checker.py", line 2770, in check_results_dir
  File "/home/arjun/CM/repos/local/cache/32b44644f91e4836/inference/tools/submission/submission_checker.py", line 3220, in check_compliance_dir
  File "/home/arjun/CM/repos/local/cache/32b44644f91e4836/inference/tools/submission/submission_checker.py", line 3105, in check_compliance_acc_dir
KeyError: ('F1', 90.783126)

CM error: Portable CM script failed (name = run-mlperf-inference-submission-checker, return code = 256)
```
Repository: mlcommons/inference
Issue: Failure with 3d-unet accuracy script using Nvidia implementation
Root cause: Environment Incompatibility

While running nvidia implementation for 3.0 and then using the official accuracy script we are getting the below error on Nvidia Jetson Orin running Ubuntu 20.04. It worked fine on x86 machines using Ubuntu 22.04. 

```
Traceback (most recent call last):
  File "/mount/CM/repos/local/cache/cedb99c765c24b90/inference/vision/medical_imaging/3d-unet-kits19/accuracy_kits.py", line 320, in <module>
    main()
  File "/mount/CM/repos/local/cache/cedb99c765c24b90/inference/vision/medical_imaging/3d-unet-kits19/accuracy_kits.py", line 295, in main
    predictions = load_loadgen_log(log_file, output_dtype, target_files, aux)
  File "/mount/CM/repos/local/cache/cedb99c765c24b90/inference/vision/medical_imaging/3d-unet-kits19/accuracy_kits.py", line 250, in load_loadgen_log
    predictions = json.load(f)
  File "/usr/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 34 column 69204498 (char 2203058177)

CM error: Portable CM script failed (name = process-mlperf-accuracy, return code = 256)
```
Repository: mlcommons/inference
Issue: Submission checker is not counting "No results" for not truncated accuracy logs
Root cause: Validation Gap

If accuracy logs are not truncated, submission checker is outputting an error but not including this in "No Results". So, this error can be missed by the submitters. 

The checker also mentions that file size is large. That's a possible hint. I Don't remember there is an explicit fix it recommends.
Repository: mlcommons/inference
Issue: Waive TEST01 for GPT-J until compliance issues are resolved
Root cause: Validation Gap

Even when the accuracy run is compliant (meets accuracy of GPT-J), there is a chance that TEST01 can fail since the TEST01 accuracy compared to accuracy run can sometimes be better than 0.1% of reference accuracy. We investigated and found multiple reasons such as non-determinism in the ROUGE metric computation, variation in kernels dynamically chosen at runtime, TEST01 passing criteria `accuracy_log_sampling_target` being low or other undiscovered reasons in the workload/metric.

Since modifying TEST01 to better suit LLM workload/metric requires more time, we are requesting a waive on TEST01 for GPT-J.

The submitter could also be requested to submit full compliance logs for full inspection if they are opting for waive.

@nv-ananjappa I agree with it. During the review period, one can ask for any submitter's TEST01 log. 

Waived for v3.1
Repository: mlcommons/inference
Issue: Issue with accuracy logic in submission_checker when comparing multiple metrics
Root cause: Algorithmic Error

Given the following entries in a sample `accuracy.txt` for GPTJ
```
{ 
    'rouge1': 43, 
    'rouge2': 21,
    'rougeL': 30,
    'gen_len': 300000000000
}
```
I see the following warnings
```
[2023-08-04 16:36:08,376 submission_checker.py:1605 WARNING] gptj-99.9/Server/accuracy accuracy not met: expected=3615190.200000, found=43
[2023-08-04 16:36:08,376 submission_checker.py:1605 WARNING] gptj-99.9/Server/accuracy accuracy not met: expected=29.958112, found=21
[2023-08-04 16:36:08,376 submission_checker.py:1605 WARNING] gptj-99.9/Server/accuracy accuracy not met: expected=3615190.200000, found=21
[2023-08-04 16:36:08,376 submission_checker.py:1605 WARNING] gptj-99.9/Server/accuracy accuracy not met: expected=42.943514, found=21
[2023-08-04 16:36:08,376 submission_checker.py:1605 WARNING] gptj-99.9/Server/accuracy accuracy not met: expected=3615190.200000, found=30
[2023-08-04 16:36:08,376 submission_checker.py:1605 WARNING] gptj-99.9/Server/accuracy accuracy not met: expected=42.943514, found=30
```
I think this is because now that we have multiple metrics to compare, the logic mistakenly compares the wrong scores. i.e. it compares the actual `rouge1` in accuracy.txt versus the 99.9% target `gen_len=3615190.200000`.

@yeandy I think that just adding the last line should fix the issue:
```python
for line in f:
    for i, (pattern, acc_target) in enumerate(zip(patterns, acc_targets)):
        m = re.match(pattern, line)
        if m:
            acc = m.group(1)
        m = re.match(r"^hash=([\w\d]+)$", line)
        if m:
            hash_val = m.group(1)
        if acc is not None and float(acc) >= acc_target:
            all_accuracy_valid &= True
            acc_seen[i] = True
        elif acc is not None:
            all_accuracy_valid = False
            log.warning("%s accuracy not met: expected=%f, found=%s", path, acc_target, acc)
        acc = None
```
Here the comparison is made only if the pattern matches. The loop was taking the previous `acc` and making the comparison even if it didn’t match.

Would you be able to test this? Or do you already have an alternative solution in your PR?


I tested by adding `acc = None`, and it seems to work now. Thanks @pgmpablo157321 !

Let me make the update in the PR and tag your for review

@yeandy I notice an issue with that solution, it was returning the variable `acc` which was set to `None`.

I opened this [PR](https://github.com/yeandy/inference/pull/5) fixing the issue. Here we return the main/first metric. Could you merge those changes? After that I think this can be merge in the main repo.

Thanks for catching this. Tested myself, and merged.
Repository: mlcommons/inference
Issue: Not able to preprocess Criteo dataset using the reference scripts
Root cause: Validation Gap

As with old DLRM, the DLRMv2 preprocessing script is also tough to use. After spending 5 days on a server with 1 TB RAM and 7TB disk, the preprocessing script is giving the below error. By default, if the script is restarted it does all the processing from the beginning - another 5 days just for that. Is there any workaround here?


```
Output will be saved to
/home/cmuser/results_dir/44d6ff18ef3d47b2/tmp/temp_output_of_step_1/day_16_dense.npy
/home/cmuser/results_dir/44d6ff18ef3d47b2/tmp/temp_output_of_step_1/day_16_sparse.npy
/home/cmuser/results_dir/44d6ff18ef3d47b2/tmp/temp_output_of_step_1/day_16_labels.npy
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/cmuser/.local/lib/python3.10/site-packages/torchrec/datasets/scripts/npy_preproc_criteo.py", line 92, in <module>
    main(sys.argv[1:])
  File "/home/cmuser/.local/lib/python3.10/site-packages/torchrec/datasets/scripts/npy_preproc_criteo.py", line 81, in main
    BinaryCriteoUtils.tsv_to_npys(
  File "/home/cmuser/.local/lib/python3.10/site-packages/torchrec/datasets/criteo.py", line 237, in tsv_to_npys
    for (row_dense, row_sparse, row_label) in CriteoIterDataPipe(
  File "/home/cmuser/.local/lib/python3.10/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 185, in wrap_generator
    response = gen.send(request)
  File "/home/cmuser/.local/lib/python3.10/site-packages/torchrec/datasets/criteo.py", line 110, in __iter__
    yield from datapipe
  File "/home/cmuser/.local/lib/python3.10/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 185, in wrap_generator
    response = gen.send(request)
  File "/home/cmuser/.local/lib/python3.10/site-packages/torch/utils/data/datapipes/iter/callable.py", line 123, in __iter__
    yield self._apply_fn(data)
  File "/home/cmuser/.local/lib/python3.10/site-packages/torch/utils/data/datapipes/iter/callable.py", line 88, in _apply_fn
    return self.fn(data)
  File "/home/cmuser/.local/lib/python3.10/site-packages/torchrec/datasets/criteo.py", line 228, in row_mapper
    sparse = [
  File "/home/cmuser/.local/lib/python3.10/site-packages/torchrec/datasets/criteo.py", line 229, in <listcomp>
    int(row[i] or "0", 16)
IndexError: list index out of range
This exception is thrown by __iter__ of MapperIterDataPipe(datapipe=ReadLinesFromCSV, fn=row_mapper, input_col=None, output_col=None)
+ exit
```

It turns out that day_16 data was not completely downloaded. The local copy of [this script](https://github.com/pytorch/torchrec/blob/main/torchrec/datasets/scripts/npy_preproc_criteo.py#L63) had to be modified to make the preprocessing script resume from the previous location. or else it is another 5 extra days of waiting.
Repository: mlcommons/inference
Issue: Submission WebUI Still Check Compliance Tests For GPT-J
Root cause: Configuration Error

Based on the latest submission checker, the compliance tests for gptj have been dropped. I can pass the submission checking by adding --skip_compliance args. However, when I uploaded the gz file through the submission web-UI, I got the compliance dir missing error:
[2023-08-14 17:19:43,538 submission_checker.py:2771 ERROR] no compliance dir for closed/Dell/results/xxxx/Server
[2023-08-14 17:19:43,546 submission_checker.py:2771 ERROR] no compliance dir for closed/Dell/results/xxxx/gptj-99/Offline
Did the related PR get merged?

I see the PR being merged [here](https://github.com/mlcommons/inference/pull/1473) 

@taozhang9527 @arjunsuresh The PR avoids checking TEST01, but I think we are still checking for the [compliance folder](https://github.com/mlcommons/inference/blob/728680b0ab8ffc6ed7788db3cb5e51acd92dab7f/tools/submission/submission_checker.py#L2770-L2772).
I think uploading this folder with just some readme should pass the checker. Does this sound ok? Do you have any suggestions how we should fix this?

Hi @pgmpablo157321 , It may not be easy to inform all submitters to do this. So, maybe in the check for compliance folder can be changed as follows:
```
if not os.path.exists(compliance_dir) and "gpt-j" not in model_name:
```

Ok, that should work for now

When will this change happen? I just checked the web-UI submission. It still has the same error.

I guess it is "gptj*" instead of "gpt-j" to cover different gpt models (gptj-99, gptj-99.9 etc.)?


From: Arjun Suresh ***@***.***>
Sent: Tuesday, August 15, 2023 7:41 AM
To: mlcommons/inference
Cc: Zhang, Tao; Mention
Subject: Re: [mlcommons/inference] Submission WebUI Still Check Compliance Tests For GPT-J (Issue #1479)


[EXTERNAL EMAIL]

Hi @pgmpablo157321 [github.com]<https://urldefense.com/v3/__https:/github.com/pgmpablo157321__;!!LpKI!nym7Ayzy0JCyZxpMJWMJKGSvD0M0RxOuqRlkLFjPGByRFRG8s61F4BUCdQQ5wMB90MBqOGDJi9TI69bQvq74hU9pQQ$> , It may not be easy to inform all submitters to do this. So, maybe in the check for compliance folder can be changed as follows:

if not os.path.exists(compliance_dir) and "gpt-j" not in model_name:

-
Reply to this email directly, view it on GitHub [github.com]<https://urldefense.com/v3/__https:/github.com/mlcommons/inference/issues/1479*issuecomment-1679052494__;Iw!!LpKI!nym7Ayzy0JCyZxpMJWMJKGSvD0M0RxOuqRlkLFjPGByRFRG8s61F4BUCdQQ5wMB90MBqOGDJi9TI69bQvq69twEQsg$>, or unsubscribe [github.com]<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BBUKMMM3BIZW7DSKORQ657DXVOC7LANCNFSM6AAAAAA3QLKKGI__;!!LpKI!nym7Ayzy0JCyZxpMJWMJKGSvD0M0RxOuqRlkLFjPGByRFRG8s61F4BUCdQQ5wMB90MBqOGDJi9TI69bQvq4-z4lX4w$>.
You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>


Internal Use - Confidential