Repository: mlcommons/inference
Issue: [Bug] SDXL CLIP delta for TEST01 should be 0.2%, not 1%
Root cause: Configuration Error

[Part III](https://github.com/mlcommons/inference/tree/master/compliance/nvidia/TEST01#part-iii) of compliance TEST01 specifies:

> For each target accuracy metric, the delta between the two accuracy results [baseline and compliance] should be within (1-x)%.
> For example, for 99.9% accuracy metric, the delta of the accuracy results should be within 0.1%.

For each accuracy metric, the delta between the metric value from the baseline run and the metric value from the compliance runs must be within a specified value, `required_delta_perc`.

[For SDXL](https://github.com/mlcommons/inference/blob/2035f3f8982fd8e689abfae1ff18fe6ee2dde2b8/tools/submission/submission_checker.py#L219), the delta for the CLIP metric is set to 1%, and the delta for the FID metric to 2%:

<pre>
        "accuracy-delta-perc": {
            "stable-diffusion-xl": {"CLIP_SCORE": 1, "FID_SCORE": 2}
        },
</pre>

However, the accuracy ranges for SDXL, `[23.01085758, 23.95007626]` for FID and `[31.68631873, 31.81331801]` for CLIP, were selected by taking &plusmn;2.0% for FID and &plusmn;0.2% for CLIP from the reference values.

<details><summary>Maths</summary>
- FID: ((23.95007626 - 23.01085758) / 2) / 23.01085758  * 100% = 2.0%
- CLIP: ((31.81331801 - 31.68631873) / 2) / 31.68631873 * 100% = 0.2%
</details>

So the CLIP metric delta should be reduced to 0.2%:

<pre>
        "accuracy-delta-perc": {
            "stable-diffusion-xl": {"CLIP_SCORE": 0.2, "FID_SCORE": 2.0}
        },
</pre>

A couple of v5.1 SDXL submissions that required Part III would not pass under the 0.2% delta for CLIP.

<details><summary>v5.1 SDXL submissions that required Part III (found by the failures).</summary><pre>
$ find inference_results_v5.1/closed -name verify_accuracy.txt -exec grep -H FAIL {} \; | grep stable-diffusion-xl
closed/GigaComputing/compliance/G894-SD1_B200-SXM-180GBx8_TRT/stable-diffusion-xl/Server/TEST01/verify_accuracy.txt:TEST FAIL
closed/GigaComputing/compliance/G894-SD1_B200-SXM-180GBx8_TRT/stable-diffusion-xl/Offline/TEST01/verify_accuracy.txt:TEST FAIL
closed/The_Stage/compliance/h100_x8/stable-diffusion-xl/offline/TEST01/verify_accuracy.txt:TEST FAIL
closed/The_Stage/compliance/h100_x8/stable-diffusion-xl/server/TEST01/verify_accuracy.txt:TEST FAIL
closed/Dell/compliance/XE7740_L40S_PCIe_48GBx8_TRT/stable-diffusion-xl/Offline/TEST01/verify_accuracy.txt:TEST FAIL
</pre></details>
<details><summary>v5.1 SDXL submissions that required Part III (found by the number of mismatches).</summary><pre>
$ find inference_results_v5.1/closed -name verify_accuracy.txt -exec grep -H "num_perf_log_data_mismatch" {} \; | grep stable-diffusion-xl | grep -v "= 0"
closed/GigaComputing/compliance/G894-SD1_B200-SXM-180GBx8_TRT/stable-diffusion-xl/Server/TEST01/verify_accuracy.txt:num_perf_log_data_mismatch = 109
closed/GigaComputing/compliance/G894-SD1_B200-SXM-180GBx8_TRT/stable-diffusion-xl/Offline/TEST01/verify_accuracy.txt:num_perf_log_data_mismatch = 113
closed/The_Stage/compliance/h100_x8/stable-diffusion-xl/offline/TEST01/verify_accuracy.txt:num_perf_log_data_mismatch = 122
closed/The_Stage/compliance/h100_x8/stable-diffusion-xl/server/TEST01/verify_accuracy.txt:num_perf_log_data_mismatch = 125
closed/Dell/compliance/XE7740_L40S_PCIe_48GBx8_TRT/stable-diffusion-xl/Offline/TEST01/verify_accuracy.txt:num_perf_log_data_mismatch = 56
</pre></details>


Run with the 0.2 delta and instrumentation (see below)

```
/usr/bin/python3.12 "/mnt/nvme2/krai/anton/work_collection/mlperf_inference_git_master/tools/submission/submission_checker.py" \
--input "/mnt/nvme2/krai/anton/inference_results_v5.1/" --csv /dev/null --skip-extra-files-in-root-check &> \
submission_checker.delta-clip-0.2.log
```
<details><summary><code>grep CLIP_SCORE submission_checker.delta-clip-0.2.log -B 1 -A 3</code></summary><pre>
TEST_ACC_PATH: closed/GigaComputing/compliance/G894-SD1_B200-SXM-180GBx8_TRT/stable-diffusion-xl/Server/TEST01/accuracy
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.47262859435122095
VALID? False
TEST_ACC_PATH: closed/GigaComputing/compliance/G894-SD1_B200-SXM-180GBx8_TRT/stable-diffusion-xl/Offline/TEST01/accuracy
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.3045142068258011
VALID? False
--
TEST_ACC_PATH: closed/The_Stage/compliance/h100_x8/stable-diffusion-xl/offline/TEST01/accuracy
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.168084356792364
VALID? True
--
TEST_ACC_PATH: closed/The_Stage/compliance/h100_x8/stable-diffusion-xl/server/TEST01/accuracy
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.460193941157061
VALID? False
--
TEST_ACC_PATH: closed/Dell/compliance/XE7740_L40S_PCIe_48GBx8_TRT/stable-diffusion-xl/Offline/TEST01/accuracy
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.13336220810908905
VALID? True
</pre></details>

#### Instrumentation

<pre>
--- a/tools/submission/submission_checker.py
+++ b/tools/submission/submission_checker.py
@@ -516,7 +516,7 @@ MODEL_CONFIG = {
             "deepseek-r1": ("TOKENS_PER_SAMPLE", 1.1 * 3886.2274)
         },
         "accuracy-delta-perc": {
-            "stable-diffusion-xl": {"CLIP_SCORE": 1, "FID_SCORE": 2}
+            "stable-diffusion-xl": {"CLIP_SCORE": 0.2, "FID_SCORE": 2}
         },
         "performance-sample-count": {
             "resnet": 1024,
@@ -3199,14 +3199,17 @@ def check_compliance_acc_dir(test_dir, model, config):
                                 if m:
                                     acc_compliance[acc_type] = float(
                                         m.group(1))
+                    print("TEST_ACC_PATH:", test_acc_path)
                     for acc_type in acc_types:
+                        print("ACC_TYPE:", acc_type)
                         if acc_baseline[acc_type] == 0 or acc_compliance[acc_type] == 0:
                             is_valid = False
                             break
                         else:
                             required_delta_perc = config.get_delta_perc(
-                                model, acc_type[0]
+                                model, acc_type
                             )
+                            print("REQUIRED_DELTA_PERC:", required_delta_perc)
                             delta_perc = (
                                 abs(
                                     1
@@ -3215,14 +3218,17 @@ def check_compliance_acc_dir(test_dir, model, config):
                                 )
                                 * 100
                             )
+                            print("DELTA_PERC:", delta_perc)
                             if delta_perc <= required_delta_perc:
                                 is_valid = True
+                                print("VALID?", is_valid)
                             else:
                                 log.error(
                                     "Compliance test accuracy check (non-deterministic mode) in %s failed",
                                     test_dir,
                                 )
                                 is_valid = False
+                                print("VALID?", is_valid)
                                 break
         elif "TEST06" in test_dir:
             """
</pre>

Perhaps we should consider decreasing the CLIP delta to 0.2% along with another change - increasing the number of samples dumped in a TEST01 compliance run through [`accuracy_log_sampling_target`](https://github.com/mlcommons/inference/blob/2035f3f8982fd8e689abfae1ff18fe6ee2dde2b8/compliance/nvidia/TEST01/stable-diffusion-xl/audit.config#L9).

I took an Offline/Server pair of SDXL experiments passing the submissions checker (barring #2336), and rerun TEST01 for both scenarios with `accuracy_log_sampling_target=256`. The deltas reduced several fold for both FID and CLIP scores.

#### `accuracy_log_sampling_target=128`

The number of samples dumped: 125-130.

##### Offline
<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.2123741664745049
VALID? False
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 0.7769926144530093
VALID? True
</pre>

##### Server
<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.24943275362702133
VALID? False
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 1.5099146469846114
VALID? True
</pre>


#### `accuracy_log_sampling_target=256`

The number of samples dumped: 250-270.

##### Offline
<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.0005145860810240599
VALID? True
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 0.12457248412022226
VALID? True
</pre>

##### Server
<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.06640037966166856
VALID? True
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 0.24716082038236387
VALID? True
</pre>

Consider the accuracy sampling target for different scenarios:
<pre>
$ grep accuracy_log_sampling_target compliance/nvidia/TEST01/*/audit.config
3d-unet/audit.config:*.*.accuracy_log_sampling_target = 64
bert/audit.config:*.*.accuracy_log_sampling_target = 4096
dlrm/audit.config:*.*.accuracy_log_sampling_target = 4096
dlrm-v2/audit.config:*.*.accuracy_log_sampling_target = 4096
gpt-j/audit.config:*.*.accuracy_log_sampling_target = 4096
pointpainting/audit.config:*.*.accuracy_log_sampling_target = 256
resnet50/audit.config:*.*.accuracy_log_sampling_target = 4096
retinanet/audit.config:*.*.accuracy_log_sampling_target = 256
rgat/audit.config:*.*.accuracy_log_sampling_target = 4096
rnnt/audit.config:*.*.accuracy_log_sampling_target = 4096
stable-diffusion-xl/audit.config:*.*.accuracy_log_sampling_target = 128
whisper/audit.config:*.*.accuracy_log_sampling_target = 4096
</pre>

Compare this with the minimum Offline sample count:
<pre>
$ grep Offline\.min_query_count loadgen/mlperf.conf | sort
3d-unet.Offline.min_query_count = 43
bert.Offline.min_query_count = 10833
deepseek-r1.Offline.min_query_count = 4388
dlrm-v2.Offline.min_query_count = 24576
gptj.Offline.min_query_count = 13368
llama2-70b.Offline.min_query_count = 24576
llama3_1-405b.Offline.min_query_count = 8313
llama3_1-8b-edge.Offline.min_query_count = 5000
llama3_1-8b.Offline.min_query_count = 13368
mixtral-8x7b.Offline.min_query_count = 15000
resnet50.Offline.min_query_count = 24576
retinanet.Offline.min_query_count = 24576
rgat.Offline.min_query_count = 788379
rnnt.Offline.min_query_count = 2513
stable-diffusion-xl.Offline.min_query_count = 5000
whisper.Offline.min_query_count = 1633
</pre>

and with the performance sample count (buffer size):

<pre>
$ grep performance_sample_count loadgen/mlperf.conf | sort
3d-unet.*.performance_sample_count_override = 0
bert.*.performance_sample_count_override = 10833
deepseek-r1.*.performance_sample_count_override = 4388
dlrm.*.performance_sample_count_override = 204800
dlrm-v2.*.performance_sample_count_override = 204800
gptj.*.performance_sample_count_override = 13368
llama2-70b-interactive.*.performance_sample_count_override = 24576
llama2-70b.*.performance_sample_count_override = 24576
llama3_1-405b-interactive.*.performance_sample_count_override = 8313
llama3_1-405b.*.performance_sample_count_override = 8313
llama3_1-8b-edge.*.performance_sample_count_override = 5000
llama3_1-8b-interactive.*.performance_sample_count_override = 13368
llama3_1-8b.*.performance_sample_count_override = 13368
mixtral-8x7b.*.performance_sample_count_override = 15000
pointpainting.*.performance_sample_count_override = 1024
resnet50.*.performance_sample_count_override = 1024
retinanet.*.performance_sample_count_override = 64
rgat.*.performance_sample_count_override = 788379
rnnt.*.performance_sample_count_override = 2513
# Set performance_sample_count for each model.
ssd-mobilenet.*.performance_sample_count_override = 256
stable-diffusion-xl.*.performance_sample_count_override = 5000
whisper.*.performance_sample_count_override = 1633
</pre>

Another data point similar to [above](https://github.com/mlcommons/inference/issues/2338#issuecomment-3299039238), but required a higher sampling target to pass.

#### `accuracy_log_sampling_target=128`

Both scenarios fail on CLIP.

##### Offline
<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.27189986636354524
VALID? False
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 1.2138623684510907
VALID? True
</pre>

##### Server
<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.3979958228748304
VALID? False
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 1.094191157630875
VALID? True
</pre>

#### `accuracy_log_sampling_target=256`

One scenario still fails on CLIP. Server FID delta increased compared with `accuracy_log_sampling_target=128`.

##### Offline
<pre>
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.11276446421106812
VALID? True
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 0.3782969448748674
VALID? True
</pre>

##### Server
<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.26375361473660597
VALID? False
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 1.3214616263359846
VALID? True
</pre>

#### `accuracy_log_sampling_target=512`

One scenario still fails on CLIP. Offline FID delta increased compared with `accuracy_log_sampling_target=256`.

##### Offline
<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.02916976838076568
VALID? True
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 0.49769773932597516
VALID? True
</pre>

##### Server 
<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.21148273158384523
VALID? False
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 1.2532924271156376
VALID? True
</pre>

#### `accuracy_log_sampling_target=768`

Server CLIP delta passed. Offline FID/CLIP delta's increased slightly.

##### Offline

<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 1.2
DELTA_PERC: 0.07686692667840944
VALID? True
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 0.5366446368466726
VALID? True
</pre>

##### Server 
<pre>
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 0.2
DELTA_PERC: 0.04973686752067774
VALID? True
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 0.3421055655415728
VALID? True
</pre>

https://github.com/mlcommons/inference/pull/2337 resolved by this. Closing the issue