Repository: mlcommons/inference
Issue: [Bug] For benchmarks with multiple accuracy metrics, 1% delta applies to all metrics
Root cause: Algorithmic Error

Some benchmarks have multiple accuracy metrics. For example, [Stable Diffusion XL](https://github.com/mlcommons/inference/tree/master/text_to_image) (SDXL) has two metrics: FID and CLIP.

[Part III](https://github.com/mlcommons/inference/tree/master/compliance/nvidia/TEST01#part-iii) of compliance TEST01 specifies:

> For each target accuracy metric, the delta between the two accuracy results [baseline and compliance] should be within (1-x)%.
> For example, for 99.9% accuracy metric, the delta of the accuracy results should be within 0.1%.

This is reflected in the submission checker in the form of a [loop](https://github.com/mlcommons/inference/blob/2035f3f8982fd8e689abfae1ff18fe6ee2dde2b8/tools/submission/submission_checker.py#L3202) over accuracy metrics (types):
<details><summary><code>for acc_type in acc_types:</code></summary><pre>
                    for acc_type in acc_types:
                        if acc_baseline[acc_type] == 0 or acc_compliance[acc_type] == 0:
                            is_valid = False
                            break
                        else:
                            required_delta_perc = config.get_delta_perc(
                                model, acc_type[0]
                            )
                            delta_perc = (
                                abs(
                                    1
                                    - acc_baseline[acc_type] /
                                    acc_compliance[acc_type]
                                )
                                * 100
                            )
                            if delta_perc <= required_delta_perc:
                                is_valid = True
                            else:
                                log.error(
                                    "Compliance test accuracy check (non-deterministic mode) in %s failed",
                                    test_dir,
                                )
                                is_valid = False
                                break
</pre></details>

For each accuracy metric, the delta between the metric value from the baseline run and the metric value from the compliance runs must be within a specified value, `required_delta_perc`.

For example, [for SDXL](https://github.com/mlcommons/inference/blob/2035f3f8982fd8e689abfae1ff18fe6ee2dde2b8/tools/submission/submission_checker.py#L219), the delta for the CLIP metric is 1%, and the delta for the FID metric is 2%:

<pre>
        "accuracy-delta-perc": {
            "stable-diffusion-xl": {"CLIP_SCORE": 1, "FID_SCORE": 2}
        },
</pre>

Unfortunately, the submission checker currently does not take the specific deltas into account because of a bug [here](https://github.com/mlcommons/inference/blob/2035f3f8982fd8e689abfae1ff18fe6ee2dde2b8/tools/submission/submission_checker.py#L3207):
<pre>
                            required_delta_perc = config.get_delta_perc(
                                model, acc_type[0]
                            )
</pre>

Here's a submission checker trace for sample SDXL results:
<pre>
// Scenario: Offline.
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 1
DELTA_PERC: 0.2123741664745049
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 1
DELTA_PERC: 0.7769926144530093
// Scenario: Server.
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 1
DELTA_PERC: 0.24943275362702133
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 1
DELTA_PERC: 1.5099146469846114
</pre>

For the CLIP score, `acc_type[0] == 'C'`. For the FID score, `acc_type[0] == `F'. [The lookup](https://github.com/mlcommons/inference/blob/2035f3f8982fd8e689abfae1ff18fe6ee2dde2b8/tools/submission/submission_checker.py#L1145) fails, and the returned delta is [always 1](https://github.com/mlcommons/inference/blob/2035f3f8982fd8e689abfae1ff18fe6ee2dde2b8/tools/submission/submission_checker.py#L1150). The submission checker considers that TEST01 failed and rejects the Server results.

A small fix, `acc_type[0]` -> `acc_type`, fixes the issue, so the trace looks like:
<pre>
// Scenario: Offline.
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 1
DELTA_PERC: 0.2123741664745049
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 0.7769926144530093
// Scenario: Server.
ACC_TYPE: CLIP_SCORE
REQUIRED_DELTA_PERC: 1
DELTA_PERC: 0.24943275362702133
ACC_TYPE: FID_SCORE
REQUIRED_DELTA_PERC: 2
DELTA_PERC: 1.5099146469846114
</pre>

The submission checker considers that TEST01 passed and accepts the Server results.