Repository: mlcommons/inference
Issue: TEST05 uses old seeds
GitHub labels: Inference v3.0

I believe the [`audit.conf`](https://github.com/mlcommons/inference/blob/master/compliance/nvidia/TEST05/audit.config) was meant to be changed to:

```
diff --git a/compliance/nvidia/TEST05/audit.config b/compliance/nvidia/TEST05/audit.config
index 6cc9249..f607494 100644
--- a/compliance/nvidia/TEST05/audit.config
+++ b/compliance/nvidia/TEST05/audit.config
@@ -3,8 +3,4 @@
 # Model maybe '*' as wildcard. In that case the value applies to all models.
 # All times are in milli seconds

-# mode dictionary (0 = submission, 1 = accuracy, 2 = performance, 3 = find peak perf)
-*.*.mode = 2
-*.*.qsl_rng_seed = 313588358309856706
-*.*.sample_index_rng_seed = 471397156132239067
-*.*.schedule_rng_seed = 413914573387865862
+*.*.test05 = 1
```

Otherwise, it hasn't been updated for 3 years...

I forgot to update them there. I put them in the [mlperf.conf](https://github.com/mlcommons/inference/blob/b1c4cb972df6f51c5f08953dcb6f8ce5d163b41e/mlperf.conf#L24-L31) file, since now you can run test_05 using only that file. However you can also run this test using the `audit.conf` file, so they should be updated there as well. Thanks for the catch

Probably best option is to just delete the seed overrides in the audit.conf to avoid duplicating the values. 

@arjunsuresh @psyhtest What would be your desired behaviour for running this test be?

The current behavior can be found [here](https://github.com/mlcommons/inference/tree/master/compliance/nvidia/TEST05#part-i), but TLDR you can run this test by:

1. Copying the [audit.config](https://github.com/mlcommons/inference/blob/master/compliance/nvidia/TEST05/audit.config) file into the folder you are running the benchmark. Here the seeds are taken from the `audit.config` file.
2. Setting the flag `*.*.test05 = 1` in the [mlperf.conf](https://github.com/mlcommons/inference/blob/master/mlperf.conf) file. Here the seeds are taken from the `mlperf.conf` file.

How would you like to change this behaviour exactly?

@pgmpablo157321 It is not a big deal but since mlperf.conf file is always used, I think it is better to keep all the seeds there even when submitters use audit.conf file for turning on test05. If you feel it does not make sense please ignore it. 

@arjunsuresh So, you mean like when running the test05 with the `audit.config` file, we should read the seeds from `mlperf.conf` file to avoid having them duplicated? Is that correct?

I think this requires some small loadgen changes, but they might ok

yes exactly. But this can wait until 3.1 also. But I'm not sure why @psyhtest reopened this issue. 

> Setting the flag *.*.test05 = 1 in the [mlperf.conf](https://github.com/mlcommons/inference/blob/master/mlperf.conf) file.
> Here the seeds are taken from the mlperf.conf file.

Ideally, `mlperf.conf` should never be modified directly. Otherwise, it's easy to forget to modify it back after running TEST05.

The TEST05 seeds have different names, so the user would just need to request LoadGen to use them for a particular run. Using `audit.config` containing simply `*.*.test05 = 1` sounds ideal to me (as well as dropping `*.*.test05 = 0` from `mlperf.conf`).

Having said that, the current solution works too, so we can revisit for v3.1.