Repository: mlcommons/inference
Issue: Audit test

Hello Submitters,
 
A sample audit test has been uploaded to the MLPerf Community shared drive under the Inference folder.
 
Here is a link:
https://drive.google.com/open?id=1KU4ZtZBcy4fMLrsasDHyB817Q4zqjZRT
 
The tarball follows the directory structure as outlined in the Audit Test Infrastructure doc:
https://docs.google.com/document/d/1JsA1_iqzGZFtFLwIM45IfMLstYtE5tflvfD7oiqAiBo/edit?usp=sharing
 
Please give it a try and let me know if there are any issues.
 
Thank you
George

Known issues (also communicated by email):
1. The `verify_*` scripts are not Python 3 compliant.
```
$ python3 /data/anton/mlperf/audit_release_nvidia_0.1/TEST01/mobilenet/verify_accuracy.py \
-a output/tf-cpu/mobilenet.accuracy/mlperf_log_accuracy.json \
-p output/tf-cpu/mobilenet/mlperf_log_accuracy.json

  File "/data/anton/mlperf/audit_release_nvidia_0.1/TEST01/mobilenet/verify_accuracy.py",
line 68
    print "num_acc_log_entries = ", len(acc_data)
```

2. The `verify_performance` script tries to match "Samples per second" in `mlperf_log_summary.txt`, but there's no such line from the SingleStream scenario.
```
$ python /data/anton/mlperf/audit_release_nvidia_0.1/TEST01/mobilenet/verify_performance.py \
-r output/tf-cpu/mobilenet.performance/mlperf_log_summary.txt \
-t output/tf-cpu/mobilenet.audit/mlperf_log_summary.txt

Verifying performance.
reference score =  0
test score =  0
TEST FAIL
```
3. The values in `audit.config` should be specified in percentages, as we use the same unit
for e.g. percentiles. (I'd actually suggest to rename the config file to `audit.conf` to make it consistent with the default `mlperf.conf`.)

For `TEST01`, the directory structure is:
```
-rw-rw-r-- 1 anton dvdt 1906 Sep 27 23:33 README

gnmt:
total 20
drwxrwxr-x 2 anton dvdt 4096 Sep 27 23:10 .
drwxrwxr-x 7 anton dvdt 4096 Oct  1 17:34 ..
-rw-rw-r-- 1 anton dvdt  736 Sep 27 23:10 audit.config
-rwxrwxr-x 1 anton dvdt 2658 Sep 27 23:10 verify_accuracy.py
-rwxrwxr-x 1 anton dvdt 1366 Sep 27 23:10 verify_performance.py

mobilenet:
total 20
drwxrwxr-x 2 anton dvdt 4096 Oct  1 17:33 .
drwxrwxr-x 7 anton dvdt 4096 Oct  1 17:34 ..
-rw-rw-r-- 1 anton dvdt  736 Sep 27 23:10 audit.config
-rwxrwxr-x 1 anton dvdt 2658 Sep 27 23:10 verify_accuracy.py
-rwxrwxr-x 1 anton dvdt 1366 Sep 27 23:10 verify_performance.py

resnet:
total 20
drwxrwxr-x 2 anton dvdt 4096 Sep 27 23:10 .
drwxrwxr-x 7 anton dvdt 4096 Oct  1 17:34 ..
-rw-rw-r-- 1 anton dvdt  736 Sep 27 23:10 audit.config
-rwxrwxr-x 1 anton dvdt 2658 Sep 27 23:10 verify_accuracy.py
-rwxrwxr-x 1 anton dvdt 1366 Sep 27 23:10 verify_performance.py

ssd-large:
total 20
drwxrwxr-x 2 anton dvdt 4096 Sep 27 23:10 .
drwxrwxr-x 7 anton dvdt 4096 Oct  1 17:34 ..
-rw-rw-r-- 1 anton dvdt  736 Sep 27 23:10 audit.config
-rwxrwxr-x 1 anton dvdt 2658 Sep 27 23:10 verify_accuracy.py
-rwxrwxr-x 1 anton dvdt 1366 Sep 27 23:10 verify_performance.py

ssd-small:
total 20
drwxrwxr-x 2 anton dvdt 4096 Sep 27 23:10 .
drwxrwxr-x 7 anton dvdt 4096 Oct  1 17:34 ..
-rw-rw-r-- 1 anton dvdt  736 Sep 27 23:10 audit.config
-rwxrwxr-x 1 anton dvdt 2658 Sep 27 23:10 verify_accuracy.py
-rwxrwxr-x 1 anton dvdt 1366 Sep 27 23:10 verify_performance.py
```
but all the scripts and config files appear to be the same for all the benchmarks. Is this correct? If so, can we avoid duplication?

`TEST02` (custom weights) does not sound to be easy!

`TEST05` (RNG seeds) - is this applicable to every scenario?

> `TEST05` (RNG seeds) - is this applicable to every scenario?

I believe so. Performance should not be RNG seed-dependent.

> For `TEST01`, the directory structure is:
> 
> ```
> -rw-rw-r-- 1 anton dvdt 1906 Sep 27 23:33 README
> 
> gnmt:
> total 20
> drwxrwxr-x 2 anton dvdt 4096 Sep 27 23:10 .
> drwxrwxr-x 7 anton dvdt 4096 Oct  1 17:34 ..
> -rw-rw-r-- 1 anton dvdt  736 Sep 27 23:10 audit.config
> -rwxrwxr-x 1 anton dvdt 2658 Sep 27 23:10 verify_accuracy.py
> -rwxrwxr-x 1 anton dvdt 1366 Sep 27 23:10 verify_performance.py
> 
> mobilenet:
> total 20
> drwxrwxr-x 2 anton dvdt 4096 Oct  1 17:33 .
> drwxrwxr-x 7 anton dvdt 4096 Oct  1 17:34 ..
> -rw-rw-r-- 1 anton dvdt  736 Sep 27 23:10 audit.config
> -rwxrwxr-x 1 anton dvdt 2658 Sep 27 23:10 verify_accuracy.py
> -rwxrwxr-x 1 anton dvdt 1366 Sep 27 23:10 verify_performance.py
> 
> resnet:
> total 20
> drwxrwxr-x 2 anton dvdt 4096 Sep 27 23:10 .
> drwxrwxr-x 7 anton dvdt 4096 Oct  1 17:34 ..
> -rw-rw-r-- 1 anton dvdt  736 Sep 27 23:10 audit.config
> -rwxrwxr-x 1 anton dvdt 2658 Sep 27 23:10 verify_accuracy.py
> -rwxrwxr-x 1 anton dvdt 1366 Sep 27 23:10 verify_performance.py
> 
> ssd-large:
> total 20
> drwxrwxr-x 2 anton dvdt 4096 Sep 27 23:10 .
> drwxrwxr-x 7 anton dvdt 4096 Oct  1 17:34 ..
> -rw-rw-r-- 1 anton dvdt  736 Sep 27 23:10 audit.config
> -rwxrwxr-x 1 anton dvdt 2658 Sep 27 23:10 verify_accuracy.py
> -rwxrwxr-x 1 anton dvdt 1366 Sep 27 23:10 verify_performance.py
> 
> ssd-small:
> total 20
> drwxrwxr-x 2 anton dvdt 4096 Sep 27 23:10 .
> drwxrwxr-x 7 anton dvdt 4096 Oct  1 17:34 ..
> -rw-rw-r-- 1 anton dvdt  736 Sep 27 23:10 audit.config
> -rwxrwxr-x 1 anton dvdt 2658 Sep 27 23:10 verify_accuracy.py
> -rwxrwxr-x 1 anton dvdt 1366 Sep 27 23:10 verify_performance.py
> ```
> 
> but all the scripts and config files appear to be the same for all the benchmarks. Is this correct? If so, can we avoid duplication?

Good point. I had created benchmark subdirectories to handle benchmark-specific custom weights, but this would not apply to any of the tests for v0.5. I can move duplicate files to the parent directory.

For TEST03, we'd like to see the plan for distribution to label category, distribution to box aspect ratio & size, etc. to ensure the test is not prefer one implementation over another. 

Thanks!