Repository: mlcommons/inference
Issue: Compliance tests
Root cause: Documentation Deficiency

I'm having issues running the compliance tests.
For example, to run the compliance test TEST_01, the instructions say:
https://github.com/mlcommons/inference/tree/r1.0/compliance/nvidia/TEST01#part-i
```
Run test with the provided audit.config in the corresponding benchmark subdirectory.
Note that audit.config must be copied to the directory where the benchmark is being run from.
Verification that audit.config was properly read can be done by checking that loadgen has found audit.config in mlperf_log_detail.txt
```
Then I copy `audit.config` to the directory from I run the tests, in my case:
```
vision/classification_and_detection
```
that is where I usually run `run_local.sh` script.

But later on, there is no confirmation about finding the `audit.config` file in `mlperf_log_detail.txt`.
Should the `audit.config` file be placed in another location? Or run the scripts from another directory?

P.S. why the compliance scripts have the name of "nvidia" as part of the directory?


If you're running using `main.py`, it actually changes directory before it starts the official run. The lines here...
https://github.com/mlcommons/inference/blob/r1.0/vision/classification_and_detection/python/main.py#L451-L454

... change to the output directory that is specified by the `output` arg. So if you're using `main.py`, try putting the `audit.config` file in the same location that the `output` argument specifies.

The default `output` argument is the subdirectory `vision/classification_and_detection/output`, if you're running `main.py` from `vision/classification_and_detection`.

I personally would like to see this requirement for copying `audit.config` to the "run directory" changed, as is inevitably seems to throw a wrench in our organization's automation flow whenever we make changes. I would prefer if specifying the location of `audit.config` was argument-based, rather than having `Loadgen` detect it and pick it up. I think the current approach was intentional, though, but I don't know the history behind the decision.

> P.S. why the compliance scripts have the name of "nvidia" as part of the directory?

Everyone can propose compliance tests to the WG. And it just indicates that those compliance tests were proposed by NVIDIA.

@georgelyuan Could you answer these questions? Also @guschmue for visibility

I think we cd'd to the output directory because long time ago there was no way to tell loadgen where to write its output.
Now that is possible and I added it to main.py so we should be able to remove the cd.
Adding an argument might be even better but there might be a complication for the docker image.
Need to check but didn't the run script copy mlperf.conf because of the the docker image. Maybe we can also copy audit.conf.

@mthmsn Thanks Mike, I can now run it in audit mode if I copy the configuration file to the output directory.
I didn't think of copying to the output directory because it doesn't exist before running the scripts.
If I create the directory in advance and then copy the audit file there it works.

@corehalt Hi , could you please explain it to me how you run  run_local.sh in audit mode?
In vision/classification I make a directory "output" and put the config file in TEST_01 into the output directory,
but I didn't see any relative information in output/tf-gpu/resnet50/mlperf_log_detail.txt.
```
cd inference/vision/classification_and_detection/
mkdir output
cp inference/compliance/nvidia/TEST01/resnet50/audit.config inference/vision/classification_and_detection/output/
./run_local.sh tf resnet50 gpu --scenario Offline --count 100
```
Below is the content of the output file mlperf_log_detail.txt:
<details>
  <summary>Click to expand</summary>
"pid": 5129, "tid": 5129, "ts": 7841ns : LoadgenVersionInfo:
"pid": 5129, "tid": 5129, "ts": 7841ns : version : .5a1 @ cd105ff3a5
"pid": 5129, "tid": 5129, "ts": 7841ns : build_date_local : 2021-03-16T10:30:22.571418
"pid": 5129, "tid": 5129, "ts": 7841ns : build_date_utc   : 2021-03-16T02:30:22.571427
"pid": 5129, "tid": 5129, "ts": 7841ns : git_commit_date  : 2021-01-11T10:09:48-08:00
"pid": 5129, "tid": 5129, "ts": 7841ns : git_log :

cd105ff3a5c7f918ee0a153ef325a3610f59751f Update backend_tf.py (#774)
5ba3acfd3932c18d91235eddbc0910b7818f4dd5 Merge pull request #737 from nv-rborkar/r0.7_test4_doc
8f4a68c2ec53509aef08f2f5ad18446c23cdfcd2 split results in closed/open, datacenter/edge
7501816760424527619024322470c05495b60dc8 use quote in csv for system name, dont use 0 for accelorators
d38dca2f6feb81f413bfa836a273294128c541c7 some fixes to final results spreadsheet
a54ef34a777c61e9dc812262800fa77404868b8d change the way we infer for offline, add final report generation
84c8a0614b42bb0ea93f626470c8eb31a66f0ce1 support for infered offline results
aee0fa146f0091d2b00821b430951418d7840ea0 Merge pull request #751 from nv-rborkar/r0.7
646cb2d7380e7f13227c6100d2a720e7773e9e37 Patch to ignore extraneous files in compliance dir
013c6d026f7bbad52ad8121643e7613e10fc0f8b Merge pull request #735 from nv-rborkar/r0.7
a85508153f3623e0ac9a63ea9f3d91dddcd3fc0e Merge branch 'r0.7' into r0.7
efb29186004cb04913ac2565eb96e4eed382e837 workarounds for v0.7 submission
ff7ca12d1ebd37e63a03db97cf0e76d903c21956 few fixes to checker to deal with the full submission
7a9533cbea7c0a82a0f3ebc6797616ed14487d99 Merge pull request #747 from Chujingjun/alvin_custom_model_name
7693be66ac6816585e8d2ab8960b61135ffb8945 add necessary double quotes for csv
8bf5670a6130e506f8bd8cd1a829abbcfdaa94e6 allow custom model name in open division

"pid": 5129, "tid": 5129, "ts": 7841ns : git_status :



"pid": 5129, "tid": 5129, "ts": 7841ns : SHA1 of files :

012aad77e5206c89d50718c46c119d1f3cb056b2 /.clang-format
b82905303c4756c4a2ab40ceed92cdddbf04bf2d /BUILD.gn
7067c2704e8de3be0a0fb9ef5984d52a88f9b91e /CMakeLists.txt
79fbce4843967690e0750fd7ffc021f4f1e4c5c4 /README.md
34b89a712a0ca1691b89715fe2ce459a676d0f77 /README_BUILD.md
f75741b91085a20ebb23de1ce03d45a56d3457fe /README_FAQ.md
a00620bb4c9b583c1d2f5beef1a5e8d772aefb9b /bindings/c_api.cc
81873e098bc60c7bc59045c8329ce153dd1ec055 /bindings/c_api.h
227149dc287e93a28da9ce92346158e4e403b694 /bindings/python_api.cc
4b68d3399b31e14e554cff678a0db9b4d1299e23 /demos/BUILD.gn
07d85ea5fefade8d3d66ca832b783bf90e5bce7d /demos/py_demo_multi_stream.py
2f941e949770a2ea9ad6d6eea2f33fc19150edcd /demos/py_demo_multi_stream_free.py
73c60cf12cfb69619249e637f9bcad18a47cda6c /demos/py_demo_offline.py
02a6ddbe8b51bde2cfed32abeccdb7fdd9cfe4e7 /demos/py_demo_server.py
3e7d786882099ac784bf878f1cde9dd3db2f6d9f /demos/py_demo_single_stream.py
8e76b6b70bc724c4a75dde2931c060ac831335b3 /issue_query_controller.cc
703be7bdc58a8fa83555917f89054cbdb50038d4 /issue_query_controller.h
434abcabb2191ae63af37c928d87cd02a78c51df /loadgen.cc
ef11e328aa06b4e1996d68e4746a969b9c9ade67 /loadgen.h
47f748307536f80cfc606947b440dd732afc2637 /loadgen_integration_diagram.svg
9960bfaf44da0925444c74d54e1b97d75bf5e812 /logging.cc
fdb61b8a2e2593bab4ae5bf9758434cf6a60e218 /logging.h
a879f127c4bb64d131056942734ea1a6ba9c3033 /query_sample.h
61feb478b15668f48245c5083d559dee5de1c082 /query_sample_library.h
bd41a2b3f4ed365566e411fddd012c66a320e87b /setup.py
744482c84336420adbbe31ef8c2b39612c6ca3f8 /system_under_test.h
34eec1f8fb4a2db273b715686580f19cffa2f3a1 /test_settings.h
08bc8aec88abd99fa5633c23fb94d090138b475e /test_settings_internal.cc
085aaa57819f88c585c1f3a627a2d9a432a01bbc /test_settings_internal.h
86fc4c8035e7b73132c56d1781aca647b6bd6916 /utils.cc
5395a04f82e90288f52bf918d7b1d6027bfe0964 /utils.h
e1c46896da912ff2a0187276c3a8ba4eec057a17 /version.cc
9d574baa64424e9c708fcfedd3dbb0b518a65fcc /version.h
07dbee77c9707efdb95def65477c288fe6667e92 /version_generator.py

"pid": 5129, "tid": 5129, "ts": 30991ns : Date + time of test: 2021-03-16T15:54:15Z
"pid": 5129, "tid": 5129, "ts": 30991ns : System Under Test (SUT) name: PySUT
"pid": 5129, "tid": 5129, "ts": 30991ns : Query Sample Library (QSL) name: PyQSL
"pid": 5129, "tid": 5129, "ts": 30991ns : QSL total size: 100
"pid": 5129, "tid": 5129, "ts": 30991ns : QSL performance size*: 100
"pid": 5129, "tid": 5129, "ts": 30991ns : *TestSettings (performance_sample_count_override) can override
"pid": 5129, "tid": 5129, "ts": 30991ns : *Refer to Effective Settings for actual value
"pid": 5129, "tid": 5129, "ts": 46366ns :
"pid": 5129, "tid": 5129, "ts": 46366ns : Effective Settings:
"pid": 5129, "tid": 5129, "ts": 46366ns : Scenario : Offline
"pid": 5129, "tid": 5129, "ts": 46366ns : Test mode : Performance
"pid": 5129, "tid": 5129, "ts": 46366ns : samples_per_query : 100
"pid": 5129, "tid": 5129, "ts": 46366ns : target_qps : 1
"pid": 5129, "tid": 5129, "ts": 46366ns : target_latency (ns): 0
"pid": 5129, "tid": 5129, "ts": 46366ns : target_latency_percentile : 4.66209e-310
"pid": 5129, "tid": 5129, "ts": 46366ns : max_async_queries : 1
"pid": 5129, "tid": 5129, "ts": 46366ns : target_duration (ms): 0
"pid": 5129, "tid": 5129, "ts": 46366ns : min_duration (ms): 60000
"pid": 5129, "tid": 5129, "ts": 46366ns : max_duration (ms): 0
"pid": 5129, "tid": 5129, "ts": 46366ns : min_query_count : 1
"pid": 5129, "tid": 5129, "ts": 46366ns : max_query_count : 100
"pid": 5129, "tid": 5129, "ts": 46366ns : min_sample_count : 100
"pid": 5129, "tid": 5129, "ts": 46366ns : qsl_rng_seed : 12786827339337101903
"pid": 5129, "tid": 5129, "ts": 46366ns : sample_index_rng_seed : 12640797754436136668
"pid": 5129, "tid": 5129, "ts": 46366ns : schedule_rng_seed : 3135815929913719677
"pid": 5129, "tid": 5129, "ts": 46366ns : accuracy_log_rng_seed : 0
"pid": 5129, "tid": 5129, "ts": 46366ns : accuracy_log_probability : 0
"pid": 5129, "tid": 5129, "ts": 46366ns : accuracy_log_sampling_target : 0
"pid": 5129, "tid": 5129, "ts": 46366ns : print_timestamps : false
"pid": 5129, "tid": 5129, "ts": 46366ns : performance_issue_unique : false
"pid": 5129, "tid": 5129, "ts": 46366ns : performance_issue_same : false
"pid": 5129, "tid": 5129, "ts": 46366ns : performance_issue_same_index : 0
"pid": 5129, "tid": 5129, "ts": 46366ns : performance_sample_count : 1024
"pid": 5129, "tid": 5129, "ts": 48534ns :
"pid": 5129, "tid": 5129, "ts": 48534ns : Requested Settings:
"pid": 5129, "tid": 5129, "ts": 48534ns : Scenario : Offline
"pid": 5129, "tid": 5129, "ts": 48534ns : Test mode : Performance
"pid": 5129, "tid": 5129, "ts": 48534ns : offline_expected_qps : 1
"pid": 5129, "tid": 5129, "ts": 48534ns : min_duration_ms : 60000
"pid": 5129, "tid": 5129, "ts": 48534ns : max_duration_ms : 0
"pid": 5129, "tid": 5129, "ts": 48534ns : min_query_count : 100
"pid": 5129, "tid": 5129, "ts": 48534ns : max_query_count : 100
"pid": 5129, "tid": 5129, "ts": 48534ns : qsl_rng_seed : 12786827339337101903
"pid": 5129, "tid": 5129, "ts": 48534ns : sample_index_rng_seed : 12640797754436136668
"pid": 5129, "tid": 5129, "ts": 48534ns : schedule_rng_seed : 3135815929913719677
"pid": 5129, "tid": 5129, "ts": 48534ns : accuracy_log_rng_seed : 0
"pid": 5129, "tid": 5129, "ts": 48534ns : accuracy_log_probability : 0
"pid": 5129, "tid": 5129, "ts": 48534ns : accuracy_log_sampling_target : 0
"pid": 5129, "tid": 5129, "ts": 48534ns : print_timestamps : false
"pid": 5129, "tid": 5129, "ts": 48534ns : performance_issue_unique : false
"pid": 5129, "tid": 5129, "ts": 48534ns : performance_issue_same : false
"pid": 5129, "tid": 5129, "ts": 48534ns : performance_issue_same_index : 0
"pid": 5129, "tid": 5129, "ts": 48534ns : performance_sample_count_override : 1024
"pid": 5129, "tid": 5129, "ts": 48534ns :
"pid": 5129, "tid": 5129, "ts": 53500ns : Starting performance mode:
"pid": 5129, "tid": 5129, "ts": 72361ns : Loading QSL : "set" : "[76,11,51,30,48,42,16,61,23,40,68,83,36,17,52,18,57,71,84,27,9,22,99,54,97,70,89,13,15,59,85,4,35,3,12,87,24,56,29,38,45,19,14,21,78,86,44,25,66,74,73,92,55,98,81,69,7,93,90,37,28,64,39,6,50,79,8,10,49,43,5,33,47,67,60,65,41,31,77,96,91,58,72,95,82,0,88,62,20,46,1,53,94,26,75,2,63,80,32,34]"
"pid": 5129, "tid": 5129, "ts": 80269150ns : GeneratedQueries: "queries" : 1, "samples per query" : 100, "duration" : 1000000000
"pid": 5129, "tid": 5129, "ts": 170242062ns :  Ending naturally: Minimum query count and test duration met.
"pid": 5129, "tid": 5129, "ts": 1275932714ns : Log Contention Counters:
"pid": 5129, "tid": 5129, "ts": 1275932714ns : 0 : swap_request_slots_retry_count
"pid": 5129, "tid": 5129, "ts": 1275932714ns : 0 : swap_request_slots_retry_retry_count
"pid": 5129, "tid": 5129, "ts": 1275932714ns : 0 : swap_request_slots_retry_reencounter_count
"pid": 5129, "tid": 5129, "ts": 1275932714ns : 0 : start_reading_entries_retry_count
"pid": 5129, "tid": 5129, "ts": 1275932714ns : 0 : tls_total_log_cas_fail_count
"pid": 5129, "tid": 5129, "ts": 1275932714ns : 0 : tls_total_swap_buffers_slot_retry_count
"pid": 5129, "tid": 5129, "ts": 1275937097ns : POWER_BEGIN: "mode" : Performance, "time" : 03-16-2021 15:54:15.764
"pid": 5129, "tid": 5129, "ts": 1275937097ns : POWER_END: "mode" : Performance, "time" : 03-16-2021 15:54:16.950

@georgelyuan please take a look

Looks like there's no confirmation because that message got removed when we transitioned to MLLOG. We don't see it in our logs either.

Do you see a line that looks like this? That means a log sampling target has been set.
`:::MLLOG {"key": "effective_accuracy_log_sampling_target", "value": 4096, "time_ms": <redacted>, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": <redacted>, "line_no": 405, "pid": 1386, "tid": 1386}}`

Also, if you check your mlperf_log_accuracy.json, it should be not empty, as one would expect otherwise for a PerformanceOnly run.



I get an MLLOG line that references the `audit.config` file. Our organization is not using the python harness this time around, but that shouldn't matter, since loadgen should produce an equivalent output log either way, right? Here's the message I see in our `mlperf_log_detail.txt`:
```
:::MLLOG {"key": "warning_generic_message", "value": "Found Audit Config file (audit.config). Overriding TestSettings from audit.config file.", (... etc.)
```

Ah thanks, I missed it. I see it now too. I got the issues a bit mixed up.

BeyondCloud, can you doublecheck your commands? With the way you've written them, I don't think those commands will work because of the directory structures. I.e. I don't think your third line will work if you already cd'd into classification_and_detection (from the first line)

I'm also curious why you don't see the MLLOG format in your log...

My mlperf_log_detail.txt is empty
To reproduce my issue (assume datasets are already in /workspace/data/):
```
git clone https://github.com/mlperf/inference.git --recurse-submodules
cd inference
git checkout r0.7
cd loadgen;CFLAGS="-std=c++14 -O3" python setup.py develop;cd ..
cd vision/classification_and_detection; python setup.py develop
mkdir data
ln -s /workspace/data/imagenet2012 data/
ln -s /workspace/data/coco data/
mkdir models
wget -q https://zenodo.org/record/2535873/files/resnet50_v1.pb -O models/resnet50_v1.pb 
wget -q https://zenodo.org/record/2592612/files/resnet50_v1.onnx -O models/resnet50_v1.onnx
mkdir output
cp ../../compliance/nvidia/TEST01/resnet50/audit.config ./output/
export DATA_DIR=$PWD/data/imagenet2012
export MODEL_DIR=$PWD/models
./run_local.sh tf resnet50 gpu --scenario Offline --count 100
```
<details>
  <summary>mlperf_log_accuracy.json (click to expand)</summary>
[
]
</details>
<details>
  <summary>mlperf_log_detail.txt</summary>
"pid": 29722, "tid": 29722, "ts": 6664ns : LoadgenVersionInfo:
"pid": 29722, "tid": 29722, "ts": 6664ns : version : .5a1 @ cd105ff3a5
"pid": 29722, "tid": 29722, "ts": 6664ns : build_date_local : 2021-03-17T10:17:50.614858
"pid": 29722, "tid": 29722, "ts": 6664ns : build_date_utc   : 2021-03-17T02:17:50.614867
"pid": 29722, "tid": 29722, "ts": 6664ns : git_commit_date  : 2021-01-11T10:09:48-08:00
"pid": 29722, "tid": 29722, "ts": 6664ns : git_log :

cd105ff3a5c7f918ee0a153ef325a3610f59751f Update backend_tf.py (#774)
5ba3acfd3932c18d91235eddbc0910b7818f4dd5 Merge pull request #737 from nv-rborkar/r0.7_test4_doc
8f4a68c2ec53509aef08f2f5ad18446c23cdfcd2 split results in closed/open, datacenter/edge
7501816760424527619024322470c05495b60dc8 use quote in csv for system name, dont use 0 for accelorators
d38dca2f6feb81f413bfa836a273294128c541c7 some fixes to final results spreadsheet
a54ef34a777c61e9dc812262800fa77404868b8d change the way we infer for offline, add final report generation
84c8a0614b42bb0ea93f626470c8eb31a66f0ce1 support for infered offline results
aee0fa146f0091d2b00821b430951418d7840ea0 Merge pull request #751 from nv-rborkar/r0.7
646cb2d7380e7f13227c6100d2a720e7773e9e37 Patch to ignore extraneous files in compliance dir
013c6d026f7bbad52ad8121643e7613e10fc0f8b Merge pull request #735 from nv-rborkar/r0.7
a85508153f3623e0ac9a63ea9f3d91dddcd3fc0e Merge branch 'r0.7' into r0.7
efb29186004cb04913ac2565eb96e4eed382e837 workarounds for v0.7 submission
ff7ca12d1ebd37e63a03db97cf0e76d903c21956 few fixes to checker to deal with the full submission
7a9533cbea7c0a82a0f3ebc6797616ed14487d99 Merge pull request #747 from Chujingjun/alvin_custom_model_name
7693be66ac6816585e8d2ab8960b61135ffb8945 add necessary double quotes for csv
8bf5670a6130e506f8bd8cd1a829abbcfdaa94e6 allow custom model name in open division

"pid": 29722, "tid": 29722, "ts": 6664ns : git_status :



"pid": 29722, "tid": 29722, "ts": 6664ns : SHA1 of files :

012aad77e5206c89d50718c46c119d1f3cb056b2 /.clang-format
b82905303c4756c4a2ab40ceed92cdddbf04bf2d /BUILD.gn
7067c2704e8de3be0a0fb9ef5984d52a88f9b91e /CMakeLists.txt
79fbce4843967690e0750fd7ffc021f4f1e4c5c4 /README.md
34b89a712a0ca1691b89715fe2ce459a676d0f77 /README_BUILD.md
f75741b91085a20ebb23de1ce03d45a56d3457fe /README_FAQ.md
a00620bb4c9b583c1d2f5beef1a5e8d772aefb9b /bindings/c_api.cc
81873e098bc60c7bc59045c8329ce153dd1ec055 /bindings/c_api.h
227149dc287e93a28da9ce92346158e4e403b694 /bindings/python_api.cc
4b68d3399b31e14e554cff678a0db9b4d1299e23 /demos/BUILD.gn
07d85ea5fefade8d3d66ca832b783bf90e5bce7d /demos/py_demo_multi_stream.py
2f941e949770a2ea9ad6d6eea2f33fc19150edcd /demos/py_demo_multi_stream_free.py
73c60cf12cfb69619249e637f9bcad18a47cda6c /demos/py_demo_offline.py
02a6ddbe8b51bde2cfed32abeccdb7fdd9cfe4e7 /demos/py_demo_server.py
3e7d786882099ac784bf878f1cde9dd3db2f6d9f /demos/py_demo_single_stream.py
8e76b6b70bc724c4a75dde2931c060ac831335b3 /issue_query_controller.cc
703be7bdc58a8fa83555917f89054cbdb50038d4 /issue_query_controller.h
434abcabb2191ae63af37c928d87cd02a78c51df /loadgen.cc
ef11e328aa06b4e1996d68e4746a969b9c9ade67 /loadgen.h
47f748307536f80cfc606947b440dd732afc2637 /loadgen_integration_diagram.svg
9960bfaf44da0925444c74d54e1b97d75bf5e812 /logging.cc
fdb61b8a2e2593bab4ae5bf9758434cf6a60e218 /logging.h
a879f127c4bb64d131056942734ea1a6ba9c3033 /query_sample.h
61feb478b15668f48245c5083d559dee5de1c082 /query_sample_library.h
bd41a2b3f4ed365566e411fddd012c66a320e87b /setup.py
744482c84336420adbbe31ef8c2b39612c6ca3f8 /system_under_test.h
34eec1f8fb4a2db273b715686580f19cffa2f3a1 /test_settings.h
08bc8aec88abd99fa5633c23fb94d090138b475e /test_settings_internal.cc
085aaa57819f88c585c1f3a627a2d9a432a01bbc /test_settings_internal.h
86fc4c8035e7b73132c56d1781aca647b6bd6916 /utils.cc
5395a04f82e90288f52bf918d7b1d6027bfe0964 /utils.h
e1c46896da912ff2a0187276c3a8ba4eec057a17 /version.cc
9d574baa64424e9c708fcfedd3dbb0b518a65fcc /version.h
07dbee77c9707efdb95def65477c288fe6667e92 /version_generator.py

"pid": 29722, "tid": 29722, "ts": 26364ns : Date + time of test: 2021-03-17T10:28:43Z
"pid": 29722, "tid": 29722, "ts": 26364ns : System Under Test (SUT) name: PySUT
"pid": 29722, "tid": 29722, "ts": 26364ns : Query Sample Library (QSL) name: PyQSL
"pid": 29722, "tid": 29722, "ts": 26364ns : QSL total size: 100
"pid": 29722, "tid": 29722, "ts": 26364ns : QSL performance size*: 100
"pid": 29722, "tid": 29722, "ts": 26364ns : *TestSettings (performance_sample_count_override) can override
"pid": 29722, "tid": 29722, "ts": 26364ns : *Refer to Effective Settings for actual value
"pid": 29722, "tid": 29722, "ts": 39377ns :
"pid": 29722, "tid": 29722, "ts": 39377ns : Effective Settings:
"pid": 29722, "tid": 29722, "ts": 39377ns : Scenario : Offline
"pid": 29722, "tid": 29722, "ts": 39377ns : Test mode : Performance
"pid": 29722, "tid": 29722, "ts": 39377ns : samples_per_query : 100
"pid": 29722, "tid": 29722, "ts": 39377ns : target_qps : 1
"pid": 29722, "tid": 29722, "ts": 39377ns : target_latency (ns): 0
"pid": 29722, "tid": 29722, "ts": 39377ns : target_latency_percentile : 4.67377e-310
"pid": 29722, "tid": 29722, "ts": 39377ns : max_async_queries : 1
"pid": 29722, "tid": 29722, "ts": 39377ns : target_duration (ms): 0
"pid": 29722, "tid": 29722, "ts": 39377ns : min_duration (ms): 60000
"pid": 29722, "tid": 29722, "ts": 39377ns : max_duration (ms): 0
"pid": 29722, "tid": 29722, "ts": 39377ns : min_query_count : 1
"pid": 29722, "tid": 29722, "ts": 39377ns : max_query_count : 100
"pid": 29722, "tid": 29722, "ts": 39377ns : min_sample_count : 100
"pid": 29722, "tid": 29722, "ts": 39377ns : qsl_rng_seed : 12786827339337101903
"pid": 29722, "tid": 29722, "ts": 39377ns : sample_index_rng_seed : 12640797754436136668
"pid": 29722, "tid": 29722, "ts": 39377ns : schedule_rng_seed : 3135815929913719677
"pid": 29722, "tid": 29722, "ts": 39377ns : accuracy_log_rng_seed : 0
"pid": 29722, "tid": 29722, "ts": 39377ns : accuracy_log_probability : 0
"pid": 29722, "tid": 29722, "ts": 39377ns : accuracy_log_sampling_target : 0
"pid": 29722, "tid": 29722, "ts": 39377ns : print_timestamps : false
"pid": 29722, "tid": 29722, "ts": 39377ns : performance_issue_unique : false
"pid": 29722, "tid": 29722, "ts": 39377ns : performance_issue_same : false
"pid": 29722, "tid": 29722, "ts": 39377ns : performance_issue_same_index : 0
"pid": 29722, "tid": 29722, "ts": 39377ns : performance_sample_count : 1024
"pid": 29722, "tid": 29722, "ts": 41278ns :
"pid": 29722, "tid": 29722, "ts": 41278ns : Requested Settings:
"pid": 29722, "tid": 29722, "ts": 41278ns : Scenario : Offline
"pid": 29722, "tid": 29722, "ts": 41278ns : Test mode : Performance
"pid": 29722, "tid": 29722, "ts": 41278ns : offline_expected_qps : 1
"pid": 29722, "tid": 29722, "ts": 41278ns : min_duration_ms : 60000
"pid": 29722, "tid": 29722, "ts": 41278ns : max_duration_ms : 0
"pid": 29722, "tid": 29722, "ts": 41278ns : min_query_count : 100
"pid": 29722, "tid": 29722, "ts": 41278ns : max_query_count : 100
"pid": 29722, "tid": 29722, "ts": 41278ns : qsl_rng_seed : 12786827339337101903
"pid": 29722, "tid": 29722, "ts": 41278ns : sample_index_rng_seed : 12640797754436136668
"pid": 29722, "tid": 29722, "ts": 41278ns : schedule_rng_seed : 3135815929913719677
"pid": 29722, "tid": 29722, "ts": 41278ns : accuracy_log_rng_seed : 0
"pid": 29722, "tid": 29722, "ts": 41278ns : accuracy_log_probability : 0
"pid": 29722, "tid": 29722, "ts": 41278ns : accuracy_log_sampling_target : 0
"pid": 29722, "tid": 29722, "ts": 41278ns : print_timestamps : false
"pid": 29722, "tid": 29722, "ts": 41278ns : performance_issue_unique : false
"pid": 29722, "tid": 29722, "ts": 41278ns : performance_issue_same : false
"pid": 29722, "tid": 29722, "ts": 41278ns : performance_issue_same_index : 0
"pid": 29722, "tid": 29722, "ts": 41278ns : performance_sample_count_override : 1024
"pid": 29722, "tid": 29722, "ts": 41278ns :
"pid": 29722, "tid": 29722, "ts": 45557ns : Starting performance mode:
"pid": 29722, "tid": 29722, "ts": 68453ns : Loading QSL : "set" : "[76,11,51,30,48,42,16,61,23,40,68,83,36,17,52,18,57,71,84,27,9,22,99,54,97,70,89,13,15,59,85,4,35,3,12,87,24,56,29,38,45,19,14,21,78,86,44,25,66,74,73,92,55,98,81,69,7,93,90,37,28,64,39,6,50,79,8,10,49,43,5,33,47,67,60,65,41,31,77,96,91,58,72,95,82,0,88,62,20,46,1,53,94,26,75,2,63,80,32,34]"
"pid": 29722, "tid": 29722, "ts": 79084725ns : GeneratedQueries: "queries" : 1, "samples per query" : 100, "duration" : 1000000000
"pid": 29722, "tid": 29722, "ts": 183673566ns :  Ending naturally: Minimum query count and test duration met.
"pid": 29722, "tid": 29722, "ts": 1322918310ns : Log Contention Counters:
"pid": 29722, "tid": 29722, "ts": 1322918310ns : 0 : swap_request_slots_retry_count
"pid": 29722, "tid": 29722, "ts": 1322918310ns : 0 : swap_request_slots_retry_retry_count
"pid": 29722, "tid": 29722, "ts": 1322918310ns : 0 : swap_request_slots_retry_reencounter_count
"pid": 29722, "tid": 29722, "ts": 1322918310ns : 0 : start_reading_entries_retry_count
"pid": 29722, "tid": 29722, "ts": 1322918310ns : 0 : tls_total_log_cas_fail_count
"pid": 29722, "tid": 29722, "ts": 1322918310ns : 0 : tls_total_swap_buffers_slot_retry_count
"pid": 29722, "tid": 29722, "ts": 1322921077ns : POWER_BEGIN: "mode" : Performance, "time" : 03-17-2021 10:28:43.663
"pid": 29722, "tid": 29722, "ts": 1322921077ns : POWER_END: "mode" : Performance, "time" : 03-17-2021 10:28:44.899
</details>
<details>
  <summary>mlperf_log_summary.txt</summary>
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : Performance
Samples per second: 80.9122
Result is : INVALID
  Min duration satisfied : NO
  Min queries satisfied : Yes
Recommendations:
 * Increase expected QPS so the loadgen pre-generates a larger (coalesced) query.

================================================
Additional Stats
================================================
Min latency (ns)                : 1180995762
Max latency (ns)                : 1235907967
Mean latency (ns)               : 1196566677
50.00 percentile latency (ns)   : 1189759990
90.00 percentile latency (ns)   : 1214026618
95.00 percentile latency (ns)   : 1214026618
97.00 percentile latency (ns)   : 1235907967
99.00 percentile latency (ns)   : 1235907967
99.90 percentile latency (ns)   : 1235907967

================================================
Test Parameters Used
================================================
samples_per_query : 100
target_qps : 1
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 60000
max_duration (ms): 0
min_query_count : 1
max_query_count : 100
qsl_rng_seed : 12786827339337101903
sample_index_rng_seed : 12640797754436136668
schedule_rng_seed : 3135815929913719677
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
accuracy_log_sampling_target : 0
print_timestamps : false
performance_issue_unique : false
performance_issue_same : false
performance_issue_same_index : 0
performance_sample_count : 1024

No warnings encountered during test.

No errors encountered during test.
</details>

Accuracy only run
```
./run_local.sh tf resnet50 gpu --scenario Offline --accuracy --count 100
```
<details>
  <summary>mlperf_log_accuracy.json</summary>
[
{ "seq_id" : 32, "qsl_idx" : 35, "data" : "00C05D44" }, 
{ "seq_id" : 33, "qsl_idx" : 3, "data" : "00404A44" },
{ "seq_id" : 34, "qsl_idx" : 12, "data" : "0000B943" },
...(100 lines)
]
</details>

I think the reason that I didn't see MLLOG is because I used r0.7 branch
I could see MLLOG details log after switching to the master branch,
but still couldn't see the following message in mlperf_log_details.txt
```
:::MLLOG {"key": "warning_generic_message", "value": "Found Audit Config file (audit.config). Overriding TestSettings from audit.config file.", (... etc.)
```
<details>
  <summary>mlperf_log_details.txt</summary>
:::MLLOG {"key": "loadgen_version", "value": ".5a1 @ 63c7df5b86", "time_ms": 0.005503, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "version.cc", "line_no": 54, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "loadgen_build_date_local", "value": "2021-03-17T13:21:57.713081", "time_ms": 0.005503, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "version.cc", "line_no": 55, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "loadgen_build_date_utc", "value": "2021-03-17T05:21:57.713093", "time_ms": 0.005503, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "version.cc", "line_no": 56, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "loadgen_git_commit_date", "value": "2021-03-16T07:44:00-07:00", "time_ms": 0.005503, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "version.cc", "line_no": 57, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "loadgen_git_log_message", "value": "63c7df5b86e26de38e6ca07c76347070100bfd06 Guard max_latency_ with latencies_mutex_ (#878)\n83af0658e069d7023358f4b0687360689487c1e2 New PyTorch ResNet50-v1.5 links (#876)\n8ef29262d91d1150944d6a2a16fbe52de905fe88 Add PyTorch ImageNet pre-processing (#861)\nc826478389cae32b785ad767e49efba3aefcc1e6 Add optional WAR for test04-B - offline test breaks contiguous memory requirements to stress SUT (#845)\n6294f23da3450bd84331138cecf506e7fbda08df Add flags and annotations links (#855)\na7512b33693b1fad0b6fb68c97a74e70ab3857e6 Change the way bert/run.py launches the accuracy-squad script by accounting the absolute path (#848)\ncf6cd3e75207c8c06316c73f2bbccb126669222f updates seeds in submission-checker (#852)\n9b7ab7d87bdc5c501352fb58d625e62bafef38ba fix docker images (#844)\n84bcae42946aec7c41959047afabd8d6c8536f4b Update Seeds to 1.0 (#847)\n2d41f9642cacd02f72b964425a7ddb50b0394ce8 submission packing script for https://github.com/mlcommons/policies/issues/61 (#846)\n34f8be57fa7f52f11aca2b81ee989279c037fe01 Add missing letters (#843)\n13a3c54fd577b59333ae17ae090ec02c57e43785 merge r1.0 back to master\nbe63f631dd5ff37ecc6a89c4a95791941e0199ae Add missing return statement (#839)\n946d95b730b7e7a3742043a5984438dae977f282 fix typo\ndf0221ee94570e6b161f75fdb8b55e6432931967 merge fix for 841\n55dabd38541045698cb093fdea82f625c4cc8a29 merge some changes from 1.0 branch back to master", "time_ms": 0.005503, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "version.cc", "line_no": 59, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "loadgen_git_status_message", "value": "", "time_ms": 0.005503, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "version.cc", "line_no": 61, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "loadgen_file_sha1", "value": {"/.clang-format":"012aad77e5206c89d50718c46c119d1f3cb056b2","/BUILD.gn":"b82905303c4756c4a2ab40ceed92cdddbf04bf2d","/CMakeLists.txt":"7067c2704e8de3be0a0fb9ef5984d52a88f9b91e","/README.md":"6f0405317cbd98c08d22958e2e43c6e8c9ba939e","/README_BUILD.md":"34b89a712a0ca1691b89715fe2ce459a676d0f77","/README_FAQ.md":"f75741b91085a20ebb23de1ce03d45a56d3457fe","/bindings/c_api.cc":"a00620bb4c9b583c1d2f5beef1a5e8d772aefb9b","/bindings/c_api.h":"81873e098bc60c7bc59045c8329ce153dd1ec055","/bindings/python_api.cc":"227149dc287e93a28da9ce92346158e4e403b694","/demos/BUILD.gn":"4b68d3399b31e14e554cff678a0db9b4d1299e23","/demos/py_demo_multi_stream.py":"07d85ea5fefade8d3d66ca832b783bf90e5bce7d","/demos/py_demo_multi_stream_free.py":"2f941e949770a2ea9ad6d6eea2f33fc19150edcd","/demos/py_demo_offline.py":"73c60cf12cfb69619249e637f9bcad18a47cda6c","/demos/py_demo_server.py":"02a6ddbe8b51bde2cfed32abeccdb7fdd9cfe4e7","/demos/py_demo_single_stream.py":"3e7d786882099ac784bf878f1cde9dd3db2f6d9f","/issue_query_controller.cc":"51609e637eb47121bf34af468e4e8d06cb37e667","/issue_query_controller.h":"703be7bdc58a8fa83555917f89054cbdb50038d4","/loadgen.cc":"3af7b13050292512605cff7cb87106b6fc00e3cc","/loadgen.h":"d9576ca8175b86f2074e95f8fe96743400cf4fa1","/loadgen_integration_diagram.svg":"47f748307536f80cfc606947b440dd732afc2637","/logging.cc":"4c9f151afd0740082680b576b91aec93857588a5","/logging.h":"6e662ad0cdbb30d81140b6a73e89f11200e1f834","/query_sample.h":"a879f127c4bb64d131056942734ea1a6ba9c3033","/query_sample_library.h":"61feb478b15668f48245c5083d559dee5de1c082","/setup.py":"bd41a2b3f4ed365566e411fddd012c66a320e87b","/system_under_test.h":"744482c84336420adbbe31ef8c2b39612c6ca3f8","/test_settings.h":"34eec1f8fb4a2db273b715686580f19cffa2f3a1","/test_settings_internal.cc":"036b1a1cab9ee3a9471e67e1f379c302ff095bcc","/test_settings_internal.h":"81603212f824ceefbf2d0fe88d2ef94c67d28f6e","/utils.cc":"c02bceb264acb94e86ef7a32f970dc46fe2b9356","/utils.h":"40775e32d619ea6356826ae5ea4174c7911f6894","/version.cc":"cbec2a5f98f9786c8c3d8b06b3d12df0b6550fa0","/version.h":"9d574baa64424e9c708fcfedd3dbb0b518a65fcc","/version_generator.py":"07dbee77c9707efdb95def65477c288fe6667e92"}, "time_ms": 0.005503, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "version.cc", "line_no": 68, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "test_datetime", "value": "2021-03-17T13:24:07Z", "time_ms": 0.026275, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 1534, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "sut_name", "value": "PySUT", "time_ms": 0.026275, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 1535, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "qsl_name", "value": "PyQSL", "time_ms": 0.026275, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 1536, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "qsl_reported_total_count", "value": 100, "time_ms": 0.026275, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 1537, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "qsl_reported_performance_count", "value": 100, "time_ms": 0.026275, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 1539, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_scenario", "value": "Offline", "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 265, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_test_mode", "value": "PerformanceOnly", "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 266, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_offline_expected_qps", "value": 1, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 293, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_min_duration_ms", "value": 600000, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 298, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_max_duration_ms", "value": 0, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 299, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_min_query_count", "value": 100, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 300, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_max_query_count", "value": 100, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 301, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_qsl_rng_seed", "value": 7322528924094909334, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 302, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_sample_index_rng_seed", "value": 1570999273408051088, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 303, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_schedule_rng_seed", "value": 3507442325620259414, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 304, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_accuracy_log_rng_seed", "value": 0, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 305, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_accuracy_log_probability", "value": 0, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 306, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_accuracy_log_sampling_target", "value": 0, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 307, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_print_timestamps", "value": false, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 308, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_performance_issue_unique", "value": false, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 309, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_performance_issue_same", "value": false, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 310, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_performance_issue_same_index", "value": 0, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 311, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "requested_performance_sample_count_override", "value": 1024, "time_ms": 0.041230, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 312, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_scenario", "value": "Offline", "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 383, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_test_mode", "value": "PerformanceOnly", "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 384, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_samples_per_query", "value": 660, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 386, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_target_qps", "value": 1, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 387, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_target_latency_ns", "value": 0, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 388, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_target_latency_percentile", "value": 0, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 389, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_max_async_queries", "value": 1, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 390, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_target_duration_ms", "value": 0, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 391, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_min_duration_ms", "value": 600000, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 392, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_max_duration_ms", "value": 0, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 393, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_min_query_count", "value": 1, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 394, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_max_query_count", "value": 100, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 395, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_min_sample_count", "value": 660, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 396, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_qsl_rng_seed", "value": 7322528924094909334, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 397, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_sample_index_rng_seed", "value": 1570999273408051088, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 398, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_schedule_rng_seed", "value": 3507442325620259414, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 399, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_accuracy_log_rng_seed", "value": 0, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 400, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_accuracy_log_probability", "value": 0, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 401, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_accuracy_log_sampling_target", "value": 0, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 402, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_print_timestamps", "value": false, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 403, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_performance_issue_unique", "value": false, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 404, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_performance_issue_same", "value": false, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 405, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_performance_issue_same_index", "value": 0, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 406, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "effective_performance_sample_count", "value": 1024, "time_ms": 0.043361, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "test_settings_internal.cc", "line_no": 407, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "generic_message", "value": "Starting performance mode", "time_ms": 0.057215, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 1183, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "loaded_qsl_set", "value": [48,99,86,56,39,37,12,97,14,44,52,96,79,88,60,94,19,75,84,29,11,22,66,31,82,55,18,15,93,78,83,46,81,53,10,74,85,51,68,42,61,89,58,76,9,35,3,6,54,90,34,80,28,23,69,72,71,47,26,45,92,13,1,73,95,20,57,43,32,36,38,8,30,41,7,24,59,64,91,65,33,63,98,62,16,70,50,21,40,49,5,77,4,0,17,2,27,67,87,25], "time_ms": 0.082663, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 953, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "generated_query_count", "value": 1, "time_ms": 78.424277, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 315, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "generated_samples_per_query", "value": 660, "time_ms": 78.424277, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 316, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "generated_query_duration", "value": 1000000000, "time_ms": 78.424277, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 317, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "generic_message", "value": "Ending naturally: Minimum query count and test duration met.", "time_ms": 645.158520, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "issue_query_controller.cc", "line_no": 551, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "logger_swap_request_slots_retry_count", "value": 0, "time_ms": 4057.997763, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "logging.cc", "line_no": 687, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "logger_swap_request_slots_retry_retry_count", "value": 0, "time_ms": 4057.997763, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "logging.cc", "line_no": 689, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "logger_swap_request_slots_retry_reencounter_count", "value": 0, "time_ms": 4057.997763, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "logging.cc", "line_no": 691, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "logger_start_reading_entries_retry_count", "value": 0, "time_ms": 4057.997763, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "logging.cc", "line_no": 693, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "logger_tls_total_log_cas_fail_count", "value": 0, "time_ms": 4057.997763, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "logging.cc", "line_no": 695, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "logger_tls_total_swap_buffers_slot_retry_count", "value": 0, "time_ms": 4057.997763, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "logging.cc", "line_no": 697, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "power_begin", "value": "03-17-2021 13:24:07.935", "time_ms": 4058.001711, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 455, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "power_end", "value": "03-17-2021 13:24:11.912", "time_ms": 4058.001711, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 457, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_validity", "value": "INVALID", "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 857, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_min_duration_met", "value": false, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 861, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_min_queries_met", "value": true, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 862, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_invalid_reason", "value": "Increase expected QPS so the loadgen pre-generates a larger (coalesced) query. ", "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 875, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_samples_per_second", "value": 165.955, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 931, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_min_latency_ns", "value": 3354550922, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 937, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_max_latency_ns", "value": 3976971838, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 938, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_mean_latency_ns", "value": 3560759161, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 939, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_50.00_percentile_latency_ns", "value": 3533685566, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 944, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_90.00_percentile_latency_ns", "value": 3742507643, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 944, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_95.00_percentile_latency_ns", "value": 3749547165, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 944, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_97.00_percentile_latency_ns", "value": 3976971838, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 944, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_99.00_percentile_latency_ns", "value": 3976971838, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 944, "pid": 6592, "tid": 6592}}
:::MLLOG {"key": "result_99.90_percentile_latency_ns", "value": 3976971838, "time_ms": 4058.267687, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 944, "pid": 6592, "tid": 6592}}
</details>


@BeyondCloud I think you should define the output directory explicitly instead of using the default one.
Please try to use `--output` parameter of `run_local.sh` as in this example:
```
    # define output directories
    t01_out=$SUBMISSION_DIR/compliance/$system_id/$model/$scenario
    t01_out_offline=$SUBMISSION_DIR/compliance/$system_id/$model/Offline
    t01_dir=$t01_out/run
    t01_perf=$t01_dir/performance/run_1
    t01_acc=$t01_dir/accuracy
    
    # create them in advance
    mkdir -p $t01_perf
    mkdir -p $t01_acc
    
    # copy the audit file to the output directory
    cp ../../compliance/nvidia/TEST01/$model/audit.config $t01_perf/
    cp ../../compliance/nvidia/TEST01/$model/audit.config $t01_acc/
    
    # define the output directory explicitly when running `run_local.sh`
    ./run_local.sh $@ --scenario $scenario --output $t01_perf
    ./run_local.sh $@ --scenario $scenario --accuracy --output $t01_acc
```

Exactly what I am looking for. I appreciate it.