Repository: mlcommons/inference
Issue: Something wrong with ds.last_loaded
Root cause: Algorithmic Error

If my understanding is correct, ds.last_loaded is supposed to capture the timestamp of when the dataset is last loaded. However, I think it's used incorrectly in the main.py script:
v0.5/classification_and_detection/python/main.py

    add_results(final_results, "{}".format(scenario),
                result_dict, last_timeing, time.time() - ds.last_loaded, args.accuracy)

The difference time.time() - ds.last_loaded is in my understanding trying to capture the runtime of the network. However, ds.last_loaded is repeatedly called while the model is running. As a result, we end up with an incorrect calculation of qps in add_results.

In particular, I think this only manifests when the dataset is large enough for ds to load images after lg.StartTest is called.

Here is the output:
TestScenario.SingleStream qps=218.86, mean=0.0250, time=22.846, acc=77.200%, queries=5000, tiles=50.0:0.0244,80.0:0.0280,90.0:0.0298,95.0:0.0314,99.0:0.0348,99.9:0.0385
accuracy=77.200%, good=3860, total=5000

If we take mean as an example, 1/0.0250 ~= 40, which should be close enough to qps. Instead qps is calculated to be much higher because last_loaded has been updated after lg.StartTest was called.


The result tracking in the reference app pre-dates loadgen - we’ll remove it going forward.
Only the loadgen output (aka mlperf_log_summary.txt ) is the truth for the submission.

From: ggeor84 <notifications@github.com>
Sent: Thursday, February 6, 2020 5:33 PM
To: mlperf/inference <inference@noreply.github.com>
Cc: Subscribed <subscribed@noreply.github.com>
Subject: [mlperf/inference] Something wrong with ds.last_loaded (#559)


If my understanding is correct, ds.last_loaded is supposed to capture the timestamp of when the dataset is last loaded. However, I think it's used incorrectly in the main.py script:
v0.5/classification_and_detection/python/main.py

add_results(final_results, "{}".format(scenario),

            result_dict, last_timeing, time.time() - ds.last_loaded, args.accuracy)

The difference time.time() - ds.last_loaded is in my understanding trying to capture the runtime of the network. However, ds.last_loaded is repeatedly called while the model is running. As a result, we end up with an incorrect calculation of qps in add_results.

In particular, I think this only manifests when the dataset is large enough for ds to load images after lg.StartTest is called.

Here is the output:
TestScenario.SingleStream qps=218.86, mean=0.0250, time=22.846, acc=77.200%, queries=5000, tiles=50.0:0.0244,80.0:0.0280,90.0:0.0298,95.0:0.0314,99.0:0.0348,99.9:0.0385
accuracy=77.200%, good=3860, total=5000

If we take mean as an example, 1/0.0250 ~= 40, which should be close enough to qps. Instead qps is calculated to be much higher because last_loaded has been updated after lg.StartTest was called.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmlperf%2Finference%2Fissues%2F559%3Femail_source%3Dnotifications%26email_token%3DAFPA3CHCZ7ZWJZGVB2K7XFLRBS25LA5CNFSM4KRG43CKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4ILVZVHQ&data=02%7C01%7Cguschmue%40microsoft.com%7C8fa58306cc22425d78a508d7ab6db0da%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637166359914044765&sdata=XFW7sYQt6WWzSO90XlV604djxj4%2FBkxyzaQm1%2Febqao%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFPA3CDF6YO7B4F2ZKK7HDDRBS25LANCNFSM4KRG43CA&data=02%7C01%7Cguschmue%40microsoft.com%7C8fa58306cc22425d78a508d7ab6db0da%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637166359914054731&sdata=8ew7RSwtiBDd0YhJAKRHqu0ckeatwme31zr4Z15CsHs%3D&reserved=0>.


mlperf_log_summary.txt does not contain any information:

No warnings encountered during test.

1 ERROR encountered. See detailed log.

Here is mlperf_log_detail.txt

"pid": 23352, "tid": 23352, "ts": 3493ns : LoadgenVersionInfo:
"pid": 23352, "tid": 23352, "ts": 3493ns : version : .5a1 @ 16a5661eea
"pid": 23352, "tid": 23352, "ts": 3493ns : build_date_local : 2020-02-05T15:21:45.159247
"pid": 23352, "tid": 23352, "ts": 3493ns : build_date_utc   : 2020-02-05T23:21:45.159256
"pid": 23352, "tid": 23352, "ts": 3493ns : git_commit_date  : 2019-12-17T09:41:32-08:00
"pid": 23352, "tid": 23352, "ts": 3493ns : git_log :

16a5661eea8f0545e04c86029362e22113c2ec09 Merge pull request #254 from lji72/master
cb4d2f32a912f50fa67fd4f48114f4ab5b767a75 Update README.md
c811fa52f0389ca98b94198574e4f80b61fb440a Update README.md
876c6e2e390b188d69675a59a71360ab39007bde support for tf-2.0 (#533)
37c605b087d060c003e5b47cebc5b12fd3d57ed4 Update README.md
28115aadab5becd36578517ce96f5e9627845364 Update README.md
311f017bca643dd72da895896b7afac87c35c29d Update README.md
65321a839046d1c4bdc202648c5ba26993f4af96 Update README.md
7a084d39c9324470e489d93c5d4917f13d62bd91 remove assert which blocks runs (#502)
5047bfd1ac5b34dead6b2e91cd1e58e1e153995f Bump pillow from 5.2.0 to 6.2.0 in /others/cloud/single_stage_detector (#526)
9acf3fbae88649b22eb4513b211c4fd1131d2b3c Bump werkzeug from 0.14.1 to 0.15.3 in /others/cloud/language_modeling (#524)
0af8aabe6e938bf3272b995c66e7bc4729d9c02a loadgen: Fix spurious offline error. (#479)
d7c7d4c0fc7fb8b707bb2c5ea37d811ea03a7112 tool to create a csv from the submission directory (#523)
411e090d946a6fed5af126c76caf6d31b8dab261 undelete test_settings.h (#521)
675557a5ea3f2c9012d488095270034d337f4088 some fixes for checker (#511)
40d7ed775b49cd1eb805fdb730be27ba30aa6d19 Missing `)`. (#516)

"pid": 23352, "tid": 23352, "ts": 3493ns : git_status :

 M ../v0.5/classification_and_detection/python/backend_onnxruntime.py
 M ../v0.5/classification_and_detection/python/main.py
 M ../v0.5/classification_and_detection/python/version.py

"pid": 23352, "tid": 23352, "ts": 3493ns : ERROR : Loadgen built with uncommitted changes!
"pid": 23352, "tid": 23352, "ts": 3493ns : SHA1 of files :

012aad77e5206c89d50718c46c119d1f3cb056b2 /.clang-format
a3355ff8f163cabbf26adb86d3aa0e5da72900c7 /BUILD.gn
25ddf9bbe1429de0d036f073497c1e90cb419519 /CMakeLists.txt
79fbce4843967690e0750fd7ffc021f4f1e4c5c4 /README.md
34b89a712a0ca1691b89715fe2ce459a676d0f77 /README_BUILD.md
f75741b91085a20ebb23de1ce03d45a56d3457fe /README_FAQ.md
fa2add871dc7440cbda1f8c7a69e96eb6499951a /bindings/c_api.cc
8fdaa27778cf3ee2eb85d5b7040e8ddfcab68108 /bindings/c_api.h
9c778599614f2a42cb95ad662213cde5fcf791ec /bindings/python_api.cc
4b68d3399b31e14e554cff678a0db9b4d1299e23 /demos/BUILD.gn
07d85ea5fefade8d3d66ca832b783bf90e5bce7d /demos/py_demo_multi_stream.py
2f941e949770a2ea9ad6d6eea2f33fc19150edcd /demos/py_demo_multi_stream_free.py
73c60cf12cfb69619249e637f9bcad18a47cda6c /demos/py_demo_offline.py
02a6ddbe8b51bde2cfed32abeccdb7fdd9cfe4e7 /demos/py_demo_server.py
3e7d786882099ac784bf878f1cde9dd3db2f6d9f /demos/py_demo_single_stream.py
7da06e033de9e271568c49d3bc129bbce166e1bd /loadgen.cc
5526ae8ce9547a95f10e1ac465056bfba2fccb47 /loadgen.h
47f748307536f80cfc606947b440dd732afc2637 /loadgen_integration_diagram.svg
9960bfaf44da0925444c74d54e1b97d75bf5e812 /logging.cc
fdb61b8a2e2593bab4ae5bf9758434cf6a60e218 /logging.h
cc0a677c2a5ef488c53496a43be056847f1dd5b1 /mlperf_loadgen.cpython-36m-x86_64-linux-gnu.so
a879f127c4bb64d131056942734ea1a6ba9c3033 /query_sample.h
922acb63aa734d36e3fa34d87ff0f361052e0c51 /query_sample_library.h
369f4320e027fc276d2519021696c7d1027b7320 /setup.py
744482c84336420adbbe31ef8c2b39612c6ca3f8 /system_under_test.h
ad3fa5b4e0395e77cd5e00ebae2d0a0eb538969d /test_settings.h
56a654a4bd7e5f5faef94189c21ddcbfba3f7008 /test_settings_internal.cc
1c15731be5abc26a6c90356720146862da7c22b4 /test_settings_internal.h
86fc4c8035e7b73132c56d1781aca647b6bd6916 /utils.cc
5395a04f82e90288f52bf918d7b1d6027bfe0964 /utils.h
e1c46896da912ff2a0187276c3a8ba4eec057a17 /version.cc
9d574baa64424e9c708fcfedd3dbb0b518a65fcc /version.h
07dbee77c9707efdb95def65477c288fe6667e92 /version_generator.py

"pid": 23352, "tid": 23352, "ts": 11165ns : Date + time of test: 2020-02-07T12:59:52Z
"pid": 23352, "tid": 23352, "ts": 11165ns : System Under Test (SUT) name: PySUT
"pid": 23352, "tid": 23352, "ts": 11165ns : Query Sample Library (QSL) name: PyQSL
"pid": 23352, "tid": 23352, "ts": 11165ns : QSL total size: 5000
"pid": 23352, "tid": 23352, "ts": 11165ns : QSL performance size*: 500
"pid": 23352, "tid": 23352, "ts": 11165ns : *TestSettings (performance_sample_count_override) can override
"pid": 23352, "tid": 23352, "ts": 11165ns : *Refer to Effective Settings for actual value
"pid": 23352, "tid": 23352, "ts": 17238ns : 
"pid": 23352, "tid": 23352, "ts": 17238ns : Effective Settings:
"pid": 23352, "tid": 23352, "ts": 17238ns : Scenario : Single Stream
"pid": 23352, "tid": 23352, "ts": 17238ns : Test mode : Accuracy
"pid": 23352, "tid": 23352, "ts": 17238ns : samples_per_query : 1
"pid": 23352, "tid": 23352, "ts": 17238ns : target_qps : 100
"pid": 23352, "tid": 23352, "ts": 17238ns : target_latency (ns): 0
"pid": 23352, "tid": 23352, "ts": 17238ns : target_latency_percentile : 0.9
"pid": 23352, "tid": 23352, "ts": 17238ns : max_async_queries : 1
"pid": 23352, "tid": 23352, "ts": 17238ns : target_duration (ms): 60000
"pid": 23352, "tid": 23352, "ts": 17238ns : min_duration (ms): 60000
"pid": 23352, "tid": 23352, "ts": 17238ns : max_duration (ms): 0
"pid": 23352, "tid": 23352, "ts": 17238ns : min_query_count : 5000
"pid": 23352, "tid": 23352, "ts": 17238ns : max_query_count : 5000
"pid": 23352, "tid": 23352, "ts": 17238ns : min_sample_count : 5000
"pid": 23352, "tid": 23352, "ts": 17238ns : qsl_rng_seed : 3133965575612453542
"pid": 23352, "tid": 23352, "ts": 17238ns : sample_index_rng_seed : 665484352860916858
"pid": 23352, "tid": 23352, "ts": 17238ns : schedule_rng_seed : 3622009729038561421
"pid": 23352, "tid": 23352, "ts": 17238ns : accuracy_log_rng_seed : 0
"pid": 23352, "tid": 23352, "ts": 17238ns : accuracy_log_probability : 0
"pid": 23352, "tid": 23352, "ts": 17238ns : print_timestamps : false
"pid": 23352, "tid": 23352, "ts": 17238ns : performance_issue_unique : false
"pid": 23352, "tid": 23352, "ts": 17238ns : performance_issue_same : false
"pid": 23352, "tid": 23352, "ts": 17238ns : performance_issue_same_index : 0
"pid": 23352, "tid": 23352, "ts": 17238ns : performance_sample_count : 1024
"pid": 23352, "tid": 23352, "ts": 17586ns : 
"pid": 23352, "tid": 23352, "ts": 17586ns : Requested Settings:
"pid": 23352, "tid": 23352, "ts": 17586ns : Scenario : Single Stream
"pid": 23352, "tid": 23352, "ts": 17586ns : Test mode : Accuracy
"pid": 23352, "tid": 23352, "ts": 17586ns : single_stream_expected_latency_ns : 10000000
"pid": 23352, "tid": 23352, "ts": 17586ns : single_stream_target_latency_percentile : 0.9
"pid": 23352, "tid": 23352, "ts": 17586ns : min_duration_ms : 60000
"pid": 23352, "tid": 23352, "ts": 17586ns : max_duration_ms : 0
"pid": 23352, "tid": 23352, "ts": 17586ns : min_query_count : 5000
"pid": 23352, "tid": 23352, "ts": 17586ns : max_query_count : 5000
"pid": 23352, "tid": 23352, "ts": 17586ns : qsl_rng_seed : 3133965575612453542
"pid": 23352, "tid": 23352, "ts": 17586ns : sample_index_rng_seed : 665484352860916858
"pid": 23352, "tid": 23352, "ts": 17586ns : schedule_rng_seed : 3622009729038561421
"pid": 23352, "tid": 23352, "ts": 17586ns : accuracy_log_rng_seed : 0
"pid": 23352, "tid": 23352, "ts": 17586ns : accuracy_log_probability : 0
"pid": 23352, "tid": 23352, "ts": 17586ns : print_timestamps : false
"pid": 23352, "tid": 23352, "ts": 17586ns : performance_issue_unique : false
"pid": 23352, "tid": 23352, "ts": 17586ns : performance_issue_same : false
"pid": 23352, "tid": 23352, "ts": 17586ns : performance_issue_same_index : 0
"pid": 23352, "tid": 23352, "ts": 17586ns : performance_sample_count_override : 1024
"pid": 23352, "tid": 23352, "ts": 17586ns : 
"pid": 23352, "tid": 23352, "ts": 19581ns : Starting accuracy mode:
"pid": 23352, "tid": 23352, "ts": 130011ns : Loading QSL : "set" : "[4927,3740,3179,4579,952,3046,2067,2039,3434,115,3229,471,52,3167,4250,4726,4107,1857,505,2320,4749,790,3505,3053,1601,1994,717,665,1775,323,3278,4004,1147,1907,736,2865,2377,1694,2344,3414,1986,1893,1609,4413,697,1817,4207,3992,1525,1059,520,789,1384,3266,857,4224,2673,421,945,2126,964,541,874,2759,3701,455,4099,1566,2454,1659,1157,1462,3132,2676,816,4955,486,2754,698,3211,2919,4760,1616,4863,4319,427,1688,2382,4428,4774,1997,3361,1445,1274,1403,975,826,1266,2198,587,3324,1090,2871,449,3344,3646,617,462,128,484,3452,3845,4733,3863,1063,1945,2228,1983,836,1734,44,661,3996,2937,2027,1912,1816,2236,1430,4618,3638,4564,3123,3877,4028,42,4009,2571,2150,3339,4147,3943,853,237,2267,4331,4293,3169,966,2989,4346,4898,2506,3430,2747,2351,1716,4003,3702,1415,3517,3082,4685,3074,1950,101,2461,3187,2757,46,633,3621,1010,2834,3426,822,4324,3634,4545,18,2657,1977,2028,3252,1490,2181,710,1138,4458,1113,3357,2186,4956,4533,4688,1913,3299,2105,4435,831,4722,925,1537,2866,1038,4261,911,1980,2343,4558,4272,2614,11,2804,1381,1953,2457,4729,1241,3836,4140,1285,2065,4759,1627,3402,443,3696,4521,3916,3054,2085,2258,4334,3670,4608,3319,1660,3404,1626,4339,1040,3560,2875,3440,4067,4742,3553,2684,1373,4707,2398,2241,3043,1558,3283,4546,4658,3290,4045,1326,1812,3365,1562,1468,1879,732,3442,2557,4979,2121,2448,725,1896,1975,2788,1265,2631,2502,3010,4308,537,1344,3527,3514,4842,3005,4415,2250,2544,1966,4807,3977,1175,1498,4695,204,1929,4641,1973,4363,4291,3429,4189,2781,2152,2061,4974,955,4629,2253,588,4594,1592,2611,1888,3795,692,2818,4437,1416,2407,868,1855,2613,1393,1633,405,426,2874,273,2662,397,33,3340,706,1667,4628,1809,4192,662,1540,3749,3585,2096,4167,1166,1151,2715,4741,607,1813,4494,2072,4159,3518,1052,562,1428,1480,4512,62,4735,4399,3674,4234,2301,1753,2429,1331,3706,2526,304,3576,1509,1085,928,391,4011,677,2332,1289,2927,245,4757,2683,459,1016,801,4095,2325,3301,814,3450,3309,1346,2921,3291,765,877,4821,4473,572,3874,4600,341,4510,890,4304,3392,1772,1021,3962,3401,3902,1374,3312,4161,2485,2953,643,2165,4439,4323,2528,2200,4135,3215,2664,1606,1969,4400,2988,1949,907,199,1112,3625,4423,4349,4141,2162,2455,1928,2318,257,1561,4407,659,4080,134,3779,2623,3538,3022,3382,4137,3669,1015,3095,476,4240,2793,2089,4535,2972,2750,438,3718,3601,3768,762,3238,746,4134,124,3214,1392,1957,2533,2304,2679,163,3273,2700,3833,3951,3359,4165,349,2420,3223,3310,622,3655,1853,2058,4507,392,2851,1181,2163,208,2077,409,4845,4587,3678,2540,3050,1220,3753,4804,1272,2580,434,2507,465,3838,198,2632,3891,620,1161,2056,4822,1510,3794,4817,1519,2199,2785,3276,1164,2194,1532,1661,3367,1002,1833,1077,2080,3358,2306,1987,2923,4000,4402,1407,173,938,4063,315,4262,750,1748,939,412,3185,618,1243,2815,366,2406,994,2164,2601,3040,1434,843,4305,1246,4221,3006,2295,674,1314,589,1848,1773,4007,131,287,1441,1650,1585,1466,1500,1133,3173,4412,235,672,2354,1418,4410,817,1647,1889,244,3147,547,3400,3294,1452,3792,4705,2334,3822,1023,2546,898,3018,3445,4263,1720,3315,1547,819,2957,1187,1221,4947,3552,189,4191,4849,764,2605,4985,4493,1340,2719,262,1032,2707,3923,1119,3588,3014,479,4784,4408,1815,3013,2691,1563,1150,3471,4984,4150,4499,540,2877,151,337,3540,3741,1823,2895,2452,1425,929,3052,1835,2249,2453,4350,2816,104,654,4890,2820,1048,2760,4998,4472,311,775,579,251,691,3613,1457,2006,4748,4296,3542,1698,3160,338,2417,2278,2172,329,1671,3830,1548,4313,4321,1941,2619,2117,4679,626,4143,1169,2675,3493,3298,3259,1210,3494,3881,3176,4241,2639,2930,74,3111,2764,2475,609,1212,182,1193,1906,557,4178,3866,4205,3964,1064,4344,3378,4836,2045,915,4219,3369,2520,29,1153,2739,1565,1125,4789,1831,2051,3002,4385,3133,1538,2292,4826,145,4665,1092,1014,2913,2530,166,2168,4917,4216,130,612,2431,4021,1115,4580,871,3592,283,2489,4217,2305,590,2586,388,1632,395,2527,1049,4042,958,1386,3876,2863,1129,1173,833,1819,2847,3264,3207,2316,1911,1611,2224,2918,3954,3809,2629,4487,4949,1830,1368,2529,1067,2720,10,782,810,3117,3092,4213,1191,91,3965,2034,4718,1350,1708,4193,3716,842,2624,2545,3271,4271,4966,2219,328,2127,4351,431,1681,1190,275,497,2751,2736,3879,1554,2646,2976,2496,1107,2886,317,2647,2209,2104,2076,1222,3230,194,2446,3217,1810,917,1060,548,2263,3183,3773,3447,3583,718,723,3939,1617,2794,203,3948,4235,2385,2772,1263,724,1195,555,1176,4686,3108,704,1422,1801,1728,839,998,336,140,39,4995,1842,2146,2179,2125,3697,363,3422,1897,4652,3245,3077,4388,1172,784,3851,1447,996,2447,406,4053,2940,3915,238,4626,4647,4367,3757,1755,3235,120,3672,2358,3496,3221,3852,3366,2255,2412,1311,4661,3469,4176,2092,4017,2282,3847,2414,2537,1590,2543,1301,3608,3926,670,3654,2387,2233,1961,3704,2525,1312,713,1397,1104,3776,3448,2762,1183,4119,79,705,1414,3150,320,4763,2570,4620,269,4873,2659,3044,4531,3675,2177,2805,4881,2437,3720,2415,2182,3772,4959,4317,4190,1781,1293,2746,4783,4302,1779,3190,4153,2176,4275,2640,3477,2725,4046,3237,1663,4912,3398,1004,1249,1545,1641,1372,2682,2368,4005,648,1337,4260,967,1036,824,2974,3437]"
"pid": 23352, "tid": 23352, "ts": 394994611ns : GeneratedQueries: "queries" : 1024, "samples per query" : 1, "duration" : 10240000000
"pid": 23352, "tid": 23352, "ts": 26211786406ns : Log Contention Counters:
"pid": 23352, "tid": 23352, "ts": 26211786406ns : 0 : swap_request_slots_retry_count
"pid": 23352, "tid": 23352, "ts": 26211786406ns : 0 : swap_request_slots_retry_retry_count
"pid": 23352, "tid": 23352, "ts": 26211786406ns : 0 : swap_request_slots_retry_reencounter_count
"pid": 23352, "tid": 23352, "ts": 26211786406ns : 0 : start_reading_entries_retry_count
"pid": 23352, "tid": 23352, "ts": 26211786406ns : 0 : tls_total_log_cas_fail_count
"pid": 23352, "tid": 23352, "ts": 26211786406ns : 0 : tls_total_swap_buffers_slot_retry_count
"pid": 23352, "tid": 23352, "ts": 26211786951ns : POWER_BEGIN: "mode" : Accuracy, "time" : 02-07-2020 12:59:52.773
"pid": 23352, "tid": 23352, "ts": 26211786951ns : POWER_END: "mode" : Accuracy, "time" : 02-07-2020 13:00:18.588
"pid": 23352, "tid": 23352, "ts": 26231941392ns : Loading QSL : "set" : "[3396,2862,4921,2341,2395,2769,2741,3274,3953,2677,3007,1789,1037,2386,84,3188,4197,2926,4862,4169,4983,808,848,213,646,2088,4432,4111,3462,4867,1483,2846,3475,3878,3502,482,1390,59,2932,169,3204,4649,4379,4796,469,458,4027,1985,900,350,4475,1042,1622,448,454,2379,297,3328,25,4200,3802,4386,3895,4179,3733,93,136,807,2901,4570,1874,2716,3619,913,4952,3065,3556,4516,6,4380,1521,2681,1550,4575,3591,2103,2630,1654,23,4186,4215,1618,798,4824,1631,2138,4778,4556,2216,1520,313,1822,3269,2007,2456,2309,675,1269,3937,1080,651,289,4371,4194,2644,1313,1442,3316,1739,4337,1690,1856,107,3818,3000,564,452,1095,1322,4714,401,3472,2724,743,1165,3399,1837,3673,3417,2433,1154,1995,466,346,3746,1105,3222,1653,669,2811,531,1114,4358,386,218,1595,1580,2195,3717,2702,14,2012,3466,2297,4401,2086,3313,2321,2408,1818,1424,2050,3470,3886,1820,2436,507,4728,4734,1594,3227,1767,3826,1724,4214,2650,3559,1934,474,2465,1197,393,3081,1944,480,1919,3200,4806,4737,4329,3858,67,2518,66,1495,2810,4404,3549,2602,1359,3486,2220,2285,2427,2066,511,2595,2859,797,783,4430,1124,3416,3543,132,428,347,4585,4036,4230,4168,4022,4229,142,4348,2952,971,884,1007,4295,1250,3584,1427,2947,4765,3645,1774,2153,3118,2279,1993,3986,326,3609,1758,2574,4659,2559,4941,3240,1762,4630,4411,536,4382,1892,4290,127,1705,4393,4020,3759,2660,3831,4894,2588,1054,485,343,2956,4519,129,2696,1214,1636,2590,3446,2075,396,4145,3151,2196,3618,4157,4711,4222,230,2212,4061,4251,286,4929,2765,4814,3970,1499,930,4611,4709,920,1026,2319,760,4968,247,3306,2566,1786,4488,3287,880,1841,1867,1881,4314,4471,4247,575,1475,823,860,1505,423,4940,3555,1942,1714,3532,3023,776,1646,673,2174,3777,3206,3326,2575,123,38,591,4977,2893,2284,3490,3164,444,1864,3547,2572,2022,4935,1338,2779,1624,3607,876,3030,2426,90,2482,4056,1356,1347,3231,1706,3961,1612,1504,2790,1440,3062,667,827,4552,2118,4491,2082,3388,1658,4071,2561,2160,3708,2157,26,1061,2514,4899,2327,1251,1741,3191,2112,2621,2539,4543,4657,1803,1143,1342,2821,918,3824,429,4332,3333,456,3041,4280,4964,57,4978,4860,2188,1639,4747,3089,1530,1640,4208,1645,4571,687,2178,3360,411,270,1208,1127,3020,4619,144,4638,4550,4918,155,3859,4717,3184,1572,908,219,3438,2803,1467,1051,919,3136,2166,3644,1030,4969,3025,2268,4678,3800,4481,4928,3284,4376,501,4164,2430,1502,1976,1890,2610,2573,4555,3744,380,3705,3331,1898,4406,1747,2068,3435,4976,4696,3467,2948,4636,271,636,3024,3869,2531,12,450,139,1237,2029,1577,3722,1008,519,2903,4043,2422,4588,4810,1875,4857,1435,362,1702,1962,1271,1960,4739,1207,2393,1035,892,80,2074,4292,1444,2247,4865,1334,3525,4436,598,422,3474,265,4105,1219,694,3668,514,655,361,3957,2711,475,2180,1771,4888,4445,4162,3683,3482,2551,3667,4590,4526,4727,1838,4032,3763,1317,2478,4780,1718,3593,3958,1096,984,957,2958,3285,3028,3797,2934,4853,4650,4343,2151,2230,3600,3564,4035,1575,2055,87,1072,2904,1482,4052,1291,242,2924,1722,1305,2286,2912,4315,2581,1712,1662,296,3648,4506,1310,1808,4664,582,1132,780,1943,2053,1383,639,1501,854,4466,770,3098,3561,546,2908,4811,285,2612,2113,2898,1643,973,1365,2828,3498,4449,3455,4484,545,2685,3125,4244,3393,4454,2798,108,936,3856,4539,1996,3850,1689,2670,2466,3739,657,565,4850,685,862,4087,1711,2281,2963,4896,299,4762,1926,1691,2322,834,4274,135,601,2880,4909,1675,727,530,4751,3342,3069,432,1473,2845,3567,3513,3750,2442,4773,3817,1915,1028,298,1101,234,2470,2938,3162,1046,1395,2257,1970,1178,4270,2109,1494,20,3698,2347,778,78,4026,254,4667,4252,2350,2813,1978,3784,4942,830,2102,2123,154,821,2215,3436,2003,2472,2149,2038,1382,1766,4278,1707,340,703,2122,4496,4576,2674,1523,3864,0,2235,3408,211,924,4482,2795,2333,1571,2291,2423,2775,3244,281,2473,4242,3798,2564,49,3380,324,2234,1260,2899,370,4094,3934,3693,1635,4138,1839,914,342,308,4209,4677,2394,3689,1630,4601,990,3080,2625,2037,699,2704,4936,1686,2822,3928,2749,3642,2361,4394,1489,3415,2439,4060,550,615,1073,3241,1377,2695,3463,2881,3596,68,1948,688,4584,2922,3352,4096,291,3522,3993,4809,3888,1866,1159,8,1798,2783,141,4524,1843,2653,4092,103,595,159,4175,4680,1287,4621,3703,1619,4523,4992,4536,2266,3348,2891,2887,3639,2463,146,4653,1593,2975,224,2606,63,3303,2857,553,2678,656,1955,1062,2000,2905,602,3837,1991,1703,4511,1409,3523,2789,171,53,3449,3048,4398,4457,1123,229,668,1863,3021,3575,977,3684,2271,4195,2876,3337,4320,2097,3572,2191,1564,4919,3115,2535,3811,4668,4529,3819,3424,4771,4300,1020,726,963,4115,3425,4602,2824,4880,4124,1458,3530,223,3719,552,502,3562,3945,1806,2244,4180,2450,1885,3083,2087,1732,3504,4870,282,3662,2404,221,568,2832,4248,1273,905,2808,1179,1134,3589,3995,377,3346,763,4815,4893,3605,1254,840,314,2013,4616,800,2508,869,4615,4490,2261,1145,2964,51,457,4504,2491,4885,4122,1951,3161,1965,2708,2620,2954,2084,284,4879,3114,1091,1852,1731,408,4549,3506,2638,3371,1597,4708,1735,1860,767,4149,1655,2190,1121,433,3846,4132,2990,3737,430,3805,81,2140,2083,1683,2008,947,1745,2758,4712,3898,2462,525,303,3840,993,306,445]"
"pid": 23352, "tid": 23352, "ts": 26636886770ns : GeneratedQueries: "queries" : 1024, "samples per query" : 1, "duration" : 10240000000
"pid": 23352, "tid": 23352, "ts": 52410801813ns : Log Contention Counters:
"pid": 23352, "tid": 23352, "ts": 52410801813ns : 0 : swap_request_slots_retry_count
"pid": 23352, "tid": 23352, "ts": 52410801813ns : 0 : swap_request_slots_retry_retry_count
"pid": 23352, "tid": 23352, "ts": 52410801813ns : 0 : swap_request_slots_retry_reencounter_count
"pid": 23352, "tid": 23352, "ts": 52410801813ns : 0 : start_reading_entries_retry_count
"pid": 23352, "tid": 23352, "ts": 52410801813ns : 0 : tls_total_log_cas_fail_count
"pid": 23352, "tid": 23352, "ts": 52410801813ns : 0 : tls_total_swap_buffers_slot_retry_count
"pid": 23352, "tid": 23352, "ts": 52410802352ns : POWER_BEGIN: "mode" : Accuracy, "time" : 02-07-2020 13:00:19.015
"pid": 23352, "tid": 23352, "ts": 52410802352ns : POWER_END: "mode" : Accuracy, "time" : 02-07-2020 13:00:44.787
"pid": 23352, "tid": 23352, "ts": 52430685309ns : Loading QSL : "set" : "[1657,1011,679,829,3239,4639,838,1587,3012,2942,4683,4391,3812,2264,2495,4030,832,1588,4392,4409,4713,4426,1999,943,3322,2217,1336,1603,1204,1901,1406,4226,3769,4340,4495,1003,1471,2208,1255,4461,1918,866,3755,2493,2246,3659,1033,3439,1865,2299,4048,4040,111,820,4613,3626,1746,339,4073,4663,2302,1182,2270,3412,2499,576,4188,4090,3386,4691,3257,739,3713,2154,2418,4997,358,1938,4799,4259,3029,268,3897,567,2809,4403,3660,3528,2915,4937,3751,3989,1093,3849,1682,4255,730,4892,3907,3832,2296,2111,2777,1858,3484,3325,1396,3113,3174,4462,2658,2046,4434,533,4895,4670,2879,1450,720,846,4805,3571,2883,1047,2842,2800,1258,3566,2806,4855,1750,4497,1199,2885,477,1579,658,1637,4973,292,2497,3539,2205,1226,4103,356,2936,3825,4478,288,599,2120,3362,1733,3226,3930,2910,3411,1493,114,4786,2308,3017,1923,1751,4957,4889,3979,3464,1917,3201,16,3119,1158,4859,1459,3391,2556,2873,4767,4869,2637,2935,850,3444,70,158,962,118,4114,2009,212,1188,447,2712,387,2609,1379,3835,1873,3727,3686,2884,4716,419,3781,2023,2323,2723,3925,1832,305,1583,272,806,2703,4018,682,2124,47,3573,2823,301,4761,2554,2063,3904,774,1230,4730,1491,1144,2549,1472,583,809,4381,4144,385,1846,3421,882,3661,372,985,4012,1936,4170,4939,4823,3857,1794,2374,1723,3138,496,4702,605,4265,4655,3457,4612,942,2231,1599,2371,1111,492,3969,2019,3647,1910,4554,2396,3707,4971,4723,3038,1257,1959,4593,2780,4835,3260,2425,3512,3871,4281,4284,2911,4429,394,3428,2986,1106,2137,906,1488,2726,65,1727,904,2277,4990,3296,359,4441,4932,4370,3884,759,2835,2721,1420,3807,4442,729,3318,3557,4897,2686,3175,2479,110,3097,3047,1615,1680,3882,1811,71,613,2243,382,1000,1685,2946,2498,2367,4455,2015,1829,186,4583,4961,1156,2717,2488,3894,3920,3796,2524,1018,1925,4581,4513,4572,571,2480,1862,4443,4945,1610,2441,3664,4066,371,4528,1511,1952,3293,2829,167,258,2587,3246,98,2317,60,175,4832,4577,2680,2134,3154,2665,3168,384,4911,1057,2831,2443,2697,1296,2694,2668,45,3129,3604,3745,4276,490,3731,1543,3443,896,1697,745,69,3511,4148,2328,2628,4062,1673,1041,3786,4906,2604,1902,1721,1162,1508,974,95,3950,1423,2492,1546,1870,2925,3509,1884,2227,1184,1098,1932,2550,3268,4365,2110,1904,4286,2363,2116,711,1387,4333,4574,2767,3379,3612,4109,4573,276,200,1083,638,3349,2280,3541,803,1066,4108,2378,1551,1136,1515,3908,1664,2390,2538,2565,3087,122,4904,4474,3700,3423,2843,2998,1213,2992,4054,2960,3534,2206,4389,3461,611,1215,2730,3300,3828,875,2991,2419,4253,2044,2718,2106,2951,3336,3507,728,4359,3039,4725,2690,1804,825,3516,1574,2768,3687,1598,2376,451,4322,1005,3872,3565,2770,3197,4916,2959,2603,2812,3793,2133,3829,1908,181,1139,539,712,4142,2171,3334,4689,3630,3712,635,4803,353,1216,2589,1433,3280,594,3459,3034,3320,4603,756,2583,2360,1228,149,4440,1518,4335,735,4146,4640,3178,3288,4646,3363,2914,2207,4425,4744,3635,1268,3657,3146,3627,1607,2184,225,3106,4559,4766,4753,2260,3710,2567,3096,1958,845,2070,335,2698,2370,4851,1972,2966,4981,2438,2755,2656,4843,3454,489,424,3813,2283,844,3341,3570,2242,4246,2641,1349,4383,527,4227,2521,463,4249,246,2467,1256,3213,3250,83,4453,4325,4309,4100,3033,4064,2830,1333,700,1045,4181,2850,3889,785,4831,4049,946,3297,2380,4239,3406,954,1362,2701,1569,3375,518,2391,3317,4419,15,3088,2929,4631,2229,325,3145,4547,1309,3875,4561,3973,1297,4553,3181,3139,3067,1353,2692,2931,2474,1536,4591,3481,1478,4031,4299,494,2839,2652,3783,2486,2729,36,1174,4288,3432,3623,3254,344,3120,1464,1039,277,3960,2978,4866,3286,2059,2722,4568,2298,3476,3510,4847,2036,3158,217,252,793,3870,4480,804,3508,3695,1882,935,2338,2346,4999,4133,2356,3172,4204,1292,4151,3966,3323,4693,1992,2654,1074,4072,506,1149,2458,4225,1012,1684,2402,2917,1318,85,3975,1914,4698,3814,3270,1883,2409,2858,761,1670,937,4645,3027,1126,3261,4604,1553,133,178,3431,1264,4833,3451,3056,2005,2095,4121,3143,415,2855,3403,4463,2607,2064,1451,1167,1752,2584,4196,2933,1358,2002,3049,1120,3762,4793,3767,3617,2445,2627,3383,4557,606,522,185,4076,1672,3946,2484,1196,2238,3182,43,4838,88,1231,4946,3799,3500,2778,2597,2383,4565,1419,1476,4212,4777,1118,183,2033,180,1056,3488,3180,573,3086,3126,351,4948,859,2303,3171,2324,2558,3479,4915,4268,4926,1576,1363,4923,4330,2523,4444,650,4922,1652,4515,2979,3649,4920,1261,2878,369,266,3473,3234,3885,216,1001,1410,3304,893,1872,4864,3804,2155,3116,4520,4527,886,3216,280,4818,3249,1567,3535,3841,3314,3492,1100,4289,27,4627,1146,970,3938,2004,1448,2761,2275,2107,3003,3503,4345,4883,3574,126,413,749,3971,4732,3650,795,3045,2962,4736,3487,2201,1613,420,2232,1797,3726,239,2825,3079,1589,4610,3109,4342,1217,179,3861,2848,4500,4318,986,453,1634,3747,2900,4989,1726,201,4395,4485,4476,2598,76,754,2312,926,4294,1699,3094,1431,2339,1438,3302,2512,596,2949,3791,2366,1695,944,1110,791,4237,987,610,1278,684,1375,1876,4015,495,2156,206,2517,2709,3699,40,1453,2965,3225,481,1887,4802,357,757,3104,3997,4390,4483,1946,3008,1234,3629,3929,2867,3581,1404,19,959,99,4041,1954,3243,2294,742,3736,4468,4687,2043,2592,888,948,4033,4731,2203,4871,631]"
"pid": 23352, "tid": 23352, "ts": 52833387842ns : GeneratedQueries: "queries" : 1024, "samples per query" : 1, "duration" : 10240000000
"pid": 23352, "tid": 23352, "ts": 78672308166ns : Log Contention Counters:
"pid": 23352, "tid": 23352, "ts": 78672308166ns : 0 : swap_request_slots_retry_count
"pid": 23352, "tid": 23352, "ts": 78672308166ns : 0 : swap_request_slots_retry_retry_count
"pid": 23352, "tid": 23352, "ts": 78672308166ns : 0 : swap_request_slots_retry_reencounter_count
"pid": 23352, "tid": 23352, "ts": 78672308166ns : 0 : start_reading_entries_retry_count
"pid": 23352, "tid": 23352, "ts": 78672308166ns : 0 : tls_total_log_cas_fail_count
"pid": 23352, "tid": 23352, "ts": 78672308166ns : 0 : tls_total_swap_buffers_slot_retry_count
"pid": 23352, "tid": 23352, "ts": 78672308877ns : POWER_BEGIN: "mode" : Accuracy, "time" : 02-07-2020 13:00:45.212
"pid": 23352, "tid": 23352, "ts": 78672308877ns : POWER_END: "mode" : Accuracy, "time" : 02-07-2020 13:01:11.044
"pid": 23352, "tid": 23352, "ts": 78690270058ns : Loading QSL : "set" : "[4938,3801,1880,4085,3991,4498,1094,1294,1877,708,788,1956,4792,3189,4311,835,1940,619,4837,2340,2237,1168,1895,1701,3853,1429,125,2710,849,1460,4101,2449,3987,2591,1792,3305,1759,4427,3917,3066,30,4448,1022,4233,4548,2355,2030,1055,4384,3491,3873,2687,2593,1189,1570,1102,3968,2981,3761,1081,3919,634,1019,858,766,4182,4567,2364,4522,1559,523,551,1369,733,3533,3806,2392,4934,4354,2108,1370,3924,3051,2667,1922,3790,2649,2397,968,623,2094,4019,4820,4136,4282,2939,32,279,1900,4988,1850,1939,3497,1232,625,3121,3545,364,3100,3135,4829,4625,637,4982,4764,2536,1185,2995,2342,3978,4720,3658,2269,2714,2600,1715,2651,3295,2970,1687,156,1528,4369,3016,1454,402,2973,3026,3370,2706,4834,4788,215,1821,1506,1163,4505,1805,3063,92,3409,1408,4597,1198,3890,3900,4794,2218,3520,4361,2738,4637,4943,2797,3465,3580,4797,1845,1078,2909,3458,499,3219,2852,2819,2416,4446,1769,2403,2505,1788,632,184,1130,2477,177,2276,4503,912,2060,3001,772,4451,799,3599,3867,3766,2844,4492,2817,769,2897,4578,4914,2288,2251,2889,1378,4460,2136,1738,1761,863,1602,2705,2985,3009,2827,2210,593,2663,2494,581,461,4643,3616,2740,3906,3913,2471,1389,979,355,2896,4514,1988,2907,1785,3550,3563,498,3631,1964,2510,3124,488,73,3281,414,517,302,3681,1053,4944,524,2167,2987,1765,4128,2487,560,312,4327,1981,407,4648,3343,1836,4113,4724,2548,1137,3292,1782,2481,4131,4958,2129,330,3633,3785,3815,2745,988,439,2511,2464,2756,3351,2568,383,2213,1828,4791,2579,3224,2025,2289,3680,2833,1203,1244,1436,3770,569,4908,1844,910,4502,4360,1279,2451,109,3643,441,2688,3628,4387,1542,119,1330,1754,4353,1477,1776,895,446,1281,4479,4069,2509,1290,4772,4120,604,331,1669,3715,1620,3615,2141,504,3843,3748,2372,1713,2014,995,867,352,4544,1069,2245,2599,4336,226,4697,3308,592,3321,2773,2225,4965,1834,4844,614,3420,4220,897,1534,4740,3663,1740,300,2737,3586,3985,3558,3144,1974,1399,3709,1903,3725,3694,3389,927,4438,1623,2984,3690,3735,3156,7,1405,4672,4541,487,425,4900,3277,4886,1469,3194,4470,3354,4537,365,3839,2081,3255,13,812,510,2300,4306,263,3397,4139,855,3579,4152,813,4310,4163,1484,3251,1276,4877,1982,841,641,4256,3691,991,2440,4987,2752,931,2732,3980,3387,4417,3582,3489,722,2017,3084,3651,3263,889,1517,4,442,861,1068,1366,2148,4854,3036,2135,4614,2054,2661,2635,3810,2315,3602,4184,870,1700,2130,4756,4405,3537,403,1717,2099,1496,2389,1170,222,1512,663,3418,3070,4876,4779,2552,3782,3956,1927,2569,989,4312,2501,3816,4852,4980,4093,3131,1253,249,3820,2428,1089,3742,1847,3338,1223,3758,4210,1361,3205,737,3345,2655,2856,2513,2969,1924,233,2221,4986,4972,3911,50,2093,373,4607,1743,528,207,3808,28,1388,1275,1177,4856,1186,879,1608,1335,4326,117,1071,716,4839,1079,417,3385,4954,1277,3827,137,2352,1526,500,3868,3256,4174,3752,4913,570,3531,529,2388,4701,121,1194,4903,1307,3103,4790,3955,4960,2336,818,1905,4377,3035,3376,2187,851,2313,621,4501,2616,4769,2192,195,2563,3771,231,4721,4467,250,4768,2204,4459,1242,3381,1492,3682,1791,2608,4016,483,4283,4782,901,3963,3554,3307,645,3042,1742,852,3247,143,3949,4599,2042,740,3209,209,1327,2802,3823,2062,4776,1967,543,4341,4692,1302,290,2071,4447,3724,4418,4785,3931,805,3546,464,48,1586,1117,4328,752,2766,2837,916,3073,3165,3903,2115,2542,472,4374,318,544,647,4081,56,779,4924,348,3258,3529,168,4624,374,2078,113,3460,3905,3734,642,4902,1084,2888,170,274,3611,1402,4598,558,112,2384,205,2098,1432,2634,3395,4307,678,3854,630,94,4130,2555,3692,1628,1325,3060,3015,2020,241,4238,4014,4452,3327,21,1674,4758,1328,881,3085,4001,2578,4596,3637,696,2432,307,2459,3676,100,4279,513,1777,4037,3754,2143,1784,1357,4424,2618,321,1180,3061,4083,4660,4996,3935,715,4872,1764,148,2515,2290,2955,2185,3982,3267,1205,1584,367,1696,4347,2101,3199,2362,2193,755,4551,1790,294,4231,744,2375,978,3901,293,1878,31,4848,3760,3140,1238,4819,4373,3390,1044,3666,640,1259,3927,3157,478,1446,2994,4118,1371,2864,3347,1341,404,1869,2860,949,1267,3433,2982,1807,597,3152,584,608,1236,3976,2971,1131,4925,3933,1308,2522,815,2040,4775,2239,3220,1605,3912,1920,1474,1075,1814,3093,493,2307,2202,1329,981,1288,721,467,310,3614,4297,3,82,2734,751,3641,2032,1621,3058,1802,738,3177,1984,3311,1642,3932,3410,1549,4039,4674,1503,97,1581,1439,3218,707,660,4002,1141,3350,4102,3775,1097,2026,4673,4930,1486,1770,559,3728,3427,3671,2161,4104,2814,1756,2273,2841,379,1449,508,1443,3896,4127,1401,75,3774,1840,153,2645,157,3519,2314,3453,4203,410,1573,2869,4907,4223,709,3141,512,2310,644,4486,1799,4694,399,4816,3738,1345,2519,2671,1507,1625,3730,1299,2753,3910,4812,777,794,4931,1963,3590,627,1990,1206,2049,2967,2950,3170,4025,3236,4187,4277,4158,2329,1316,4754,2240,2983,586,3210,3940,731,2259,2849,4171,4617,4706,2421,2048,3480,2727,2504,4357,3159,319,4781,1034,4206,686,3198,2796,1437,2214,4013,574,3653,72,260,2853,4088,1825,922,509,4963,4878,664,2943,969,3732,792,2826,2293,3468,4868,4953,2776,4875,4633,4634,3332,3478,3127,2254,2916,1082,4008,61,3130,1760,4079,227,2091,2142,3577,1303,578,1704,2782]"
"pid": 23352, "tid": 23352, "ts": 79089618175ns : GeneratedQueries: "queries" : 1024, "samples per query" : 1, "duration" : 10240000000
"pid": 23352, "tid": 23352, "ts": 104941432540ns : Log Contention Counters:
"pid": 23352, "tid": 23352, "ts": 104941432540ns : 0 : swap_request_slots_retry_count
"pid": 23352, "tid": 23352, "ts": 104941432540ns : 0 : swap_request_slots_retry_retry_count
"pid": 23352, "tid": 23352, "ts": 104941432540ns : 0 : swap_request_slots_retry_reencounter_count
"pid": 23352, "tid": 23352, "ts": 104941432540ns : 0 : start_reading_entries_retry_count
"pid": 23352, "tid": 23352, "ts": 104941432540ns : 0 : tls_total_log_cas_fail_count
"pid": 23352, "tid": 23352, "ts": 104941432540ns : 0 : tls_total_swap_buffers_slot_retry_count
"pid": 23352, "tid": 23352, "ts": 104941433195ns : POWER_BEGIN: "mode" : Accuracy, "time" : 02-07-2020 13:01:11.468
"pid": 23352, "tid": 23352, "ts": 104941433195ns : POWER_END: "mode" : Accuracy, "time" : 02-07-2020 13:01:37.316
"pid": 23352, "tid": 23352, "ts": 104961804643ns : Loading QSL : "set" : "[526,1367,1300,847,4125,1282,3942,1315,773,4077,3279,2968,4258,1678,2369,2596,3789,4232,1152,2902,1783,4569,796,4808,1998,54,4059,3099,2838,2636,4676,4366,1744,2836,923,681,2159,1470,3011,1861,2728,2175,1286,4055,4669,1391,4450,1676,4431,747,1233,316,1465,1455,3195,2223,4464,2868,2872,1135,3952,1930,1679,2774,1283,1629,4752,585,2222,55,624,1247,921,4795,2337,3860,2553,4183,4126,4690,2411,2468,873,1324,4352,3064,2373,961,563,2359,4841,309,416,3155,3105,3107,542,4901,4047,436,37,3057,106,652,2079,3803,4991,1398,1935,3355,1725,885,440,719,105,909,2035,1088,3624,1693,4950,4801,3743,4644,4058,2786,3373,3090,3232,4560,1787,1894,1899,2001,1947,4905,3569,9,4874,1529,3994,538,2248,3721,1527,4746,138,4051,4993,2444,2791,2516,1481,232,3778,3289,4469,4828,2748,983,4089,702,3688,4606,1017,1351,1763,390,2582,4509,1555,951,2413,3372,1270,693,1202,150,2735,1076,2483,741,2114,491,1262,4770,4508,3947,1024,24,4065,1332,1649,2840,4755,4840,997,1321,4800,398,322,1224,4416,220,689,4129,4592,3899,2226,3483,4684,1376,3914,3526,999,2158,267,4798,1058,4681,3004,2633,972,3233,2460,1531,1211,1394,3974,4228,35,1933,1200,4160,1235,4662,2401,2410,3515,1644,2977,2476,1793,3848,1859,4642,2169,802,3091,3495,1541,3364,2274,4397,4166,980,2801,2648,535,4074,1385,2435,811,3936,4091,2326,4827,2693,4666,2617,3419,1729,4117,3780,4070,4704,1516,2577,1323,376,2183,4710,4355,1648,1343,2961,4057,1513,1557,1029,695,2139,3394,940,197,3166,2021,1921,243,96,1552,2642,1171,3208,2576,2173,1142,1485,4199,600,1354,2713,4738,1400,2944,2016,3909,3990,1709,3764,4068,1122,4635,4285,837,3988,1891,3981,4155,3984,4236,3787,1768,86,2643,1013,2792,1604,1971,1560,3068,4962,4656,1070,4421,1710,4563,188,1348,4719,4185,2861,1031,4465,3212,2672,2799,1522,4177,256,2562,4378,1116,1614,4024,4605,214,3999,864,2742,162,2945,332,1827,3578,228,2131,3248,1128,3880,3729,3636,3595,3862,2348,2626,2503,1295,516,1201,261,4368,152,4538,666,3407,953,2365,1009,3603,333,34,887,4518,1677,3685,4243,172,4267,3137,3019,2743,3128,1638,1319,521,4112,4891,375,2330,4375,3967,1140,3944,4525,5,345,4975,933,1280,941,960,1025,3598,1544,1778,566,2011,3677,4198,327,3076,334,1463,4303,259,554,4029,3262,2128,2906,628,992,4517,2615,3368,2560,89,4477,4034,4362,4245,768,1426,4609,4254,2941,534,3941,3071,3384,878,1339,4316,2357,1006,899,77,4356,2787,4266,2145,1849,4825,2069,3594,1245,856,3059,1868,3192,4994,3102,4887,2090,2928,3377,580,2534,1533,2345,4884,2031,1065,3893,4699,3972,4489,473,1851,561,1352,1148,616,4622,676,4589,255,4970,161,4651,354,671,3521,1568,4075,3536,4218,1514,3242,4671,202,3855,556,2490,3031,2024,1535,3374,2399,4257,1306,3163,1227,3072,3330,4202,190,3998,4106,4910,1252,4201,1719,2999,1043,976,17,2147,3865,1320,4098,2405,4298,3272,2771,3679,1854,1736,2252,3193,1651,1240,2892,3918,4172,378,248,4372,1524,2424,1795,1479,4882,4097,714,956,4700,1800,22,3153,2500,2594,1027,1155,1824,4540,577,2882,1103,3149,165,3983,781,460,734,4682,2287,3842,1600,2870,236,4084,3652,3413,1461,4787,982,1780,3501,2744,4420,4623,389,1757,470,2547,4010,193,3821,240,1968,1109,4861,2669,2073,787,4364,1380,2265,902,3101,4287,4422,515,680,3075,4813,1578,1749,3606,3568,1192,2256,4456,3610,3353,3834,603,1931,1086,3196,418,3037,3335,1160,4211,3329,3883,3788,4110,468,2784,2920,2119,4586,2272,116,4044,828,748,1596,174,4566,4414,4264,653,1355,2434,2,1,1539,4562,1886,4532,3405,883,4745,4654,2890,934,4530,2997,4595,3622,4086,4534,400,3922,2666,4023,4830,1665,2057,701,2807,3765,3202,1937,4273,1796,3485,3253,368,4542,176,3844,3055,1737,2047,1556,4715,3456,1591,253,4154,2585,1284,3640,164,872,3711,629,1730,1364,278,1229,4396,3597,160,147,3441,2894,2331,3032,1239,2541,102,4082,4951,1225,4078,4156,3756,2731,264,4632,1871,4743,4269,1692,3620,3275,3265,549,3723,3959,2349,3524,3282,1909,2469,4338,771,2018,1304,1497,3228,932,1411,649,950,3122,1248,3134,758,4846,2041,1099,3078,3544,1487,187,1668,3499,1582,2197,4703,2262,1298,2189,2381,894,3921,1826,753,3587,191,210,1050,4750,3142,2689,196,903,2144,3656,2763,2010,1666,2996,4173,2532,4006,786,532,3110,437,1656,690,4933,965,3112,2980,4582,2170,64,1989,3714,2132,192,3186,3356,2733,3632,295,4050,1412,2211,865,381,3548,1456,4967,2100,1087,891,3665,4858,1108,360,1413,2699,1218,1979,2854,3551,2335,1421,435,3892,1360,2400,2993,3887,4433,1417,2622,2353,4675,503,1916,683,1209,4123,3203,4301,3148,41,4038,4116,58,2311,2052]"
"pid": 23352, "tid": 23352, "ts": 105317936774ns : GeneratedQueries: "queries" : 904, "samples per query" : 1, "duration" : 9040000000
"pid": 23352, "tid": 23352, "ts": 128214551579ns : Log Contention Counters:
"pid": 23352, "tid": 23352, "ts": 128214551579ns : 0 : swap_request_slots_retry_count
"pid": 23352, "tid": 23352, "ts": 128214551579ns : 0 : swap_request_slots_retry_retry_count
"pid": 23352, "tid": 23352, "ts": 128214551579ns : 0 : swap_request_slots_retry_reencounter_count
"pid": 23352, "tid": 23352, "ts": 128214551579ns : 0 : start_reading_entries_retry_count
"pid": 23352, "tid": 23352, "ts": 128214551579ns : 0 : tls_total_log_cas_fail_count
"pid": 23352, "tid": 23352, "ts": 128214551579ns : 0 : tls_total_swap_buffers_slot_retry_count
"pid": 23352, "tid": 23352, "ts": 128214552236ns : POWER_BEGIN: "mode" : Accuracy, "time" : 02-07-2020 13:01:37.696
"pid": 23352, "tid": 23352, "ts": 128214552236ns : POWER_END: "mode" : Accuracy, "time" : 02-07-2020 13:02:00.584



I'm closing this as the reference timing logging is not the source of truth. Please use the logs from loadgen as the source of truth. If needed, please suggest edits to the logging spec doc (please see the recent WG slides here https://docs.google.com/presentation/d/1VnjrbhHUJstkXC2RO1zqHXYyhE2YforpPbdyrxbMKo4/edit. You have to be part of the WG to access the slides)