Repository: mlcommons/inference
Issue: assertion failed in loadgen/logging.cc
Root cause: Resource Mishandling

Today I encountered this assertion failure several times. Unfortunately I don't have reliable way to reproduce it. Sometimes it keeps failing and sometimes it keeps working.

How I encountered this issue:
 1. I build mlperf myself. android-v2 branch.
 2. Run the benchmark as usual.
 3. If the assertion failed, it usually failed around the starting time of OD test.

```
--------- beginning of crash
05-10 13:52:05.212 21895 21978 F libc    : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 21978 (binder:21895_5), pid 21895 (lperf.inference)
05-10 13:52:05.305 22129 22129 I crash_dump64: obtaining output fd from tombstoned, type: kDebuggerdTombstoneProto
05-10 13:52:05.306 22129 22129 I crash_dump64: performing dump of process 21895 (target tid = 21978)
05-10 13:52:05.316 21895 22068 F libc    : external/org_mlperf_inference/loadgen/logging.cc:786: mlperf::logging::TlsLogger *mlperf::logging::Logger::GetTlsLoggerThatRequestedSwap(size_t, size_t): assertion "success" failed
05-10 13:52:05.484 22129 22129 W crash_dump64: type=1400 audit(0.0:1332): avc: denied { search } for name="0" dev="tmpfs" ino=13 scontext=u:r:crash_dump:s0:c1,c257,c512,c768 tcontext=u:object_r:mnt_user_file:s0 tclass=dir permissive=0 app=org.mlperf.inference
05-10 13:52:05.484 22129 22129 W crash_dump64: type=1400 audit(0.0:1333): avc: denied { search } for name="0" dev="tmpfs" ino=13 scontext=u:r:crash_dump:s0:c1,c257,c512,c768 tcontext=u:object_r:mnt_user_file:s0 tclass=dir permissive=0 app=org.mlperf.inference
05-10 13:52:05.541 22129 22129 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
05-10 13:52:05.541 22129 22129 F DEBUG   : Build fingerprint: 'google/***/8551174:userdebug/dev-keys'
05-10 13:52:05.541 22129 22129 F DEBUG   : Revision: 'EVT1.1'
05-10 13:52:05.541 22129 22129 F DEBUG   : ABI: 'arm64'
05-10 13:52:05.541 22129 22129 F DEBUG   : Timestamp: 2022-05-10 13:52:05.313433948+0800
05-10 13:52:05.541 22129 22129 F DEBUG   : Process uptime: 71s
05-10 13:52:05.541 22129 22129 F DEBUG   : Cmdline: org.mlperf.inference
05-10 13:52:05.541 22129 22129 F DEBUG   : pid: 21895, tid: 21978, name: binder:21895_5  >>> org.mlperf.inference <<<
05-10 13:52:05.541 22129 22129 F DEBUG   : uid: 10257
05-10 13:52:05.541 22129 22129 F DEBUG   : tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
05-10 13:52:05.541 22129 22129 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
05-10 13:52:05.541 22129 22129 F DEBUG   : Abort message: 'terminating'
05-10 13:52:05.541 22129 22129 F DEBUG   :     x0  0000000000000000  x1  00000000000055da  x2  0000000000000006  x3  0000007889655a80
05-10 13:52:05.541 22129 22129 F DEBUG   :     x4  fefefefeff666d68  x5  fefefefeff666d68  x6  fefefefeff666d68  x7  7f7f7f7f7f7f7f7f
05-10 13:52:05.541 22129 22129 F DEBUG   :     x8  00000000000000f0  x9  0000007b4797f9e0  x10 0000000000000001  x11 0000007b479c0e40
05-10 13:52:05.541 22129 22129 F DEBUG   :     x12 000000003b9ac9ff  x13 000000007fffffff  x14 00000000003a9906  x15 0000004cda4574aa
05-10 13:52:05.541 22129 22129 F DEBUG   :     x16 0000007b47a2dd58  x17 0000007b47a09980  x18 0000007886d26000  x19 0000000000005587
05-10 13:52:05.541 22129 22129 F DEBUG   :     x20 00000000000055da  x21 00000000ffffffff  x22 ffffff80ffffffc8  x23 0000007889655cd0
05-10 13:52:05.541 22129 22129 F DEBUG   :     x24 0000007889655bb0  x25 0000007889655bf0  x26 000000788965827c  x27 0000007889658248
05-10 13:52:05.541 22129 22129 F DEBUG   :     x28 0000007889658130  x29 0000007889655b00
05-10 13:52:05.541 22129 22129 F DEBUG   :     lr  0000007b479b10c8  sp  0000007889655a60  pc  0000007b479b10f4  pst 0000000000001000
05-10 13:52:05.541 22129 22129 F DEBUG   : backtrace:
05-10 13:52:05.541 22129 22129 F DEBUG   :       #00 pc 00000000000530f4  /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: f4bbb65957cf1dbe210e337f4656bdbc)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #01 pc 0000000001554908  /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/lib/arm64/libmlperf_jni.so (abort_message+236)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #02 pc 0000000001554938  /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/lib/arm64/libmlperf_jni.so (demangling_terminate_handler()+44)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #03 pc 0000000001551cf4  /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/lib/arm64/libmlperf_jni.so (std::__terminate(void (*)())+12)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #04 pc 0000000001551cb4  /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/lib/arm64/libmlperf_jni.so (std::terminate()+52)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #05 pc 0000000000348a7c  /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/lib/arm64/libmlperf_jni.so (std::__ndk1::thread::operator=(std::__ndk1::thread&&)+52)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #06 pc 000000000032c668  /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/lib/arm64/libmlperf_jni.so (mlperf::logging::Logger::StartIOThread()+144)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #07 pc 00000000001dfa44  /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/lib/arm64/libmlperf_jni.so (mlperf::StartTest(mlperf::SystemUnderTest*, mlperf::QuerySampleLibrary*, mlperf::TestSettings const&, mlperf::LogSettings const&)+60)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #08 pc 00000000001d56b0  /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/lib/arm64/libmlperf_jni.so (mlperf::mobile::MlperfDriver::RunMLPerfTest(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, int, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)+324)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #09 pc 00000000001022e4  /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/lib/arm64/libmlperf_jni.so (Java_org_mlperf_inference_MLPerfDriverWrapper_nativeRun+168)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #10 pc 0000000000463954  /apex/com.android.art/lib64/libart.so (art_quick_generic_jni_trampoline+148) (BuildId: 9a4466481953960338de1cb9981b37e2)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #11 pc 000000000020b02c  /apex/com.android.art/lib64/libart.so (nterp_helper+7468) (BuildId: 9a4466481953960338de1cb9981b37e2)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #12 pc 000000000026cd4e  [anon:dalvik-classes.dex extracted in memory from /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/base.apk] (org.mlperf.inference.MLPerfDriverWrapper.runMLPerf+14)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #13 pc 000000000020a254  /apex/com.android.art/lib64/libart.so (nterp_helper+3924) (BuildId: 9a4466481953960338de1cb9981b37e2)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #14 pc 0000000000270108  [anon:dalvik-classes.dex extracted in memory from /data/app/~~lxfys1YZN1s_azZzrWSFMg==/org.mlperf.inference-aY9cgzBOPPK2gG7X7MAoLA==/base.apk] (org.mlperf.inference.RunMLPerfWorker.handleMessage+828)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #15 pc 000000000020b074  /apex/com.android.art/lib64/libart.so (nterp_helper+7540) (BuildId: 9a4466481953960338de1cb9981b37e2)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #16 pc 000000000047e798  /system/framework/framework.jar (android.os.Handler.dispatchMessage+24)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #17 pc 000000000020a254  /apex/com.android.art/lib64/libart.so (nterp_helper+3924) (BuildId: 9a4466481953960338de1cb9981b37e2)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #18 pc 00000000004a0d8e  /system/framework/framework.jar (android.os.Looper.loopOnce+346)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #19 pc 0000000000209334  /apex/com.android.art/lib64/libart.so (nterp_helper+52) (BuildId: 9a4466481953960338de1cb9981b37e2)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #20 pc 00000000004a142e  /system/framework/framework.jar (android.os.Looper.loop+162)
05-10 13:52:05.541 22129 22129 F DEBUG   :       #21 pc 0000000000209334  /apex/com.android.art/lib64/libart.so (nterp_helper+52) (BuildId: 9a4466481953960338de1cb9981b37e2)
```