Repository: mlcommons/inference
Issue: loadgen with gnmt on v0.5 seg-faults up on completion
Root cause: Resource Mishandling

When running gnmt with loadgen (loadgen_gnmt.py) it seg-faults in the logger up on finishing the run.
For e.g.
$ python loadgen_gnmt.py --mode Performance --scenario SingleStream --verbose

Fix that worked for me was to add a single line, runner.finish() to loadgen_gnmt.py. Pasting the diff below. 

```
diff --git a/v0.5/translation/gnmt/tensorflow/loadgen_gnmt.py b/v0.5/translation/gnmt/tensorflow/loadgen_gnmt.py
index 8a3410e..fb43469 100755
--- a/v0.5/translation/gnmt/tensorflow/loadgen_gnmt.py
+++ b/v0.5/translation/gnmt/tensorflow/loadgen_gnmt.py
@@ -496,6 +496,7 @@ if __name__ == "__main__":
         total_queries, perf_queries, runner.load_samples_to_ram, runner.unload_samples_from_ram)

     mlperf_loadgen.StartTest(sut, qsl, settings)
+    runner.finish()
     mlperf_loadgen.DestroyQSL(qsl)
     mlperf_loadgen.DestroySUT(sut)
```

In case anybody else tried and had no idea about the seg-fault.

@minus-one issue a PR perhaps and assign to @christ1ne 

@minus-one: Do you have a stack trace of the segfault? Just want to make sure we are addressing the root-cause and that there isn't a bug in the loadgen. Thanks!

This was the original stacktrace that I got via gdb. 

```
translate 99 (QID 95052320): Sentence ID 58
Average latency (ms) per query:
693.60307677
Median latency (ms):
601.6180425
90 percentile latency (ms):
1139.0376233000004
[Thread 0x7ffe46ffd700 (LWP 53526) exited]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
__gnu_cxx::new_allocator<std::function<void (mlperf::AsyncLog&)> >::construct<std::function<void (mlperf::AsyncLog&)>, std::function<void (mlperf::AsyncLog&)> >(std::function<void (mlperf::AsyncLog&)>*, std::function<void (mlperf::AsyncLog&)>&&) (this=0xa916a0, __p=0x6100000070) at /usr/include/c++/7/ext/new_allocator.h:136
136             { ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
(gdb) bt
#0  __gnu_cxx::new_allocator<std::function<void (mlperf::AsyncLog&)> >::construct<std::function<void (mlperf::AsyncLog&)>, std::function<void (mlperf::AsyncLog&)> >(std::function<void (mlperf::AsyncLog&)>*, std::function<void (mlperf::AsyncLog&)>&&) (this=0xa916a0, __p=0x6100000070) at /usr/include/c++/7/ext/new_allocator.h:136
#1  std::allocator_traits<std::allocator<std::function<void (mlperf::AsyncLog&)> > >::construct<std::function<void (mlperf::AsyncLog&)>, std::function<void (mlperf::AsyncLog&)> >(std::allocator<std::function<void (mlperf::AsyncLog&)> >&, std::function<void (mlperf::AsyncLog&)>*, std::function<void (mlperf::AsyncLog&)>&&) (__a=..., __p=0x6100000070) at /usr/include/c++/7/bits/alloc_traits.h:475
#2  std::vector<std::function<void (mlperf::AsyncLog&)>, std::allocator<std::function<void (mlperf::AsyncLog&)> > >::emplace_back<std::function<void (mlperf::AsyncLog&)> >(std::function<void (mlperf::AsyncLog&)>&&) (this=0xa916a0)
    at /usr/include/c++/7/bits/vector.tcc:100
#3  mlperf::TlsLogger::Log(std::function<void (mlperf::AsyncLog&)>&&) (this=0xa91380, entry=...) at logging.cc:521
#4  0x00007ffff5e0eeed in mlperf::Logger::UnRegisterTlsLogger (this=<optimized out>, tls_logger=std::unique_ptr<mlperf::TlsLogger> = {...}) at logging.cc:227
#5  0x00007ffff5e0f07f in mlperf::TlsLoggerWrapper::~TlsLoggerWrapper (this=0x4b36300, __in_chrg=<optimized out>) at logging.cc:598
#6  std::default_delete<mlperf::TlsLoggerWrapper>::operator() (this=<optimized out>, __ptr=0x4b36300) at /usr/include/c++/7/bits/unique_ptr.h:78
#7  std::unique_ptr<mlperf::TlsLoggerWrapper, std::default_delete<mlperf::TlsLoggerWrapper> >::reset (__p=0x4b36300, this=<optimized out>) at /usr/include/c++/7/bits/unique_ptr.h:376
#8  mlperf::<lambda()>::operator() (__closure=<optimized out>) at logging.cc:609
#9  std::_Function_handler<void(), mlperf::InitializeMyTlsLoggerWrapper()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#10 0x00007ffff5e0aeab in std::function<void ()>::operator()() const (this=0x7ffe54000d48) at /usr/include/c++/7/bits/std_function.h:706
#11 mlperf::TlsLogger::ForcedDetatchFromThread (this=0x7ffe54000ca0) at logging.cc:88
#12 mlperf::Logger::~Logger (this=0x7ffff6081800 <mlperf::GlobalLogger()::g_logger>, __in_chrg=<optimized out>) at logging.cc:166
#13 0x00007ffff7a27041 in __run_exit_handlers (status=0, listp=0x7ffff7dcf718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#14 0x00007ffff7a2713a in __GI_exit (status=<optimized out>) at exit.c:139
#15 0x00007ffff7a05b9e in __libc_start_main (main=0x4b4bd0 <main>, argc=7, argv=0x7fffffffe148, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe138) at ../csu/libc-start.c:344
#16 0x00000000005bdf6a in _start () at ../Python/pystate.c:543
(gdb)

```

I was digging through the whole code, but could never find the issue in loadgen. 😥 
Coming back to the python code and on comparing it with the classification example, noticed that this line was missing. 

@minus-one: Thanks for the stack trace! I think the proposed fix makes sense. Can you issue a PR?

It looks like the runner is set up as a daemon thread, which means its destruction is not sequentially ordered at all with the destruction of the main thread and static storage duration variables. It's likely that the daemon thread is destroying its TlsLogger at the same time that it is being forcefully detached by the main thread's at_exit hooks.

Calling runner.finish() ensures the daemon thread is joined before the program starts the exit process.

Alternatively, we may be able to avoid this in the loadgen as well if InitializeMyTlsLoggerWrapper were to acquire a global lock in 1) the forced_detatch lambda and 2) the destructor of tls_logger_wrapper via some helper type. @minus-one : Since you can repro, could you also take a crack at that solution if you have the cycles?


I was taking a poke at it. I am trying to understand the issue.
Is there a race on the TlsLoggerWrapper destructor itself or between TlsLoggerWrapper destructor and the TlsLogger destructor ? 
That is, the forced_detatch lambda calls TlsLoggerWrapper destructor (via reset()) while already the TlsLogger has been destroyed? In which case, shouldn't the lock be between TlsLoggerWrapper destructor and the TlsLogger destructor?
Or are there two calls on the TlsLoggerWrapper destructor itself, one of which is via the forced_detatch lambda?
I am just trying to understand who should own the mutex and where must it be acquired. 

I am new to the code and have little idea. If you think it is easier for someone else to fix than explain all these, please let me know. 

> Or are there two calls on the TlsLoggerWrapper destructor itself, one of which is via the forced_detatch lambda?

I think this is the case that would cause the error you are seeing, but I'm not 100% sure.

> I am new to the code and have little idea. If you think it is easier for someone else to fix than explain all these, please let me know.

It sounds like you have a pretty good grasp of the issue. I'm swamped atm, so would appreciate any help I can get. =)

can I just add the following line? It seems to make the error go away @briandersn @minus-one @nvmbreughe  
runner.finish()

Good catch. Yes, let's add that.

Agree with adding that line.

I was hoping we could also fix the crash in the loadgen too, in case other submitters use daemon threads too. But at this point, let's just fix the reference.

We can revisit this if submitters run into this issue. The crash is after the test completes, so it shouldn't block submissions.

just committed to master


@minus-one : Can we close this issue?

Feel free to issue a PR for the loadgen fix we talked about if you already have something, otherwise we can kick that can down the road. Thanks!

Yes. I got some code written as per your suggestion, but that does not seem to be fixing the bug. I will issue a PR if I think I have got a working fix.