Repository: mlcommons/inference
Issue: loadgen segfaults in accuracy mode
Root cause: Resource Mishandling

When LoadGen calls IssueQuery(), the SUT returns asynchronously, and then when the SUT calls QuerySampleComplete(), the asynchronous log in LoadGen tries to access some data that has been freed by LoadGen.


Hi Dilip. Can you provide steps for repro or a stack trace?

I don't see this problem with the latest reference code commit 3f23f7c. What's the steps for reproduce this issue? @DilipSequeira 

Was talking to some of the NVIDIA folks over email. Going to copy some of our discussion to this issue:

Quoting Vikram from NVIDIA:

"""
Currently, I am able to successfully run the single stream harness with LoadGen in PerformanceOnly test mode. However, when I attempt to run in AccuracyOnly mode, we observe multiple segfaults. Each segfault is identical, and here is what the stack trace from Valgrind looks like for one such segfault: 

****** Begin stack trace ******

==4012== Thread 7:
==4012== Invalid read of size 8
==4012==    at 0x48C6FE: std::_Function_handler<void (mlperf::AsyncLog&), mlperf::ResponseDelegateDetailed<(mlperf::TestScenario)0, (mlperf::TestMode)1>::SampleComplete(mlperf::SampleMetadata*, mlperf::Qu
erySampleResponse*, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(mlperf::AsyncLog&)#1}>::_M_invoke(std::_Any_data const&, mlperf::AsyncLog&) (in /work/build/bin/harness_single)
==4012==    by 0x494CA4: mlperf::Logger::IOThread() (in /work/build/bin/harness_single)
==4012==    by 0x127E6C7F: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==4012==    by 0x1210C6B9: start_thread (pthread_create.c:333)
==4012==    by 0x130D641C: clone (clone.S:109)
==4012==  Address 0x3edb3d70 is 16 bytes inside a block of size 24 free'd
==4012==    at 0x4C2F24B: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4012==    by 0x47DFE6: std::vector<mlperf::QueryMetadata, std::allocator<mlperf::QueryMetadata> >::~vector() (in /work/build/bin/harness_single)
==4012==    by 0x4833E9: mlperf::PerformanceResult mlperf::IssueQueries<(mlperf::TestScenario)0, (mlperf::TestMode)1>(mlperf::SystemUnderTest*, mlperf::TestSettingsInternal const&, mlperf::LoadableSampleSet const&, mlperf::SequenceGen*) (in /work/build/bin/harness_single)
==4012==    by 0x4835FA: void mlperf::RunAccuracyMode<(mlperf::TestScenario)0>(mlperf::SystemUnderTest*, mlperf::QuerySampleLibrary*, mlperf::TestSettingsInternal const&, std::vector<mlperf::LoadableSampleSet, std::allocator<mlperf::LoadableSampleSet> > const&, mlperf::SequenceGen*) (in /work/build/bin/harness_single)
==4012==    by 0x47DE0D: mlperf::StartTest(mlperf::SystemUnderTest*, mlperf::QuerySampleLibrary*, mlperf::TestSettings const&, mlperf::LogSettings const&) (in /work/build/bin/harness_single)
==4012==    by 0x40BC5C: doInference() (in /work/build/bin/harness_single) (NOTE: This is an internal NVIDIA function, so for now I have removed its parameters)
==4012==    by 0x409367: main (in /work/build/bin/harness_single)
==4012==  Block was alloc'd at
==4012==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4012==    by 0x47A87F: mlperf::QueryMetadata::QueryMetadata(std::vector<unsigned long, std::allocator<unsigned long> > const&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, mlperf::ResponseDelegate*, mlperf::SequenceGen*) (in /work/build/bin/harness_single)
==4012==    by 0x480B61: void std::vector<mlperf::QueryMetadata, std::allocator<mlperf::QueryMetadata> >::_M_emplace_back_aux<std::vector<unsigned long, std::allocator<unsigned long> >&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >&, mlperf::ResponseDelegate*&, mlperf::SequenceGen*&>(std::vector<unsigned long, std::allocator<unsigned long> >&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >&, mlperf::ResponseDelegate*&, mlperf::SequenceGen*&) (in /work/build/bin/harness_single)
==4012==    by 0x482E0A: std::vector<mlperf::QueryMetadata, std::allocator<mlperf::QueryMetadata> > mlperf::GenerateQueries<(mlperf::TestScenario)0, (mlperf::TestMode)1>(mlperf::TestSettingsInternal const&, mlperf::LoadableSampleSet const&, mlperf::SequenceGen*, mlperf::ResponseDelegate*) (in /work/build/bin/harness_single)
==4012==    by 0x482FBC: mlperf::PerformanceResult mlperf::IssueQueries<(mlperf::TestScenario)0, (mlperf::TestMode)1>(mlperf::SystemUnderTest*, mlperf::TestSettingsInternal const&, mlperf::LoadableSampleS
et const&, mlperf::SequenceGen*) (in /work/build/bin/harness_single)
==4012==    by 0x4835FA: void mlperf::RunAccuracyMode<(mlperf::TestScenario)0>(mlperf::SystemUnderTest*, mlperf::QuerySampleLibrary*, mlperf::TestSettingsInternal const&, std::vector<mlperf::LoadableSampleSet, std::allocator<mlperf::LoadableSampleSet> > const&, mlperf::SequenceGen*) (in /work/build/bin/harness_single)
==4012==    by 0x47DE0D: mlperf::StartTest(mlperf::SystemUnderTest*, mlperf::QuerySampleLibrary*, mlperf::TestSettings const&, mlperf::LogSettings const&) (in /work/build/bin/harness_single)
==4012==    by 0x40BC5C: doInference() (in /work/build/bin/harness_single) (NOTE: This is an internal NVIDIA function, so for now I have removed its parameters)
==4012==    by 0x409367: main (in /work/build/bin/harness_single)
==4012== 

****** End stack trace ******

This stack trace shows that we are segfaulting because memory that was allocated in `mlperf::IssueQueries()` (at line 545-546 in loadgen.cc) but already freed once the `IssueQueries()` function exited is being invalidly accessed in `mlperf::SampleComplete::{lambda(mlperf::AsyncLog&)#1}`. Since I am able to run PerformanceOnly mode without an issue, I suspect that the problem comes with the data access in the `if` blocks that begin at lines 200 and 224 of loadgen.cc.

I am currently using code from the latest version of the MLPerf/Inference repository (commit hash 27b23c22e7adf977f6889853d400d03204f6203f). I am not overriding any of the test settings for the single stream scenario, and am using a value for the `single_stream_expected_latency` field that matches what our internal testing produces for the device being used.
"""



**Quoting myself:**
"""
The call to GetLatenciesBlocking in IssueQueries is supposed to block until all expected calls to QuerySamplesComplete have been received (and logged). That should also prevent the QueryMetadata from being freed.

It looks like GetLatenciesBlocking is returning earlier than it should, implying that QuerySamplesComplete is being called more often than the loadgen expects.
"""

**After I put together PR #238 to help debug this issue. Vikram reported the following:**

"""
...there is a new error that I am getting of the following form:

"ERROR : Encountered incomplete samples at the end of a series of queries."count" : 34"

This error message shows up after every "GeneratedQueries" entry in the mlperf_log_detailed.txt files. The value of "count" at the end of each error message increments for every subsequent error. The Log Contention Counters all seem to report counts of value 0. 

A few questions: 
(1) What causes the above error message to print?

(2) I should clarify - as it stands now, that when we run LoadGen in AccuracyOnly testing mode, the program no longer crashes and it does run to completion. However, when run with Valgrind, invalid memory accesses are reported (as per the earlier stack trace). How come these memory accesses are silently ignored?

(3) Once a test is run in AccuracyOnly mode, do we need to run a separate post-processing script to actually measure the numerical value of the accuracy that was achieved? Does LoadGen do this for us? 
"""

which benchmark mode? or all of them? which backend?

**> which benchmark mode? or all of them? which backend?**

I think this is a custom setup.

**> (1) What causes the above error message to print?**

It'll print if the loadgen finishes the test, but detects that the loadgen never recorded a latency for some of the samples.

It's surprising that the test ends with that error message without hanging first.

If you compile the loadgen in debug mode, do any of the asserts hit?

**(2) I should clarify - as it stands now, that when we run LoadGen in AccuracyOnly testing mode, the program no longer crashes and it does run to completion. However, when run with Valgrind, invalid memory accesses are reported (as per the earlier stack trace). How come these memory accesses are silently ignored?**

I wonder if the allocator is attempting to recycle memory, so it doesn't necessarily result in a segfault even though the memory has been deallocated from the loadgen's perspective.

**(3) Once a test is run in AccuracyOnly mode, do we need to run a separate post-processing script to 
actually measure the numerical value of the accuracy that was achieved? Does LoadGen do this for us?**

You will have to run a script to do this. @nvmbreughe should be familiar with how to do this. The loadgen will not do this automatically, as it is not model-aware.

**"ERROR : Encountered incomplete samples at the end of a series of queries."count" : 34"**

I found a bug in the loadgen, which causes that error you see. It shouldn't be the cause of the use after free though.

 Can you try cherry-picking the latest version of https://github.com/mlperf/inference/pull/238 again? It fixes the bug, and I verified it runs locally for me in accuracy mode. (I had only tested the previous patch in performance mode.)

This is Vikram from NVIDIA. To respond to @christ1ne, I am running the test in AccuracyOnly mode using the ResNet50 benchmark and preprocessed Imagenet validation images. When you ask for which backend, are you referring to our QSL/SUT implementation? If so, yes we are using a custom implementation for those and not those provided in the demos.

@briandersn, after integrating your latest changes, I no longer get any errors reported in my detailed log. However, Valgrind still reports the invalid memory accesses (following the same stack trace as above). 

When I compile LoadGen in debug mode and then run the same test, no asserts are hit. To compile it in debug mode, I just added the `--args="is_debug=true"` flag to the `third_party/gn/gn gen out/Debug` command. That's all that is needed to compile in debug mode correct? If so, then without any reported errors or triggered asserts, how can I verify that AccuracyOnly mode is actually working for me (given that Valgrind reports invalid memory accesses)?


> That's all that is needed to compile in debug mode correct? 

Correct.

> If so, then without any reported errors or triggered asserts, how can I verify that AccuracyOnly mode is actually working for me (given that Valgrind reports invalid memory accesses)?

I don't trust the output either. =)
I'll try to see if I can reproduce with asan or valgrind on my end. Can you attach a detail log so I know which settings to run with?

Btw, can you attach the trace (zipped) you get when valgrind isn't attached? It might easy to see if something is getting logged unexpectedly. We can add more trace points depending on how the trace looks.




@v4njur, regarding your question (3): LoadGen logs the response. Once all requests are handled, a separate script can be launched to process the log file and report accuracy. Feel free to ping me offline and I can walk you through this in more detail.

@briandersn, As you requested, I have attached a zipped folder with the detailed log text file and the trace json file from a run when valgrind wasn't attached. Let me know if you need any other information for your analysis. 

[mlperf_logs.zip](https://github.com/mlperf/inference/files/3357067/mlperf_logs.zip)

@nvmbreughe, Thanks I will ping you offline about this next week. 


Thanks for the trace! That was super helpful. Your settings have 1 sample for the performance / loadable set size, which is a stress test for the use-after-free bug. =)

Does this fix the valgrind errors: https://github.com/mlperf/inference/pull/248

@briandersn Thank you for the patch! Yes, your changes in #248 fixed the Valgrind issues that I was having with AccuracyOnly mode :) I will keep an eye out for your changes to be merged into MLPerf Inference master. 

I just merged in #248. @v4njur it would be great if you are OK and happy to close.

@jveejay @briandersn Although cherry picking Brian's changes from #248 worked okay, now when I use the latest version of LoadGen (commit hash 4185c4fe9f737e35bd30656c325725c1105e984b), I am running into different Valgrind issues in both PerformanceOnly and AccuracyOnly modes.

Just as before, without Valgrind, I am able to run to completion in the AccuracyOnly and PerformanceOnly test modes with no errors detected/reported in the logs. However, Valgrind still reports an issue regarding a conditional jump that depends on uninitialized value(s). Here is the stack trace below: 

`==6937== Thread 7:`
`==6937== Conditional jump or move depends on uninitialised value(s)`
`==6937==    at 0x12830CAE: std::ostreambuf_iterator<char, std::char_traits<char> > std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x12830EDC: std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::do_put(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x1283D3F9: std::ostream& std::ostream::_M_insert<long>(long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x49AB7D: void mlperf::AsyncLog::LogArgs<long>(std::ostream*, long const&) (logging.h:334)`
`==6937==    by 0x4E92F4: void mlperf::AsyncLog::LogDetail<long>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long const) (logging.h:201)`
`==6937==    by 0x4E607C: mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}::operator()(mlperf::AsyncLog&) const (test_settings_internal.cc:193)`
`==6937==    by 0x4E6CAA: void mlperf::LogDetail<mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}>(mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}&&)::{lambda(mlperf::AsyncLog&)#1}::operator()(mlperf::AsyncLog&) const (logging.h:527)`
`==6937==    by 0x4E7324: std::_Function_handler<void (mlperf::AsyncLog&), void mlperf::LogDetail<mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}>(mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}&&)::{lambda(mlperf::AsyncLog&)#1}>::_M_invoke(std::_Any_data const&, mlperf::AsyncLog&) (functional:1871)`
`==6937==    by 0x4DAC58: std::function<void (mlperf::AsyncLog&)>::operator()(mlperf::AsyncLog&) const (functional:2267)`
`==6937==    by 0x4D4424: mlperf::Logger::IOThread() (logging.cc:565)`
`==6937==    by 0x4E4C78: void std::_Mem_fn_base<void (mlperf::Logger::*)(), true>::operator()<, void>(mlperf::Logger*) const (in /work/build/bin/harness_single)`
`==6937==    by 0x4E4C0C: void std::_Bind_simple<std::_Mem_fn<void (mlperf::Logger::*)()> (mlperf::Logger*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (functional:1531)`
`==6937== `
`==6937== Use of uninitialised value of size 8`
`==6937==    at 0x1282FB13: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x12830CD9: std::ostreambuf_iterator<char, std::char_traits<char> > std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x12830EDC: std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::do_put(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x1283D3F9: std::ostream& std::ostream::_M_insert<long>(long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x49AB7D: void mlperf::AsyncLog::LogArgs<long>(std::ostream*, long const&) (logging.h:334)`
`==6937==    by 0x4E92F4: void mlperf::AsyncLog::LogDetail<long>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long const) (logging.h:201)`
`==6937==    by 0x4E607C: mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}::operator()(mlperf::AsyncLog&) const (test_settings_internal.cc:193)`
`==6937==    by 0x4E6CAA: void mlperf::LogDetail<mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}>(mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}&&)::{lambda(mlperf::AsyncLog&)#1}::operator()(mlperf::AsyncLog&) const (logging.h:527)`
`==6937==    by 0x4E7324: std::_Function_handler<void (mlperf::AsyncLog&), void mlperf::LogDetail<mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}>(mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}&&)::{lambda(mlperf::AsyncLog&)#1}>::_M_invoke(std::_Any_data const&, mlperf::AsyncLog&) (functional:1871)`
`==6937==    by 0x4DAC58: std::function<void (mlperf::AsyncLog&)>::operator()(mlperf::AsyncLog&) const (functional:2267)`
`==6937==    by 0x4D4424: mlperf::Logger::IOThread() (logging.cc:565)`
`==6937==    by 0x4E4C78: void std::_Mem_fn_base<void (mlperf::Logger::*)(), true>::operator()<, void>(mlperf::Logger*) const (in /work/build/bin/harness_single)`
`==6937== `
`==6937== Conditional jump or move depends on uninitialised value(s)`
`==6937==    at 0x1282FB1F: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x12830CD9: std::ostreambuf_iterator<char, std::char_traits<char> > std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x12830EDC: std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::do_put(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x1283D3F9: std::ostream& std::ostream::_M_insert<long>(long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x49AB7D: void mlperf::AsyncLog::LogArgs<long>(std::ostream*, long const&) (logging.h:334)`
`==6937==    by 0x4E92F4: void mlperf::AsyncLog::LogDetail<long>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long const) (logging.h:201)`
`==6937==    by 0x4E607C: mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}::operator()(mlperf::AsyncLog&) const (test_settings_internal.cc:193)`
`==6937==    by 0x4E6CAA: void mlperf::LogDetail<mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}>(mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}&&)::{lambda(mlperf::AsyncLog&)#1}::operator()(mlperf::AsyncLog&) const (logging.h:527)`
`==6937==    by 0x4E7324: std::_Function_handler<void (mlperf::AsyncLog&), void mlperf::LogDetail<mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}>(mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}&&)::{lambda(mlperf::AsyncLog&)#1}>::_M_invoke(std::_Any_data const&, mlperf::AsyncLog&) (functional:1871)`
`==6937==    by 0x4DAC58: std::function<void (mlperf::AsyncLog&)>::operator()(mlperf::AsyncLog&) const (functional:2267)`
`==6937==    by 0x4D4424: mlperf::Logger::IOThread() (logging.cc:565)`
`==6937==    by 0x4E4C78: void std::_Mem_fn_base<void (mlperf::Logger::*)(), true>::operator()<, void>(mlperf::Logger*) const (in /work/build/bin/harness_single)`
`==6937== `
`==6937== Conditional jump or move depends on uninitialised value(s)`
`==6937==    at 0x12830D0C: std::ostreambuf_iterator<char, std::char_traits<char> > std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x12830EDC: std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::do_put(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x1283D3F9: std::ostream& std::ostream::_M_insert<long>(long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)`
`==6937==    by 0x49AB7D: void mlperf::AsyncLog::LogArgs<long>(std::ostream*, long const&) (logging.h:334)`
`==6937==    by 0x4E92F4: void mlperf::AsyncLog::LogDetail<long>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long const) (logging.h:201)`
`==6937==    by 0x4E607C: mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}::operator()(mlperf::AsyncLog&) const (test_settings_internal.cc:193)`
`==6937==    by 0x4E6CAA: void mlperf::LogDetail<mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}>(mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}&&)::{lambda(mlperf::AsyncLog&)#1}::operator()(mlperf::AsyncLog&) const (logging.h:527)`
`==6937==    by 0x4E7324: std::_Function_handler<void (mlperf::AsyncLog&), void mlperf::LogDetail<mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}>(mlperf::TestSettingsInternal::LogEffectiveSettings() const::{lambda(mlperf::AsyncLog&)#1}&&)::{lambda(mlperf::AsyncLog&)#1}>::_M_invoke(std::_Any_data const&, mlperf::AsyncLog&) (functional:1871)`
`==6937==    by 0x4DAC58: std::function<void (mlperf::AsyncLog&)>::operator()(mlperf::AsyncLog&) const (functional:2267)`
`==6937==    by 0x4D4424: mlperf::Logger::IOThread() (logging.cc:565)`
`==6937==    by 0x4E4C78: void std::_Mem_fn_base<void (mlperf::Logger::*)(), true>::operator()<, void>(mlperf::Logger*) const (in /work/build/bin/harness_single)`
`==6937==    by 0x4E4C0C: void std::_Bind_simple<std::_Mem_fn<void (mlperf::Logger::*)()> (mlperf::Logger*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (functional:1531)`
`==6937== `
`==6937== Thread 1:`
`==6937== Conditional jump or move depends on uninitialised value(s)`
`==6937==    at 0x4A26CF: mlperf::PerformanceResult mlperf::IssueQueries<(mlperf::TestScenario)0, (mlperf::TestMode)1>(mlperf::SystemUnderTest*, mlperf::TestSettingsInternal const&, mlperf::LoadableSampleSet const&, mlperf::SequenceGen*) (loadgen.cc:559)`
`==6937==    by 0x49B9C5: void mlperf::RunAccuracyMode<(mlperf::TestScenario)0>(mlperf::SystemUnderTest*, mlperf::QuerySampleLibrary*, mlperf::TestSettingsInternal const&, std::vector<mlperf::LoadableSampleSet, std::allocator<mlperf::LoadableSampleSet> > const&, mlperf::SequenceGen*) (loadgen.cc:1005)`
`==6937==    by 0x4906B2: mlperf::StartTest(mlperf::SystemUnderTest*, mlperf::QuerySampleLibrary*, mlperf::TestSettings const&, mlperf::LogSettings const&) (loadgen.cc:1212)`
`==6937==    by 0x40B286: doInference() (NOTE: This is an internal NVIDIA function, so I have removed its parameters and file location) `
`==6937==    by 0x40B61A: main (NOTE: This is an internal NVIDIA function, so I have removed its file location) `
`==6937== `

As done before, I am attaching the detailed and trace logs from the non-Valgrind run (in AccuracyOnly mode) below. Please let me know if there is any other information that I can provide to help you debug this issue. 

[mlperf_logs.zip](https://github.com/mlperf/inference/files/3369715/mlperf_logs.zip)


PR #253 should fix the valgrind errors Vikram reported in the last comment.

Vikram, please open new issues for any other valgrind errors that we run into to avoid overloading this one. Thanks!

Yep, PR #253 fixed the Valgrind issues. Thanks - I'll make sure to file new issues for future problems that I find. 

@DilipSequeira, can you please close this issue once you get back?

Closing this one, since @v4njur verified the fix.

@DilipSequeira : please re-open if needed.