Repository: mlcommons/inference
Issue: [RFE] Implement LogSettings::enable_trace = false
Root cause: Unimplemented Feature Gap

Hello,

Apologies for the late notice, but I would like to get feedback on implementing this flag:
https://github.com/mlperf/inference/blob/52699022d32271a00c7ec840acb77180a8c5d061/loadgen/test_settings.h#L333

Some background: with a high number of QPS on the server scenario, we are seeing a significant amount of time spent in the IOThread, especially in string formatting functions. This was measured using `perf` and the logs are written to a `tmpfs`.

I am aware that this is offloaded in a separate thread, but it does seem to interfere with the main workload; manually disabling the tracing improves our overall performance. It's unclear yet why this is the case, maybe this offloading causes additional context switches, syscalls, or locking; or perhaps the TLS offloading function itself has an overhead.

The internal plumbing for disabling tracing seems to already be present here:
https://github.com/mlperf/inference/blob/52699022d32271a00c7ec840acb77180a8c5d061/loadgen/logging.cc#L208-L212

But this `else` seems unreachable today as this argument can't be `NULL`:
https://github.com/mlperf/inference/blob/52699022d32271a00c7ec840acb77180a8c5d061/loadgen/loadgen.cc#L1611
Thus, it seems that implementing `LogSettings::enable_trace = false` would simply be adding an `if (log_settings.enable_trace)` to function `StartTest`.

I'm not fully familiar with all the logging code, so let me know if there any objection/concern to the above. But I would argue that the Chrome event tracing mechanism is useful for debugging, but is of no practical use for a performance run (even if I still need to dig on why exactly it impacts overall performance).

Also, note that this approach doesn't eliminate *all* the tracing/logging. Looks like the tracing lock will still be taken everywhere before the `if (tracer_)` check.


The LoadGen is scheduled to freeze on Friday. If you can land this before Friday, I am supportive. The default for the flag should probably be false.

PR here: https://github.com/mlperf/inference/pull/466

@tjablin @guschmue -- Could one of you please review and merge this PR?

@flx42 : I used to have a NanoToMicroString function that was 5x faster than the iostream formatting for timestamps. Would be curious if that allows tracing to be always on. If you have time to profile it, you could bring NanoToMicroString back by reverting https://github.com/mlperf/inference/commit/aa01f7c4fdc0b2730372a271fee036ab6f1393c1

The extra IO bandwidth of tracing could also be slowing things down.

Some of the auditing features require traces to be enabled. Are we okay only enabling traces for auditing runs? @georgelyuan 

Yes, that's fine with me, but I don't think that parameter can be set with the loadgen config file as it's meant for TestSettings.
I believe the self-run audit proposal as @petermattson had outlined it stipulates that loadgen patches can be used, so potentially we could re-enable tracing that way..

> The extra IO bandwidth of tracing could also be slowing things down.

That's my concern also. Even when writing to a `tmpfs`, the Chrome trace event format is essentially a very inefficient format in terms of space (JSON and no dedup) and speed (string formatting). But it's great to be able to visualize your execution profile in your web browser :)

I'll try to apply your patch next week.

I actually tried with enable_trace = false before on my dev box (lame, maybe ~ 220qps) and I did not see a change. Maybe at higher qps it matters.

>  I used to have a NanoToMicroString function that was 5x faster than the iostream formatting for timestamps. Would be curious if that allows tracing to be always on. If you have time to profile it, you could bring NanoToMicroString back by reverting aa01f7c

Reverting this commit did help: CPU utilization from the IO thread went from 100% to 78% and overall performance was much better, not as good as `enable_trace = false` but close.

I realized that even if tracing is disabled, trace events seem to still be issued to the TLS logger. Ideally, we would get rid of these too. Individually each event submission is fast, but when they are millions of them, it stops being negligible.