Repository: mlcommons/inference
Issue: Load Gen pending items
Root cause: Unimplemented Feature Gap

1. Model-specific constants.
2. Modes: Accuracy, Submission, FindPeakPerformance.
3. Support multiple dimensions + padding for response data.
4. Coalescing all expired queries in server.
5. Warm-up and cool-down periods.
6. General clean up and testing.
7. General performance optimizations.
8. Response data sanity checks in performance mode.

I have to miss the submitter's meeting tomorrow, so here's a quick update.

1. Model-specific constants.
Done in PR #214, but not landed yet.

2. Modes: Accuracy, Submission, FindPeakPerformance.
Accuracy logging has been in place for a while and integrated with reference models.
No progress on Submission and FindPeakPerformance modes.

3. Support multiple dimensions + padding for response data.
De-prioritizing:
* For the purposes of accuracy mode, we can work around this with an allowance in the rules for ignorable padding in the results.
* If we don't need CRCs in performance mode or other sanity checks in v0.5, we also don't need this.

4. Coalescing all expired queries in server.
With a null SUT (implemented as 4 null workers in a thread pool), the loadgen can hit 1.5M QPS in the Server scenario on my local machine. See PR #217. The current bottleneck appears to be the issuing thread, where coalescing could help reduce per-query overhead.

5. Warm-up and cool-down periods.
Not started.

6. General clean up and testing.
Some progress.

7. General performance optimizations.
Added some perf tests. With null SUTs, single stream hits ~500k QPS and server hits ~1.5M QPS.
There is some discrepancy with results other folks are getting with other null SUT setups that are only seeing ~10k QPS. More investigation needed.

8. Response data sanity checks in performance mode.
Unlikely to make it into v0.5 in a form that is enabled for all samples with (3) as a requirement. Would we want to capture model outputs during a timed, but not reported, performance run instead?

Most recent update:

**1. Model-specific constants.**
Recent changes remove all model-specifics (and mlperf-specifics) from the loadgen. Dropped mlperf-aware code for now. Do we want something everyone shares once latency targets have been finalized?

**2. Modes: Accuracy, Submission, FindPeakPerformance.**
- Accuracy:
-- Accuracy logging: done.
-- Post process scripts: GNMT done (max),  resnet in progress (pankaj), ssd (xilinx in progress)
- Submission:
-- Need to decide what submission requirements are. N performance runs?
- FindPeakPerformance:
-- On my plate.

**3. Support multiple dimensions + padding for response data.**
Dropping, unless we want sanity checks in performance mode.

**4. Coalescing all expired queries in server.**
It helps for issue-bound workloads: 1.5M QPS to 5M QPS. Patch needs some cleanup.

**5. Warm-up and cool-down periods.**
No progress.

**6. General clean up and testing.**
Fixed a couple valgrind errors thanks to Vikram's help.
Crash at exit if folks don't join their worker threads: Loadgen WAR deprioritized. Please join your threads before exiting.

**7. General performance optimizations.**
See previous update.
Coalescing can help server scenario go from 1.5M (~700ns) to 5M QPS (200ns). This is with log traces enabled. Does anyone want that?

**8. Response data sanity checks in performance mode.**
Should we just log the responses for e.g. resnet, which is a single 32-bit float?

**9. Documentation**
No updated documentation regarding interface and implementation. Need to update comments in header and README.

Pending items:

- Binary search
- Making target percentiles programmable (now that were not 90%tile everywhere)
- Triaging/debugging specific issues.
- Client-overridable logging ouput streams (useful for testing)
- Hooks for power delays.

Most recent update:

**1. Model-specific constants.**
Recent changes remove all model-specifics (and mlperf-specifics) from the loadgen. Dropped mlperf-aware code for now. Do we want something everyone shares once latency targets have been finalized?
[7/17] right now we are missing a config or helper function to create a submission compliant run.

**2. Modes: Accuracy, Submission, FindPeakPerformance.**
- Accuracy:
-- Accuracy logging: done.
-- Post process scripts: GNMT done (max),  resnet in progress (pankaj), ssd (xilinx in progress)
- Submission:
-- Need to decide what submission requirements are. N performance runs?
- FindPeakPerformance:
-- On my plate.
[7/17] need to have a rule related discussion to determine number of runs needed.

**3. Support multiple dimensions + padding for response data.**
Dropping, unless we want sanity checks in performance mode.
[7/17] we won't log response data during the performance run for any model. Please update the LoadGen spec

**4. Coalescing all expired queries in server.**
It helps for issue-bound workloads: 1.5M QPS to 5M QPS. Patch needs some cleanup.
[7/17] not in LoadGen scope for now.

**5. Warm-up and cool-down periods.**
No progress.
[7/17] not in LoadGen scope for now.

**6. General clean up and testing.**
Fixed a couple valgrind errors thanks to Vikram's help.
Crash at exit if folks don't join their worker threads: Loadgen WAR deprioritized. Please join your threads before exiting.
[7/17] Brian's main focus for now. 

**7. General performance optimizations.**
See previous update.
Coalescing can help server scenario go from 1.5M (~700ns) to 5M QPS (200ns). This is with log traces enabled. Does anyone want that?
[7/17] no more update on this since null backend testing is in the code base.

**8. Response data sanity checks in performance mode.**
Should we just log the responses for e.g. resnet, which is a single 32-bit float?
[7/17] currently not in LoadGen scope.

**9. Documentation**
No updated documentation regarding interface and implementation. Need to update comments in header and README.
[7/17] Brian will update the LoadGen Spec


Tracking pending LoadGen items and assignments here for now:
https://docs.google.com/spreadsheets/d/1RA-2QC7aXmYRWTWnuRR7fnwBhKdiCVUPkK7NDI8_oaE/edit?usp=sharing

still looking for volunteers.