Repository: mlcommons/inference
Issue: Effects of increasing minimum run time from 1 minutes to 10 minutes

Increasing the minimum required run time to 10 minutes multiplies quickly when you consider the many benchmarks and scenarios that any given organization might want to run. A few things to consider:

- It generally increases the burden on people, equipment, and schedule.
- It becomes friction against an organization submitting more benchmarks & scenarios and against newcomers to MLPerf.
- It's not clear how far-reaching this 10-minute minimum requirement is supposed to be. Is this supposed to extend to the compliance tests? The accuracy tests? If not, why not and does that have unintended consequences?
- What is the rationale behind increasing the run time to 10 minutes? Is it strictly for power measurement tests? If so, can the requirement just be for the runs that are measuring power, and not be a general requirement?

I think my biggest hangup is that it seems to just increase the burden across the board, which I think is the opposite of what the MLPerf / MLCommons folks in general want.

@psyhtest I assume this affects you more than anyone else. I'd be interested to know your thoughts and suggestions.

The ten minute runtime requirement only applies to performance tests. The minimum requirements in terms of number of queries and duration have never applied to accuracy or compliance tests. Google has observed that for certain systems the performance difference between a one minute run and a ten minute run is up to 15%. We also observe that not all systems are affected equally. Some systems show no performance difference between one, ten, and thirty minute runs. The Offline and Server scenarios capture steady state behavior. Empirically, one minute minimums are insufficient to capture steady state behavior and favor systems with good burst performance. The Single-Stream scenario is a more appropriate place to capture burst performance.

There are more details [192](https://github.com/mlcommons/inference_policies/pull/192).

Thanks for the link w/ more details. It does clear up the reasoning behind the change. Also to be clear, I'm not advocating for changing the v1.0 rules this close to the submission deadline, unless it ends up generally being needed by others.

I would point out a few more questions & considerations:

- It's not clear if you're saying that the Google data indicates that datacenters' average gets better or worse once they reach steady state (e.g., recovering from a ramp-up period versus getting past their own "burst" period at the beginning).
- Prior to this change, organizations were free to make their run time as high as they wanted (anything >= 1 minute). So datacenter submitters could have freely done this if it helped them reach a better steady state performance.
- I agree this rule change helps ensure smaller devices (e.g., fewer resources or thermal capacity) are pushed out of their "burst" zone and into longer term steady-state. Do we know if this was a problem with existing submissions?

> It's not clear if you're saying that the Google data indicates that datacenters' average gets better or worse once they reach steady state (e.g., recovering from a ramp-up period versus getting past their own "burst" period at the beginning).

MLPerf rules allow an untimed warmup period, so there is no reason for performance to ramp up during the timed period. For certain systems, Google measured performance for ten minute runs that was up to 15% worse than for one minute runs. We believe this corresponds to the system hitting thermal equilibrium after an initial burst of activity.

> Prior to this change, organizations were free to make their run time as high as they wanted (anything >= 1 minute). So datacenter submitters could have freely done this if it helped them reach a better steady state performance.

Organizations are still free to make their run time as high as they want (anything >= 10 minutes).

> Do we know if this was a problem with existing submissions?

Google's experiments were conducted in the context of trying to replicate MLPerf Inference 0.7 submissions. The 15% performance degradation in the ten minute run was performed on a replica of a system submitted to MLPerf Inference 0.7.

A few comments:

1. The small # of queries and low minimum run time in v0.5 was mostly there because of smartphones. Those are now a separate benchmark, so we shouldn't be constrained by those prior limits. 
2. 10 minutes per scenario doesn't really seem that long. That's 20 minutes per benchmark.
3. For tuning, you can do shorter runs
4. Additionally, power measurement requires a run time that is fairly long to ensure sufficient number of samples for power averaging. We would like for perf runs and power runs to be the same.

You could use multiple test systems in parallel if you are truly getting limited by throughput.

That being said, we always have post-mortems, and we can see how big an impact this was.

WG: we will keep 10min rule for v1.0 and revisit for v1.1 if needed.