Repository: iree-org/iree

Issue #15452: Revamp Android Benchmarks
Created: 2023-11-07T01:04:23Z
Request description
Given recent changes to how cpu-features are getting picked up, we need to update the Android benchmarks.

Use --iree-llvmcpu-target-cpu instead of --iree-llvmcpu-target-cpu-features. This means no separation between quant and non-quant models and manually adding CPU features such as +dotprod.

This does complicate the pipeline. We'll need a mapping for each phone and core used. So using a (taskset, target-cpu) tuple:
Pixel 6: (c0, cortex-x1), (30, cortex-a76), (0f, cortex-a55)
Pixel 8 Pro: (100, cortex-x3), (f0, cortex-a715), (0f, cortex-a510)
Motorola Edge+: (100, cortex-x3), (c0, cortex-a715), (30, cortex-a710), (0f, cortex-a510)

Change the number of threads and cpu core group naming. At the moment we bundle cores into either big and little, but there's actually a spectrum. Another reason why GPT-2 is noisy is that it's actually running on 2 different cores (cortex-x1 2.8Ghz and cortex-a76 2.25 Ghz).

To reduce noise, let's make sure the benchmarks run on homogeneous cores. So using a (number of threads, taskset) tuple:
Pixel 6: (1, 80), (2, c0)
Pixel 8 Pro: (1, 100), (4, f0)
Motorola Edge+: (1, 100), (2, c0)
It may not be the fastest configuration but should reduce noise. We can run the fastest configurations e.g. (5, 1f0) either under benchmark-large.yml or the openxla-benchmark repo.
What component(s) does this issue relate to?
Other
Additional context
No response

Duplicate issue #7342: Make pull request benchmarking more stable and indicative
Created: 2021-10-14T15:30:48Z
We are seeing stability issues when benchmarking. This is partially because of the phones in the lab, as we should root and pin down CPU/GPU frequency there when appropriate. That is captured by #6521.
Aside from that, for pull request benchmarking, we also have other issues with stability, which renders it less usable and indicative as it should be. Concretely, as of today, we just run all benchmarks and get average once, and compare with the latest remote branch tip numbers. This is one data point to one data point comparison. This is different from the Dashboard, where one data point pushed to it won't immediately trigger regression/improvement classifications. Instead we need at least 3 consecutive data points to be higher/lower than the previous average with a threshold. So it's resilient to one-off outliers.
We should consider moving the pull request benchmark analysis to be close to the dashboard for consistency:


We can consider run the benchmark multiple runs and get multiple averages and remove outliers, etc. This would mean to reduce the benchmarks run on pull request; otherwise it would take a long time. But that's fine as lots of the configurations are just permutations over, like, the number of threads; they should in general move in a coherent way. We don't need the full scale there to tell regression/improvements. This needs to modify the scripts to support general filters, like #7246.


We can consider adding APIs to Dana to allow querying the current average (and potentially other useful analysis done on historical data), so that we can compare with that, instead of one commit's result (which can be an outlier).

Closing comment (2023-12-16T00:07:29Z): I think we can close this one. The follow-up works on Android benchmarks can be tracked in #15452