[Nexus 7 General] bftb0:
Hi.

I'll admit to not reading the whole thread. Here's my take using the raw data presented in the OP (as of 2013-01-12). I made assumptions about Fmax for each kernel based on the OP's suggestions; I didn't verify those numbers myself.

PERFORMANCE

Performance was tested with Antutu.

1.3 RESULTS

Kernels set to 1.3GHz max and used the default governor for the test.

Norm1a*
1. Faux:	14088	1.0918
2.Franco	13630	1.0563
3. Trinity:	13572	1.0518
4. M-Kernel:	13521	1.0479
5. Bricked:	13463	1.0434
6. LeanKernel:	13439	1.0415
7. S-T Squirrel:	13362	1.0356
8. Motley:	13251	1.0270
9. IntersectRaven:	13030	1.0098
10. Stock:	12903	1

*Norm1a : divide raw score by Stock score, e.g. "Faux" is 9.2% higher than "Stock".

MAX RESULTS

Kernels set to highest clockspeed, performance governor.

Norm1b*	Fmax	Norm2**	Norm3***
1. Trinity (1.7):	16182	1.254	1.7	0.959	0.912
2. S-T Squirrel (1.6): 15918	1.234	1.6	1.0026	0.968
3. Bricked (1.6)	15662	1.214	1.6	0.986	0.945
4. M-Kernel (1.6):	15597	1.209	1.6	0.982 0.937
5. Faux (1.7):	15351	1.190	1.6	0.967	0.885
6. Motley (1.6):	15285	1.182	1.6	0.962	0.937
7. LeanKernel (1.5):	15005	1.163	1.5	1.163	0.968
8. Franco (1.3):	13630	1.056	1.3	1.056	1
9. IntersectRaven (1.3): 13030	1.098	1.3	1.098	1
10. Stock (1.3):	12903	1	1.3	1 1

*Norm1b : divide raw score by Stock score

**Norm2 : Downscale Norm1b by frequency ratio, e.g. Norm2 = Norm1b*(Fmax_Stock/Fmax)

***Norm3 : Ratio of AnTuTu scores (MAX_rawscore/1.3_rawscore)*(1.3/Fmax)

Norm3 tries to measure the effect of the "default" rate governor used in the "1.3" test

to the performance governor used in the MAX test, scaling the results to account for the highest

frequency available for that kernel in the MAX test. If the "default governor" has a mean expectation on it's distribution of frequency bins (over the given workload)

that is identical to that for the "performance" rate governor, this statistic would be exactly equal to 1. I suppose that means that it should usually be less than 1.0 if the performance governor behaves the way it is named (it should loiter near Fmax longer than all the other rate governors, no?).

What's the point of those renormalizations? Basically, that the bulk of the performance benchmark differences (observed for the workload chosen) from kernel to kernel can be mostly accounted for by the overclocking. All the remaining kernel software diddling with governors, schedulers, modules and other kernel drivers - for the given workload - produce modest differences of less than 10%.

For me, a good kernel is rock-solid (no oops, deadlocks or filesystem corruption) and has the features that I need to get whatever I want done.

It would be interesting to do the same renormalization analysis with the Battery Life benchmark using the P ~ fV^2 formula where the "f" value is Fmax for a kernel/ROM combo and V is the Voltage at Fmax. Offhand I don't know what the Voltage stepping is with frequency - I haven't looked around for that (yet).

Benchmarks are a little bit like trying to figure out how far a dog runs in a day by measuring it's fecal output; at the ends of the performance spectrum the two numbers are well correlated, but in the middle there's a lot of arguing about the dog and all the measurements are crap :D :D

cheers

PS There do seem to be some outliers in the various normalizations; in particular Faux (Norm1a, Norm3) and LeanKernel (Norm2). It is also curious that the scores for the kernels with 1.3 Ghz Fmax have identical Antutu scores between the MAX and 1.3 case - that suggests that the workload creates a condition where the effect of the rate governor does not matter.