[Nexus 7 General] bftb0:
Hi bcvictory,

First, thanks for keeping this current and providing the raw data. It takes a fair bit of time to run through all the test cases, no doubt.

( TL;DR Please consider using a ranking score in your spreadsheets which preserves the relative magnitude of the test(s) used, and optionally also scale the results according to Fmax ratio. )

I have a comment which runs along the same lines as what Trewyy suggested a few pages back, which is this:

Use a "total score" methodology in your spreadsheet which sums up scaled versions of the raw scores for each test rather than summing the ranking orders for individual tests. To some degree, that will preserve an actual sense of how far apart the kernels are; the problem with summing rank values is that the "distance" between the top end and bottom end of the scale only depends on the number of participants in the test - not a sense of how far apart they actually were according to the yardstick(s) which were used during testing. [*1]

I think that summing rank scores contributes to an impression that the kernels differ by far more than they actually do.

Using a (normalized) "reference = 100" scale is something folks are used to - it's sort of like a percentage scale. Take the raw score, divide by the raw score of the reference case (Stock kernel) and multiply by 100. When you add these types of ranks together - do the same thing: divide all the sums by the summed score for the stock kernel, and then multiply by 100. All scores are then on a reference = 100 scale.

Here's your 1/27/2013 (performance) raw results renormalized to a "reference = 100" scale, and also a frequency-adjusted value (which of course are identical when all kernels run @ 1.3 Ghz):

Performance @ 1.3 Ghz

Kernel Freq-Max RAW NORM F-Scaled

1. Trinity 1.3 13996 108 108
2. Faux 1.3 13972 108 108
3. Franco 1.3 13954 108 108
4. M-Kernel 1.3 13763 107 107
5. S-T Squirrel 1.3 13720 106 106
6. Omega 1.3 13673 106 106
7. LeanKernel 1.3 13625 106 106
8. IntersectRav 1.3 13491 105 105
9. Bricked 1.3 13463 104 104
10. Motley 1.3 13251 103 103
11. Stock 1.3 12903 100 100

Performance @ Fmax

Kernel Freq-Max RAW NORM F-Scaled

1. Omega 1.6 16341 127 103
2. S-T Squirrel 1.6 16338 127 103
3. Faux 1.7 16329 127 97
4. Trinity 1.7 16254 126 96
5. M-Kernel 1.7 15938 124 94
6. Bricked 1.6 15662 121 99
7. Motley 1.6 15285 118 96
8. LeanKernel 1.5 15194 118 102
9. Franco 1.3 13954 108 108
10. IntersectRa 1.3 13491 105 105
11. Stock 1.3 12903 100 100

(Note that using the Fmax- normalized score the rankings here would be completely different - the Stock kernel would be square in the middle of the rankings)

So, last night I ran the Antutu benchmark 11 times in a row on a stock kernel; I got

mean = 12874

std. dev = 80.2

That means that if I were to randomly run the Antutu benchmark (on my device with my unique set of apps & configs), 95% of the individual scores that I would observe would fall within a +/- 2 std. dev. range, or roughly 12714 - 13034. Normalized to 100 <-> 12874, this is a range of 98.8 - 101.1 ... a sampling range of roughly 2.4%. (I understand you are taking an average of three measurements, so your results should show a little less scatter than this [*2] ). [*3]

Anyhow - when I look at the scaled results - I see a couple groups of indistinguishable kernels. In the first performance test (@ 1.3 Ghz), the first four listed kernels are statistically identical; and then the next four are statistically identical. Same sort of thing for the "Fmax" performance tests, although the groupings are different - a few groups of more or less indistinguishable kernels.

Finally, for these two performance tests I'll sum up my two (Stock=100) performance ranks, renormalize them again (divide by two), and compare them to your "Performance Total" score:

Kernel N-Total F-Total P-Total

Trinity 117 102 5
Faux 117 103 5
S-T Squirrel 116 105 7
Omega 116 104 7
M-Kernel 115 101 9
Franco 108 108 12
LeanKernel 112 104 15
Bricked 113 101 15
Motley 111 99 17
IntersectRa 105 105 18
Stock 100 100 22

See what I mean?

Anyway, don't take this as a poke in the eye; far from it! Without your data I couldn't make any of these assertions! More data is always better than less data.

Also, my point here is not to try to discourage anyone from taking an interest in these things. Most of what is done here on XDA is generous donations of individual's free time - and they get to decide what piques their interest and what does not.

I do think that utilizing a frequency-scaled benchmark value is important though: we all highly value at least two things: (1) when we actively are using the device, we want it to work for us as fast and furiously as possible, and (2) when it is not in use (or we are staring at a page of text as if we were zombies), we want it to be as miserly as possible with the battery. So, using a Fmax-normalized score tends to focus the benchmark values on subtle coding changes which revolve around the rate governor behavior(s), off-lining of "spare" CPUs, and under/over-volting.

cheers

bftb0

[*1] I get it - computing synthetic benchmark scores opens up all sorts of questions about weighting values used per individual test, adding together values with different units, etc. Imagine that we were comparing 10 kinds of apples based on water percentage, sweetness, and crispness. It can well and truly be said "Hey, you can't add water percentage to sweetness and get a meaningful number" - even if the testing regimen is completely repeatable and accurate. But, that can't be avoided with synthetic benchmarks - it turns out that we might value all of those different qualities in apples. On the other hand, if those ten kinds of apples only differed by a (statistically significant) few tenths of a percent in every different measurement, the highest-ranked apple might get a "sum of ranks" score of 1, and the lowest-ranked apple would get a "sum of ranks" score of 10 - despite the fact that they were essentially indistinguishable to the average apple-eater.

[*2] An extremely disciplined approach would use a Student's T test to answer the question of what was statistically resolvable given a sampling size of 3. Thank goodness you didn't provide all the raw data; then I probably would have wasted even more time with that ;)

[*3] I made elapsed time measurements for these 11 repeated trials; curiously, the elapsed time between pressing the start button and presentation of the final results varied from 176 seconds to 213 seconds (mean = 190 s; std. dev = 12.4 s; N=11). That's a "one std. dev." variation of 6.5%, even though the total benchmark scores varied only by 0.6% (one std. dev.). Sort of odd, really.