[Nexus S General] bedalus:
This information regards the Kernel Benchmark spreadsheet.

Sheet 1: The Overall Scores

I realised that looking at all the raw data was going to give people a headache. (For those who enjoy headaches, like me, the raw data is still visible lower down.)

How did I arrive at the percentages?

First, I looked at an individual result, e.g. the AnTuTu 2D graphics score for simplekernel. I use it as the numerator of a fraction where the denominator is the average of all the AnTuTu 2D results from all the kernels. If above 1, this score is better than the average. If less than 1, this score is below. I do the same thing for all the graphics results, and then calculate the mean. This provides equal weighting for all the various graphics benchmark results. In other words, it prevents a bad quadrant result from skewing the overall picture if the rest of the graphic results are good.

How to interpret these percentages? At the time of writing, if we compare morfic's RAM result of 109% to the terryhau's stock(ish) simplekernel RAM result of 96%, we can say with some certainty that morfic's RAM performs 13% better than stock.

Sheet 2: Statistics for Variance

Firstly, thanks are due to morfic and ezekeel whose scepticism of benchmarks made me want to investigate the issue of variance in the results.

The most important column is n. Check out these rows:

Row 8: AnTuTu's Database I/O benchmark can vary by as much as 7.55%

Row 16: Quadrant Advanced Memory benchmark can vary by as much as 6.66%

Row 21: Smartbench 2012's Gaming Index can vary by a massive 24.42%

Row 25: Vellamo's Aquarium Canvas can vary by 12.11%

Row 26: Vellamo's Pixel Blender can vary by 5.66%

Row 28: Vellamo's Aquarium Canvas can vary by 5.59%

These figures are calculated within two standard deviations of the mean, or in other words, a 95% certainty. For example, we can say that 95% of the time, the Quadrant Advanced Memory benchmarks we run will give a result within 6.66% of a genuinely accurate figure. This is okay, but just not good enough for me!

By eliminating these test results, I have improved the accuracy of my overall scores to within 2% of the genuine mean, with a 95% certainty. In the spreadsheet, you can see the individual accuracies for each of the overall scores: CPU, RAM, I/O, GRAPHICS. Red highlighting of the figures takes these individual variances into account, and shows which tests are definitely at least 1% better than stock, even assuming stock tested at it's maximum, and the compared kernel tested at it's minimum.

Testing Methodology

To test the kernels I want a fair environment, so any differences in the results are down to the kernel, and nothing else. To achieve this I first make sure I have a clean system:

1) Format /system and /cache and wipe dalvic

2) Install the ROM, install the kernel

3) Boot up, use nstools to select deadline for I/O scheduler

4) Then select ondemand governor and set the minimum frequency to equal the maximum, which is not the same as performance, but close enough if you see my CPU governor comparison chart. CPU cycles are wasted as the governor tries to decide what frequency to jump to, even if it has no option! Thanks to Steve Garon for the info. This method was suggested by morfic, thanks again! I/O scheduler is set to deadline. Thanks tchaari!

5) Restore all my benchmark programs. Set everything to off, no gps, no sync, no BT, airplane mode. Force GPU rendering is selected. Wifi is on for connecting to the benchmark servers. A power adaptor is in use so the battery is always full.

Then I begin testing:

6) Power off and power back up. When lock screen arrives, wait one minute to settle the system, i.e. until screen auto-off. Count to three. Unlock, and begin testing, recording all scores at the bottom of my spreadsheet. I copy them up (from the old data to the current raw data section) once complete. This avoids the overall score constantly altering for any current viewers, and means I keep a record of all previous tests under the heading Old Data.