[Nexus S General] bedalus:
This thread is a guide on how to benchmark your phone.

The Problem

Many people are posting screenshots of their quadrant scores with captions like "9th run 6000 :D" which isn't representative of real use. It also perpetuates the notion that quadrant alone can be used to indicate how well the kernel/governor/scheduler/ROM are performing.

The Solution

If you enjoy running quadrant lots of times, you might as well collect all the data from each run in a spreadsheet. Use separate rows for CPU Memory I/O 2D and 3D.

At the end of each row add a formula like this: =average(a1:j1)

...or select the appropriate range for however many runs you did.

The more runs you do the better representation of the true mean you get.

Additionally, you could also include this formula: =median(a1:j1)

...which orders the results and gives you the middle one or a value in between the middle two depending on whether you did an odd or even number of runs respectively.

After doing more than say, 7 runs, once the mean and median start to converge, you know you have a true mean. (Assuming normally distributed data - sometimes referred to as a 'Bell Curve')

To increase the accuracy, you could split the difference between the mean and the median. Add them and divide by two. Or use: =average(mean,median)

...replacing mean and median with the correct cell reference.

Variability

Some scores vary only a little around the true mean. Some vary a lot more. If you want to work out a percentage for how much they vary, you can use something called the standard deviation.

Get the spreadsheet to work it out for you: =stdev(a1:j1)

One standard deviation around the mean will show you where you'd expect to find 'most' results, about 66% of all the tests you run should be in that range. Two standard deviations will give you 95%. Three is 99.7%.

We'll accept 95% as good enough. Here's how to get the variability percentage:

=1-((mean-2*stdev)/mean)

... replace mean and stdev with the correct cell references.

The "mean-2*stdev" bit is the lower boundary above which we would find 95% of the results. (The upper boundary could be calculated by using a plus instead of a minus, but you would end up with an improper fraction.) We use this as the numerator over the mean, and subtract the result from 1. This gives us some number below 1. Now select that cell, and tell the spreadsheet to format it as a percentage rounded to 2 decimal places.

You now have your variability expressed as a percentage of the mean. The lower this value the more you know the benchmark always produces a reliable result.

If you are going to be comparing kernels or ROMs, try to eliminate all user definable variables, such as choice of governor and scheduler, by making the same selection every time, and turn all radios off (except wifi for certain benchmark apps) turn off sync, location services, etc.

If that sounds like a lot of effort, you can always refer to benchmarks I have already produced: here.

Unreliable Results

The following apps' results vary considerably (more than 5%), and may need several runs to obtain an average:

AnTuTu > Database IO

Quadrant > CPU, Memory, and I/O

Smartbench 2012 > Gaming Index

Vellamo > Aquarium Canvas, Pixel Blender, V8 benchmark