[Nexus S General] bedalus:
Testing Methodology

Each ROM is clean installed, allowed its own default settings, then all benchmark software is installed and run once the system is settled. I also perform one screen off screen on, just out of habbit (this used to resolve a graphics bug in gingerbread, which I'm not even sure exists in ICS). All results are recorded in the spreadsheet. Then I restart in recovery, wipe cache and dalvic (thanks morfic for dalvic wiper) and install morfic's T132-I kernel. Reboot, use nstools to select performance (default I/O scheduler for morfic's kernel is deadline). Reperform all benchmarks, recording them in the spreadsheet. The calculation for the overall CPU RAM I/O and GRAPHICS scores is identical to the one I use in my kernel benchmarking spreadsheet. Please refer to that thread for more detail. However, there is one difference. Quadrant behaves very differently in different ROMs depending on what graphics tweaks they incorporate. As such, no Quadrant scores are used in this ROM benchmarking study. For more details, see under 'Graphics' below.

The final step was to average out the results of both kernel scores, and ranking on that basis. Some would say that they are best ranked by looking at T132 scores only. I would say that is a fair point, and definitely worth consideration.

There are two reasons I incorporate the default kernel score:

1) Overall scores have a worst case scenario accuracy of within 2% of the true mean due to variability in the benchmarks (more detail on how I obtain this figure is available in my kernel benchmarking thread). For normally distributed data (which we can safely assume these benchmarks produce) averaging two sets of results will result in doubling the accuracy to within 1% of the true mean, making the ranking more reliable.

2) Combining the default kernel Scores for ROMs that used a poor performing kernel, or selected bad governor/scheduler combinations by default will suffer as a consequence. These problems can be solved if the user flashes their own preferred kernel and/or adjusts the kernel settings. However, many new users will not perform this step. For that reason, I want to give a score that at least somewhat reflects the performance as the developer intended.

-

Graphics

One of the early findings is selecting force GPU rendering in the developer options improves 2D performance, in Quadrant only. Also, and only in Quadrant, 3D performance can be improved by deleting or moving /system/lib/egl/libGLES_android.so using root explorer or equivalent. These tweaks/hacks do not seem to affect other benchmarks in the slightest.

Because of the bizarre effects of these tweaks on Quadrant scores, I have removed them from the formula that calculates the overall GFX score, and this in turn impacts the overall average score. In short, it makes comparing each ROM's score fairer.

You can see evidence of how the graphics tweaks make Quadrant behave differently at the bottom of the spreadsheet, highlighted in light red. The host ROM for this particular comparison was DianXin or DX ROM for short. This is a reference to a post I make in DX ROMs thread, where I first decide that removing Quadrant is the best option: http://xdaforums.com/showpost.php?p=22323688&postcount=194

-

Android 4.0.1

These ROMs (at the bottom of the spreadsheet, highlighted in light red) were based on early maguro ports provided by koush. I have left them in, because although they are difficult to compare to 4.0.3 builds with the correct drivers and whatnot, they do compare to each other in one important respect: the kernel. One is stock, the other is built by eugene373 (galaxy nexus/nexus s dev.) Using eugene373's kernel, CPU, I/O, and Graphics were all raised to a high standard that is on a par, or above some of the current ROMs. This goes to show that a customised kernel can offer great benefits, even on a ported ROM.

-

Android 4.0.4

As of the 5th of February, my table contains all the ICS ROMs that are listed in the XDA forums. There was only one 4.0.4 version at the time, and this was the best performing in the benchmarks. When the source code is released, I'm sure many ROMs will update. At this point they will benefit from increased RAM scores and javascript performance. If all the ROMs scores improve by the same proportion, the rankings will not be affected. I will attempt to included

major

releases, but I will not reperform these tests every time there is a new nightly.