[Galaxy S II General] Mixy:
Introduction

I had some spare time, and wanted to find the best ROM for my phone. So I figured I would run a series of benchmarks on the most popular ROMs i could find for the Galaxy S II. Over 350 benchmarks later, here are my results!

About the testing

- All done on the same phone

- All benchmarks were run 3 times, and then I took the average (exception is battery bench as that takes 5 hours + charge).

- Phone connected to the same network, and lied in the same spot during all battery benchmarks.

- Battery benchmark was done from 100%-50%

- Between each ROM I did a complete data wipe including dalvik.

- There were no SD-card in the phone, and the internal memory only has the ROMs and apks used for testing.

- Display brightness was cranked to max for the battery test.

Benchmarks

- Battery Benchmark

- AnTuTu

- AndroBench

- Vellamo

- CF-Bench

The ROMs

The way I chose the ROM was going through the newest couple of pages in android development and original android dev on this forum. There was a criteria that the ROM was updated during the last month.

So far I've only done 7-8 ROMs, there will be more to follow, but this is a tedious process. The next on my list is doing a couple of the latest official Samsung ROMs to see what they do. After that I'll probably do some kernel test on one of the ROMs

How to read the spreadsheet

The raw data is entered in the first 15 lines, cells show the average score, but if you click on a cell you can see the individual test scores.

The bottom part contains a score from 0-100%, this is weighted to make the highest score from all ROMs 100% and the lowest score 0%. The average score on the right is a flat average. The "Weighted" is where I've removed "Rand Write" test as its useless, and the remainding 5 storage tests count as one to not override the other tests.

EDIT: There are currently two scoring boards with different methods of calculating how the ROM did. One will be removed sooner or later, but will keep both for now.

Top one: percentage as described above

Bottom one: Average score counts 0, twice average score counts 1, and a score of 0 would count minus 1. All scores are very small, so its multiplied by 1000 in the far right column.

Statistical error

Some of these tests are extremely close, within a statistical error. On page two of the spreadsheet I've done eash benchmark 15 times on the same ROM (CM9 for anyone interested). This was to calculate a standard deviation for each test to find out just how reliable they were:

- AnTuTu and CF-bench proved to be the most reliable, with a standard deviation of under 10% of the difference found in various ROMs.

- Vellamo followed at about 20%

- Storage tests proved to be unreliable the exception being IOPS and random read tests.

I also saw some of the ROMs had much closer test results than other.

What's tested

Battery Benchmark tries to simulate a varied use where it turns on/off wifi/BT/GPS and puts the CPU through a series of cycles.

Vellamo is designed as a web benchmark simulating surfing.

AnTuTu and CF-bench is both wide benchmarks with alot of weight on java calculations.

Andro Bench is a storage benchmark testing read and write speed of the memory.

What's not tested

The big one here is stability. I did not use the ROMs as a daily ROM, I only ran the benchmarks. The best indicator here is usually the community you find in the thread of the ROM.

Also non of the ROM-specific features were tested, skins is as always a subjective case.

Top three ROMs

Based on average score:

Sensation

CyanogenMod

Oxygen

The Spreadsheet

https://docs.google.com/spreadsheet/ccc?key=0AuBi7TywV2P4dHd3ZllJUGg4TzhicUMzQm9kQWc3ZlE

--

--

Please remember to donate to the developers of your favorite ROMs :)