yashsingh1/xdaforums huggingface dataset:
[Nexus S General] :
Sorry, no minor updates... at the moment I'm waiting for sources to be released then there will be a slew of major updates.
[Nexus S General] :
UPDATE: xperia 1.0.1 benched!
[Nexus S General] rrohanjs:
kongs miui has 2 updates already and personally its much better now so it wud be nice if u could benchmark it again
[Nexus S General] glennkaonang:
To help bedalus from too much benchmarking, I think kong's latest ROM doesn't bring many changes.

He stated it himself that there's no big changes in his latest update.
[Nexus S General] rrohanjs:
but miui has got lots of changes .. so might as well bench mark miui only .. they are constantly updating n working on performance and features each week
[Nexus S General] wafflepancake:
from cybergr to CM9 at last pixel rom the faster rom that i have used

i think it would be my last rom until 4.0.5

:D
[Nexus S General] thegtfusion:
I think the gummy rom needs to be updated.. :o . v 0.9.0 was released yesterday..
[Nexus S General] :
I'm not doing incremental updates (unless a developer asks).

I'm going to wait now for the next source code. It won't be too long.
[Nexus S General] thegtfusion:
Haha okay I hope it comes soon then
[Nexus S General] glennkaonang:
But you will do Oxygen 3.0, because almost all of us ask :D

Anyway, wouldn't it be a waste if Adam releases Oxygen 3.0 based on 4.0.3 source code, but then 4.0.5 comes immediately after?
[Nexus S General] thegtfusion:
noooo it wont!! Oxygen will beat every rom in the list I think :D :D :D
[Nexus S General] DROCSID:
Just read changelog(s)!
[Nexus S General] Cybergr:
In this way many people getting confused..!!It has been said many times in the past and i will repeat it once more, Performance benchmarks are not everything!!Yes it`s one factor but there are also other factors to take under consideration, such as Stability - Features - Battery life etc.So don`t be surprised if Oxygen 3 when it comes out don`t makes it to the top.That don`t make a ROM automatically best or worst from another!
[Nexus S General] :
This sentiment cannot be repeated often enough here.

The limiting factor of the performance stems from overheads... overheads caused by features! You can't have your cake and eat it.
[Nexus S General] wafflepancake:
i prefer choose by experience, just try every interesting rom

so i can feel every feature and performance and choose which one better

i am the one who choose the rom not the benchmarker:cool:
[Nexus S General] ferdi7103:
+1...agree also...,flashing...trying...tweaking and feeling is better to choose rom
[Nexus S General] rrohanjs:
i already do .. ur point ?
[Nexus S General] DROCSID:
What my point is? Is that if you take a better and good look on

the changelogs it might look a lot that there has been changed

in the nightly, build or release but it's not worth mentioning in

benchmarking the ROM. That's my point!
[Nexus S General] rrohanjs:
stock 4.0.4 is out .. very interesting to see how it matches the other roms especially against stock 4.0.3
[Nexus S General] thegtfusion:
+1
[Nexus S General] AeroEchelon:
I think it's more interesting if it can beat ported 4.0.4 ROMs.
[Nexus S General] wildestpixel:
Brainmaster's ICS 4.04 is in my testing now the fastest - amazing recipe he has cooked - tweak genius.
[Nexus S General] :
I think now that 4.0.4 is in the repositories, there will be a lot of updates. I think I have a problem with SDCard read speeds, and I could use a more accurate test to help me refresh all the results more accurately. Can anyone recommend an app?
[Nexus S General] :
SD write problem solved.

Will switch from antutu sdcard write benchmark to the rather nice "SD Card Tester" app.

I did a trial of 12 runs, each time taking a reading at 300MB of written data. At a 99% certainty, three passes will produce a result that is on average within 1.5% of the true mean. This is about twice as accurate as the previous method.

At 450MB I could get this figure to within 1% accurate. This means I'll be flashing about around 2GB of data to my phone every time I benchmark a ROM! High quality flash memory apparently withstands 100,000 rewrites before beginning to deteriorate.

2GB = 8 rewrites (for 16GB flash)

100,000*8 = 800,000 flashes before failure.

Oh dear. Taking into account all the flashing I've already done, I may have as little as 799,000 benchmarks left in my device!

I've attached the trial here as an excel document in case anyone is interested.
[Nexus S General] bluelyd:
I think you did very great job, and thank you for your work always.

Thank you.;)
[Nexus S General] :
Tchaari, i will start archiving/tracking the results from now on. I'm finally satisfied the selection of benchmarks is right.

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] DROCSID:
I think I'm using that one too Bedalus!!

Will wipe my phone this weekend and try to benchmark them all again.

On the i9000 too, lots of changes due the 4.0.4 source(s).
[Nexus S General] :
Cool. With that app, set the file size to 500, and just cancel it when it passes 450MB, but keep an eye on the read speed (it vanishes when you hit cancel)

If you don't cancel you have to wait a long time before you can exit.
[Nexus S General] thegtfusion:
bedalus you just created a clone of you but for the i9000 :D :D lolll

-----------------------------------------

Ben jij ook al addicted to benchmarking?? LOLLL

---------- Post added at 06:22 PM ---------- Previous post was at 06:21 PM ----------

I just saw you had 999 thanks.. I felt the need to press the 1000th one :D :D :D:p;)
[Nexus S General] :
Come on, one more...
[Nexus S General] :
Hahaha awesome!
[Nexus S General] thegtfusion:
your welcome :D
[Nexus S General] DROCSID:
Haha! Well, and many more thanking(s) to be collected for you! ;)

And @ thegtfusion

Ben er ook hard mee bezig nu haha! Pokke werk is het wel :D
[Nexus S General] tchaari:
Very nice work again bedalus. You make me more curious to see the new 4.0.4 benchmark results ;)
[Nexus S General] thegtfusion:
woooot wooooot bedalus!!! First Bugless Beast 4.0.4 build is up:

get it here :

http://download.peteralfonso.com/crespo/rom

:D
[Nexus S General] :
Pete's new 4.0.4 BB is out
[Nexus S General] :
Cool, downloading...
[Nexus S General] cowst:
Also nmv 4.0.4 is out.

http://xdaforums.com/showthread.php?t=1364411
[Nexus S General] :
Thanks for the tip. Does this mean cyanogenmod 9 is now pushing out 4.0.4 nightlies? (Since MarkVis builds on that as a base? Or has he integrated evilisto's libraries from the 4.0.4 leak?)

Anyway, it'll have to wait at least one day. I can only download when my provider allows free usage (until next week when my limit resets).
[Nexus S General] Jamalsid:
I can confirm that the 20120329 nightly of CM9 is indeed 4.0.4.

I flashed it this morning. System definitely seems more responsive but its too early to tell really, could just be a placebo effect.

edit: just read some posts in the CM9 thread and it appears that the kernel in CM9 isn't updated with the latest binaries yet, so might be worth waiting to bench this one.
[Nexus S General] :
Thanks for confirming.
[Nexus S General] Jamalsid:
Check my edit ^^
[Nexus S General] cowst:
mh, you're right, we should ask MarkVis :)
[Nexus S General] :
Checked.

Doesn't make much difference to me if only the kernel is behind. I use morfic's T132 kernel for all benchmarks.

Perhaps I'll just give it a few days to see if any bugs pop up.
[Nexus S General] :
UPDATE: Dario's ICS v6

UPDATE: Peter Alfonso's Bugless Beast nightly 29/03/2012

These are the first ROMs I'm certain have synced with the official 4.0.4 release.

As other ROMs update I will update the spreadsheet.

Once I have sufficient ROMs updated, I will deprecate AnTuTu's SDCard score in favour of the more accurate SD Card Tester app. I am also now using updated versions of the benchmark apps, and morfic's T132 CM9 ramdisk version for the 23rd march.

I am now also keeping records of all old results. This may help provide some meta-analysis once enough data has been accumulated. (Although I haven't got any ideas for what yet. Maybe just performance trajectories over time.)
[Nexus S General] rrohanjs:
will you be benchmarking a stock 4.0.4 perhaps brainmasters 4.0.4 ?

wildestpixel took out a new one today based on 4.0.4 instead of the leak but one user had installation problems so i dint bother to try it. he said an update would be coming so it would be nice to see if he betters himself .. rank 1 since quite some time .. great job
[Nexus S General] :
Oh, right.... when wildestpixel said about this, I misunderstood, thought he was talking about MIUI.

Of course, I'd love to do bm bm's (< hilarious abbreviation joke :p:o) stock.

I'll see if I can get it using my mobile data (I've run out of broadband this month).
[Nexus S General] rrohanjs:
lol good one :D :P
[Nexus S General] tchaari:
lol, I think that bm bm's was the first 4.0.4 ROM based on the new OTA. I am curiously waiting for its benchmark to see the effects of supercharging and brainmaster's tweaks on the new 4.0.4 ROMs. First rank won't be so far from brainmaster I think ;)
[Nexus S General] wildestpixel:
I need to clean mine a little when I have time as I've been in London week long working.

Its very slick though ........
[Nexus S General] rasengan82:
Wrong thread lol
[Nexus S General] tchaari:
Oh wildestpixel is worring about his ROM's first place :D

I am just joking buddy, we are really lucky to have talented and generous developers like you, brainmaster and all the others in this forum ;)

Keep up the good work and let the android party go on :D

EDIT: I almost forgot the creator of this thread, many thanks for bedalus (the benchmark man) too ;). I guess he will have a lot of work to do with all these new 4.0.4 ROMs :D
[Nexus S General] rrohanjs:
pixel 1.70 is out !!!
[Nexus S General] :
Thanks, I've got the zip but I'm trying to get cyanogenmod out the gate at the moment. The bloody ROM just refuses to get off 100% processor usage. What the hell is it doing? I don't see any processes. If I can't get it to work in the next few minutes I'll bench pixel ROM first.
[Nexus S General] rrohanjs:
a rom with no name lol - by androiduser00110000

http://xdaforums.com/showthread.php?t=1573799

EDIT : i guess its codename only
[Nexus S General] Pipusz:
Codename without any features, just as stock as possible.
[Nexus S General] :
Ah ha... conflicting tweaks in the kernel. Will sort out my kernel method then retest everything.
[Nexus S General] :
UPDATE: Pixel ROM 1.70 benched!

Unfortunately I cannot resolve the 100% CPU usage problem I'm having with cyanogenmod's 4.0.4 nightly. It's crippling the benchmark results.

I will have to wait for a new release.
[Nexus S General] wildestpixel:
Cheers peasant ;)
[Nexus S General] tchaari:
Kernel sources are available now as stated in these tweets:

http://twitter.com/#!/peteralfonso/status/185890577253007362

Bedalus, I will really miss your kernel benchmarks :p
[Nexus S General] atpiggy:
I really miss them as well :D
[Nexus S General] dinos3:
Me too :"(
[Nexus S General] Articudos:
Stuffed Crust2.3.30 released:-) With official OTA libs, CM-like BIGMEM support, & odexed:D
[Nexus S General] rrohanjs:
This would be very interesting:)
[Nexus S General] :
Cool. I'll get it today, but i think I'm going to end up retesting everything when morfic releases his next t132 based on the new kernel source.

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] :
Got CM9 working normally, and expect markvis's will work too. Also have stuffed crust... Will try to get these three done today.
[Nexus S General] :
And sorry for those people who miss the kernel benchmarks. I'd say the problem with them was that the performance was mainly down to the clock/bus speed. Decide what features you want, and choose based on that, then everything else is down to user experience.
[Nexus S General] tchaari:
Hey bedalus, I see many changes in the scores but I get a little bit confused between the old scores and the new ones. Can you please mark the recent rows (using the new testing formula) with a different colour?

many thanks
[Nexus S General] :
Everything is tested with the same formula.

What I've done with the ROMs that I've not yet tested is re-cycled the old scores...

...except for the SD Card write speed. Only the 4.0.4 ROMs have the new test result.

For all the ROMs that await updates, I have given them a 'place-holder' score for SD write speed. This is just an average based on all the results for the new 4.0.4 ROMs. See cell J71.
[Nexus S General] jcolinzheng:
I think the latest CM9 nightly deserves a higher score, due to having 398MB available RAM (for regular use, the bottleneck for Nexus S is the RAM...).

I have no idea how benchmarking software calculates the RAM scores...The amount of available RAM seems the only logical thing to measure.
[Nexus S General] wildestpixel:
Killing myself at the moment trying to add BIGMEM patch ..................
[Nexus S General] italia0101:
Yea the euroskank one I'm on is the fastest rom iv had ever
[Nexus S General] tycruickshank:
I think I can agree with this :D
[Nexus S General] tchaari:
I am also surprised by CM9 low score (especially RAM). Maybe a compatibility issue with the trinity kernel...
[Nexus S General] Maximilian Mary:
The rom patch can be found here: https://github.com/krarvind/samsung_kernel_aries/commit/77fb0b4fbc657d87599fbc40b7f142da35464e88.

This fixes the broken video when using MOAR MEM kernels (like trinity). I'm not sure where to point you for the corresponding kernel patch. Maybe morfic or kalimochoaz could help.
[Nexus S General] jcolinzheng:
AFAIK the 398MB ram is so far only possible with CM9's own kernel.
[Nexus S General] wildestpixel:
I've implemented it. Got 388mb on my rom with working YouTube and camcorder.
[Nexus S General] :
I don't think so. Morfic has been making dedicated cm kernels for long enough to know the ins and outs. Next cm release i will try another kernel though.

Ram throughput is tested, not capacity. But remember, it's user experience that counts.

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] :
Nice. Good alternative to cm9 if users like the extra ram

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] tchaari:
You are right in all you said bedalus. I just tried to understand why it is having such a low score.

By the way, pixel updated his Rom to 1.71 with bigmem and Slim ICS is getting the 4.0.4 sauce too. It is ready and Krarvind will release it soon ;)

We are also having new good 4.0.4 kernels. New life for our Nexus S :)
[Nexus S General] :
I know, wildestpixel posted above too!

Cm9 has probably got some funky way of managing ram. They've come up with their own ramdisk, so i think i can safely assume they have other changes at the OS level. It could be that they've sacrificed a little performance for some other feature. I don't know? Anyone here able to explain it?

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] :
I do, the table scrolls horizontally!

For kernel benchmarks and more, see HERE
[Nexus S General] tchaari:
thanks bedalus. I just wanted to check if there are some ROMs that are overclocked by default...

I am wo,dering if there a way to measure the battery drain of each ROM even approximatively without the amps meter :) It would be great but I guess a little bit complex and time consuming ;)
[Nexus S General] :
The kernel controls power, the rom will only have an impact if there is a rogue app preventing sleep.

One thing I realise I forgot to measure was how much extra live wallpaper consumes.

For kernel benchmarks and more, see HERE
[Nexus S General] AskingBen:
So if I trust the benchmarks CM9 should be the best ?
[Nexus S General] steve.garon:
Not necessarily... It would be the fastest out of the box but as soon as you put a custom kernel these test don't matter anymore.
[Nexus S General] tchaari:
Thanks again bedalus but some ROMs are optimized and others not. Some of them have a great impact on battery according to users experience. Maybe it needs to be proved by benchmarks :)
[Nexus S General] :
Steve, I do also test with a custom kernel. I perform two tests on each ROM.

Test#1 - ROM with included kernel, default settings

Test#2 - ROM with morfic's kernel, performance governor, deadline I/O scheduler

You can see the effect a custom kernel has in the second test.

Oodie pRiMe has a massive quadrant 2D score (forced GPU rendering? I will check). After I noticed this result, I tested this setting, and calculated that the impact on turning forced GPU rendering ON results in a 4% boost in the ROM's overall score (for either the default kernel, or morfic's).

Also, removing /system/lib/egl/libGLES_android.so helps quadrant 3D and will boost a ROM's overall score of 1%.

I will write about this more clearly once I've finished the whole project.

The OS has no control over power regulation. That is handled by the kernel. But if there is something running that won't die, that is called a rogue app and will prevent sleep. Read this: http://developer.android.com/guide/topics/fundamentals/processes-and-threads.html
[Nexus S General] steve.garon:
True, I forgot about that ...
[Nexus S General] migueldbr:
thanks for your hard work!This helps a lot of people(myself included) because there are a lot of ics roms out there. Wouldn't it be awesome if the devs worked together in order to create a single rom with all the features combined?=p
[Nexus S General] benoit2600:
I will try codename android. thank you for your benchmark !
[Nexus S General] :
Haha, that would be interesting!

Personally I like stripped out ROMs. I preferred Carbon back with gingerbread (which was based on Oxygen 2.3.1) I now use Slim ICS for my other benchmarks (kernels/schedulers/governors) which is a stripped out ICS.

The one feature I do really enjoy is the power widgets in the notification bar. But I think this is being worked on as an addon mod in the development section.
[Nexus S General] :
Thanks for looking! (I try to thank using the thanks button, but have reached my limit of 8 for the day)
[Nexus S General] tchaari:
It is true that the OS hasn't a direct control over power regulation but there are some optimizations and tweaks (in memory management for example) that make the CPU performs less work. This automatically leads to less power use. Anyway, I think that such optimizations can't be easily detectable by benchmarks but they really save battery life.
[Nexus S General] :
I may be wrong on this, but isn't memory management handled by the kernel?
[Nexus S General] steve.garon:
Yes but ROMs can pass different settings to the kernel ... it's all configurable through sys interfaces.
[Nexus S General] :
How awesome is Steve? Thanks again for helping!
[Nexus S General] tchaari:
Thank you Steve, that's what I wanted to say and this is an example :

http://xdaforums.com/showthread.php?t=991276

Brainmaster's tweaks may save battery too. There are more battery savings that can be achieved at the OS level. Some developers add extra apps to their ROMs that offer some goodies but drain more battery...
[Nexus S General] ska nexus:
excellent work !!!

i appreciate
[Nexus S General] rasengan82:
Looks like CM9 is kicking ass in the performance department.
[Nexus S General] mobile_pc:
Since 07 FEB build, Pete's ROM changed completely in performance.
[Nexus S General] Ibn Saeed:
Slim ICS 3.1 ( 4.0.4 ) Released.

http://xdaforums.com/showthread.php?t=1390118
[Nexus S General] :
Slim ICS 3.1 benched!

Isn't scoring as highly as it should on Vellamo's Sun Spider test... not sure of the reason.
[Nexus S General] Ibn Saeed:
But its certainly snappier, maybe, its coz of the 4.0.4 base.
[Nexus S General] :
UPDATE: Darios ICS v6.1 benched!

Went to #1

Nice work Dario!
[Nexus S General] rrohanjs:
nice, this is really fascinating .. updates are coming quicker and quicker now and i must say bedalus its all because of you :D ..
[Nexus S General] :
I think it's mainly because of Google lol :D
[Nexus S General] :
UPDATE: Pixel ROM v1.71 by wildestpixel is back at #1

Congrats!
[Nexus S General] Ibn Saeed:
What i see as the main benefit of this benchmark thread is that it gives a little incentive to the developers to improve their builds, and in order to do that, they improve their android development skills.

This thread gives each of them a sense of little competition, which is quite healthy.

I believe that this is the best thing that has ever happened to the Android Community in terms of getting stable, fast and efficient ROMs on a regular basis.
[Nexus S General] tchaari:
Totally agree with Ibn Saeed :D

PS: Once again, WildestPixel put the bar so high :)
[Nexus S General] Oodie:
Hey bedalus ,

I think its better to include the original scores with the kernel comes from the rom ( like you did before ) . because its the one most are using . not every one is flashing a kernel they like when flashing a rom . Some just use the rom as it is .

You are benchmarking with a same kernel , every rom . but mostly not all uses the kernel you uses . I think its better to add the kernel comes with the rom section to the equation . It'll be fair to everyone as i think .

& lastly , i don't know why you remove that from your equation & I'm sure the reason you removed it is as fair as possible . if so , just ignore this post :)
[Nexus S General] punitanasazi:
bedalus

Firstly, great work on the benchmarking!! Love this thread

Secondly, I wanted to ask whether you have benched the new stock 4.04 AOSP? Cause i cant see it in the results. It would be interesting to know how it does over the stock 4.03
[Nexus S General] :
I'll do it, but where is it?

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] ljordan2:
Evilisto's cwm flashable stock 4.0.4 rom

http://xdaforums.com/showthread.php?t=1569509
[Nexus S General] glennkaonang:
It's actually slightly modified.

AFAIK, the ROM is deodexed, and that it affects performance, though (from my experience with MIUI).

I think you could grab the 4.0.4 OTA flashable zip from the general sub-forum, bedalus.

Just flash it over stock 4.0.3 and you're good to go.
[Nexus S General] :
Probably a lot of users don't flash a kernel. Hard to say the proportion though...

More work for me! But not sure it's necessary...

I'm glad you raise these concerns, but don't worry. My final conclusion for the performance of custom kernels was that the main factor was simply clock/bus speed.

EDIT: I could test with the supplied kernel, but only if there is some sort of consensus. I'll add a poll to the thread. Thanks Oodie for bringing up this issue.
[Nexus S General] :
Cheers Glenn, you saved me some research!

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] :
UPDATE: Poll added.

Should I test using the same kernel for all ROMs? Should I instead just use the kernel chosen by the developer? What about doing both? If you have another idea, please post it!
[Nexus S General] glennkaonang:
I've voted for the supplied kernel.

My opinion is what people want to get from custom ROMs are mostly performance and customizations, but there are also people looking for stability (I'm one of them).

In most cases, users reporting to ROM devs that they have issues with this or that, are simply told by the devs to use the kernel which comes with the ROM (or basically running the ROM with the default configuration), and see if they could reproduce the issues.

My point is that, performance boost and personal customizations are OK, but when it becomes unstable, it would be a bigger problem.

And what I believe is stability is guaranteed by ROM devs if we are running their ROM with the default configuration, in this case the default kernel.