yashsingh1/xdaforums huggingface dataset:
[Nexus S General] :
Once again thegtfusion is all over it :D

Thanks, will have a look tomorrow!
[Nexus S General] thegtfusion:
Haha yess. :D thanks :D
[Nexus S General] :
Okay, here's what I've been working on today:

I took Pixel ROM 1.72 and MIUI Catalyst 2.4.6_r2 (two high performance ROMs but very different) and tested out several kernels at stock speeds using different IO schedulers to see which was best.

I wanted to know about IO specifically because this seems to be the factor holding back the scores of Supplied Kernels.

Briefly, the two best choices are,

1) Thalamus kernel

----- With the deadline scheduler

----- 9.39 MB/s in Pixel ROM

----- 9.13 MB/s in MIUI Catalyst

2) Simple Kernel 1.20

----- Choose the CFQ scheduler

----- 9.13 MB/s in Pixel ROM

----- 8.7 MB/s in MIUI Catalyst

The average speed of all ROMs using their Supplied Kernel is 7.67 MB/s

This means if you could be getting roughly 20% better I/O performance by switching kernel and choosing either CFQ (for Simple Kernel 1.20) or deadline (for Thalamus' kernel).

If anyone tries this, post back about how you get on.

Attached is the study in Excel format.
[Nexus S General] straightdrop:
nicee

Great work bedalus. I'm new to xda but I'm familiar with some of your work. I'm surprised brain masters cm9 black scored so low interms of performance, I swapped cm9 kernel for air kernel 3.9.5 and noticed improvement in battery life on stock settings :D

12hours on medium high usage with 4h 47m screen timer and a few user initiated reboots draining approximately 5% of the 94% I started with, not bad for 89% :D
[Nexus S General] :
Brainmaster has released two updates since I first tested his ROM, I will wait for it to settle, then test again.

I'd be interested to see how Air Kernel performs if r_data could implement Thalamus' deadline scheduler.
[Nexus S General] :
UPDATE: PlayROM v2 9APR ; AOKP Build 31 9APR ; MIUI Catalyst 2.4.6_r2 9APR

forgot to update the thread yesterday!

adding pool party next...
[Nexus S General] ZeppeMan:
Can you also test rascream?

For me this the most smooth/fast ROM there is together with the latest T132 test kernel.
[Nexus S General] :
Not until it has an XDA thread! But I tested it briefly, no issues found.
[Nexus S General] :
UPDATE: Pool Party V.5 ICS benched!
[Nexus S General] tchaari:
bedalus, don't shoot me. I have another suggestion for you :o

Why don't you add another column where you benchmark each ROM using T132 at 1000 Mhz frequency? This would be less complicated and more fair to evaluate the stock Kernel. If T132 at 1000 Mhz is performing better than the stock at the same speed, the supplied kernel has some problems. It is another way to express the same current state of the spreadsheet but in simpler and more fair comparison. By my suggestion, we will have just 3 columns: T132@1,32Ghz, T132@1Ghz and SuppliedKernel@1Ghz.

Just try to think about it as a long term suggestion. I don't want to let you do too much benchmarks and make your wife angry :D
[Nexus S General] rrohanjs:
even i have a suggestion .. :D

you can have a column showing the roms previous versions score, so the user knows by how much is it better or lesser ..

for example im using rom xyz version 1.5 .. and its score is say 40% .. now comes out 1.6 .. and the score is 42% or 35 % so when 1.6 is benchmarked i can see the previous versions score in the bracket if not a new column .. as a user that would be interesting .. does anyone agree ?
[Nexus S General] glennkaonang:
I have tested it myself actually.

Using T-11 with Bugless Beast & Slim ICS, and their supplied kernel.

I set the freq 1000/1000, ondemand, sampling rate 40000, up treshold 95.

I use the latest Antutu to benchmark.

Magically, T-11 performs better, even when set to 1000, not the default 1096.

I got around 180 more scores with it.

I guess bedalus might consider this (but remember, more burden to come :D :D)
[Nexus S General] :
I've requested this thread be closed. If anyone wants my advice, feel free to pm me.

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] dinos3:
Sad news :(

Will somebody continue benchmarking kernel thread ?
[Nexus S General] Oodie:
What ??

Oh nooo !! ....

Sent from my Nexus S
[Nexus S General] :
Nothing! I'm just tired of it! If anyone wants to take over, pm

Thanks again to tchaari, glenn, harbb, kong, oodie, simms22, morfic, cybergr, wildestpixel, brainmaster, steve.garon, anyone else who helped, thegtfusion, mobile_pc, android_phone, a few other people who regularly provided me with links, sorry if i forgot your name... All the developers...

Remember, the perfect ROM is the one that is perfect for you! And that could be different for everybody.

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] beppegg:
Thank you for all your work!
[Nexus S General] Andy:
Closed by OP's request.
[Nexus S General] :
Yes, the new T132 doesn't have the fsync patch, so will produce lower SQL benchmarks.
[Nexus S General] Oodie:
is this the one you used ? :rolleyes:
[Nexus S General] :
Sorry, no I used the previous incarnation, which was completely bare boned, T132-I.

Is it this: http://goo.gl/AjlCj

Can't check myself right now...
[Nexus S General] Oodie:
Thanks :) will try it .
[Nexus S General] tchaari:
I am very thankful for your long and hard work bedalus. I think that my ideas are a little bit time consuming. Just to feed the mind if they are interesting :)

- Benchmarking a GB ROM to compare them with the ICS ones.

- Benchmarking some kernels for the TOP 3 ROMS to identify wich kernel performs the best for these ROMs (this was already done for SLIM ICS in the Kernels benchmark). I think that kernels won't behave the same on different ROMs.

Thanks, this makes sense. I didn't payed attention that there are 7 days between them in the spreadsheet.

No problem, your work is already thick. I just think that this ROM deserves a benchmark because it is a serious candidate to be at the top ;)

Nothing urgent though :)

Thanks again dear bedalus ;)
[Nexus S General] :
No GB benchmarks. GB is eaten. And I don't think there's any reason to suspect that ROM/Kernel combos work like Governor/Scheduler combos.

It is done.
[Nexus S General] tchaari:
Oh thanks bedalus. That was so quick. The result is a little bit surprising. I don't know what irtehun done but his ROM is based on that port. There is almost 4% between them :eek:

Here is another benchmark idea bedalus :D

Test the impact of the freshly released ICS supercharger script on performance :

http://xdaforums.com/showthread.php?t=991276

cheers.
[Nexus S General] Solarian:
I haven't run Slim myself, but I ran another ROM that is ranked higher than BB, and it runs like utter crap. It may or may not be faster than BB under ideal conditions when benchmarked, but in use it was very choppy, with obvious lag coming up quite often.

I think benchmarks are useful and all, but don't tell the whole story.
[Nexus S General] :
I think it depends which BB nightly you use. Did you use the same one as I did on my test? Two users reported to me that the one released after the one I tested ran faster. The one I tested felt choppy. I may have to redo pete's ROM if I keep hearing this sort of thing.
[Nexus S General] :
I may be completely wrong on this, but I do believe that thing is snake oil.
[Nexus S General] Solarian:
Ahh, good point. I just noticed that the one in the benchmarks is the 6th Feb, and I'm running the 8th Feb. It's the smoothest ROM I've ever run on my phone hands down, really big difference from all the ones I've run before.
[Nexus S General] :
Well, let's hope that's the reason. Pete's got a good reputation around here so I'll give his ROM another pass, but it'll have to wait until the Deep Idle test is done in about 6 days.
[Nexus S General] tchaari:
Sorry bedalus, that wasn't correct. Irtehun told me that his ROM is not based on evilisio's one. It is based on another deodexed version...
[Nexus S General] :
afaik there was a leaked 4.0.4 for the ns4g from sprint, and that's what both of those ROMs are based on. I could be wrong.
[Nexus S General] thegtfusion:
bedalus, irtehun released nuhetri reloaded v2.1 yesterday. I think you should update your benchmarks :D (you know his 4.0.4 build :D)
[Nexus S General] tchaari:
Yes they are both based on that single leak but they worked on it independently. I thought that irtehun took evilisio ROM and worked on it to produce his reloaded one.
[Nexus S General] :
Yes, it's very impressive in the scores, but I doubt they will change. I'll just update the sheet to say it's working properly now.

Thanks.

IN OTHER NEWS: DEEP IDLE WORKS!

Amps measured. Video available. More details to come in the battery thread >> http://xdaforums.com/showthread.php?t=1478406&page=22
[Nexus S General] boenklon2:
wow u did a great job.

i always want to look for a technical data performance in each rom.

thx dude
[Nexus S General] thegtfusion:
Slim ICS 2.5 came out :p
[Nexus S General] :
Hello thegtfusion, I've noticed you've been really helpful keeping me informed. But I ran out of my 8 thanks for today already!

I don't think I'll be doing any more tests unless someone wants a brand new ROM testing. I think the new source code will be out soon, so I'll be waiting for major updates, not cosmetic ones... they are not likely to affect performance.
[Nexus S General] Ibn Saeed:
Better to wait till 4.0.4 is released and the ROMs follow suit, and then re-run the benchmarks. No use of running it for minor release versions.
[Nexus S General] :
krarvind himself told me we may in fact be waiting for 4.0.5

Was there ever a 4.0.2?
[Nexus S General] tchaari:
I think 4.0.2 was for the galaxy nexus. They received the 4.0.4 since one or two weeks I think. For the nexus S, we had only 4.0.3 in ICS. Maybe, we will directly have 4.0.5 but I am not too optimistic to have it before the second week of march :)
[Nexus S General] thegtfusion:
yeah I understand :P (it must be hard to benchmark every single thing on every single rom )
[Nexus S General] :
UPDATE: Codename Android 1.3.3 refreshed at dev request
[Nexus S General] nexusphone:
I think bedalus is the busiest man in our community. Many thanks :)
[Nexus S General] :
Thanks! Hopefully I've achieved something. Perhaps I've helped a few users to choose

1) The right ROM for their needs

2) The right kernel for their needs

3) Set it with a good governor / scheduler combo

4) Choose the correct Deep Idle settings if they use their phone with screen off

5) Not to bother with UV
[Nexus S General] Ibn Saeed:
Then, which ever version release comes up, re-run the tests again.

beladus, one advice though. it seems that the ROM's benchmark really takes a lot of your time.

1). Why don't you select the best 5 ROMs out of all of them to be tested on major releases.

2). Or just take a small fee ($5.00) from the developers of each ROM, to get their ROM benchmarked, for the hardwork you put in.

---------- Post added at 07:07 PM ---------- Previous post was at 07:06 PM ----------

Your ROM and Kernel bencmark really helped me choose the best combination for my nexus s.

I have found the perfect match for my needs, Slim ICS + Trinity T132, it really flies. Also, Slim + Steve Garon's is quite good as well.

Thanks once again for all the hardwork, even though you deserve to be treated with chocolates
[Nexus S General] :
Haha, you made me laugh. Nearly as good as a chocolate treat. :)

No, I'm not charging. Everyone just do me a favour, check out an underdog once in a while :)
[Nexus S General] Erythnul:
Heya Bedalus :)

Was wondering, does the new version of Quadrant affect scores very much? Would hate to have to see you rerun all that work because people want to see updated Quadrant scores :P
[Nexus S General] :
The problem with old quadrant was the scores weren't very reliable. The 2d and 3d ones are very consistent, but only within the same ROM. For those reasons i ended up removing quadrant from the overall calculation entirely. I may just remove those columns from the spreadsheet because they don't influence anything. No need to rerun with new quadrant.

For kernel benchmarks and more, see here: http://goo.gl/mpeHI
[Nexus S General] teddydeluxe:
really nice. thank you :)
[Nexus S General] PatchouliNim:
Wow.. amazing work! It's very awesome! :)

but I'm surprised because of Cybergr rom's low benchmark!

I fell that It is very fast and smooth to my phone;)
[Nexus S General] :
TBH there is not a massive gap between the best and worst... Just a few percent. Probably not even enough to notice!

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first post. Convenient for XDA app users!
[Nexus S General] PatchouliNim:
Oh I got it, thanks!

But in Battery life, gap between best and worst is very big!

When I use deadline + lazy, I feel bettery drains very fast! ;)

And while I am playing HD 3d games, I can feel your recommend setting's performance is better than I used!

Really thanks! :D
[Nexus S General] benoit2600:
have you planned to benchmark the last codenameAndroid (1.4.0) and the last CyberGRmod ( v.8 )

I find them more smooth than before :)
[Nexus S General] :
I'll have a look at their changelogs.

If they've made any major changes, I'll consider doing an update.

EDIT: CyberGRmod has no major changes, but CNA has a few...

Probably means there are a few less services eating up the CPU cycles

These sound like plausible improvements to the OS. They might have an impact on benchmarks. I'll try to get this updated today.
[Nexus S General] :
UPDATE: CodeName Android 1.4 refreshed result.

No change in ranking. Default kernel went up a percent, T132-I kernel went down 1%
[Nexus S General] :
UPDATE: Mandaman2k_v6 ROM moves up to 4th place*

UPDATE: updated Oodie pRiMe MIUI4 ROM

*due to a formula error, fixed now (just removed all quadrant scores from the table, not that they were included in the scores anyway)

Also, people have been reporting that bugless pete's ROM just had some glitch in it for the version that I tested. Since Mandaman2k's version is based on bugless pete's, I think I will now do another pass of his too.
[Nexus S General] :
UPDATE: Peter Alfonso's Bugless Beast latest nightly (16th Feb) updated on the sheet. Moved up a few places.
[Nexus S General] mobile_pc:
Peter Alfonso, 9th place? That ROM's public image is very different from the real? :(

But, it's a relief to see the following :)

Bedalus, Big thanks for your hard works :)
[Nexus S General] :
It's also worth remembering that there is a little variability in the results. 1% up or down. Although that is best or worst case scenario.

But yes, not a huge difference in performance between any of the ROMs

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users![/QUOTE]
[Nexus S General] irtehun:
thanks for this =)
[Nexus S General] wildestpixel:
You've one amazing ROM there irtehun in the form of 4.04.
[Nexus S General] :
Nice to see you on the thread!

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users![/QUOTE]
[Nexus S General] :
Consecutive thread hits from two of the top five!

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users![/QUOTE]
[Nexus S General] irtehun:
Yes, and its expanding to crespo4g today =)
[Nexus S General] :
Those lucky devils...

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users![/QUOTE]
[Nexus S General] irtehun:
[/QUOTE]

I think this kinda counts as bragging rights for me correct? :D
[Nexus S General] ballroom dru:
Thank you for doing this. I would not be enjoying my current ROM (the #1) If you had not done this.
[Nexus S General] :
I'm number one! I'm number one! Now run out into the street, pull your shirt over your head, and yell 'woo' while waving your arms. (it's what i do, even though I'm not number one)

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users![/QUOTE]
[Nexus S General] krarvind:
[/QUOTE]

I am the real number one, only working with sources count, he he :P. Btw F.Y.I, slim 2.6 is out. Keep up the great work mate.
[Nexus S General] irtehun:
I am the real number one, only working with sources count, he he :P. Btw F.Y.I, slim 2.6 is out. Keep up the great work mate.

So not even cool. Editing via smali is a lot more complicated then editing via source!! Lol

Sent from my Nexus S running nuhetri-reloaded
[Nexus S General] irtehun:
[/QUOTE]

Lol so now u try to get me hit by a car!

Sent from my Nexus S running nuhetri-reloaded
[Nexus S General] krarvind:
no need to get offended mate. Wasn't taking a dig at you. You prob missed the smiley. Cheers
[Nexus S General] :
FAKE NEWS BULLETIN:

Today we mourn the tragic loss of irtehun, ROM developer, killed by a newspaper delivery boy on a unicycle. Eye witness krarvind said, "The unicyclist did not just come out of nowhere, I could see him coming from a mile off. But irtehun had his shirt over his head, also he was yelling 'I'm number one!' repeatedly. I wanted to warn him, but I didn't because I had to get back to my computer to sync to the latest google sources." The unicyclist's lawyer issued the following statement, "My client could not brake to avoid irtehun, as unicycles do not have brakes." The family of irtehun refused to comment, but an old woman claiming to be a close family friend (a big fat lie) said, "He was a nice young man, with a nice ROM, here, take a look at my AnTuTu score. The Nexus S is only a single core phone you know..."

R.I.P. irtehun, 1999-2012.

The above story is made up, and stupid, no-one should read it, and I have no idea if irtehun is 13, but I bet he is. [/envy of young talented people who understand things I never will]
[Nexus S General] irtehun:
Lol was not offended I was kidding around also

Sent from my Nexus S running nuhetri-reloaded
[Nexus S General] irtehun:
Lmao. Nope actually I'm 29

Sent from my Nexus S running nuhetri-reloaded
[Nexus S General] krarvind:
All good then. Cheers.

---------- Post added at 07:11 AM ---------- Previous post was at 07:11 AM ----------

changing profession to a story teller from an analyst mate ?
[Nexus S General] :
I woke up at 2am today with indigestion caused by eating canned sausages (only 17% pork, according to the 'ingredients benchmark' (as I like to call it (I also like nested brackets)) on the label).

At 2:20am our senile cat has night terrors, and starts meowing. Meow is not a suitably onomatopoeic word for the sound our cat makes. She sounds more like Eric Cartman having a difficult poo: "Merrrghhw..."

At 2:20am + 2 seconds, our child wakes up screaming. I'm already awake so I get up and give her a cuddle.

At 2:59am our child is still staring up at me silently

At 3:00am our child starts giggling. I put her back in the cot and go downstairs. After making myself a cup of tea, it is clear our child, our cat, and our wife.... um... my wife are all sleeping soundly.

At 3:10am laptop is on, benchmarks are happening! Good times. [/insane rambling sleep deprived pseudo-explantion]
[Nexus S General] tchaari:
Never thought that benchmark addiction is such dangerous :D

You can't even imagine this danger if you wife woke up catching you doing the benchmarks :D:D:D
[Nexus S General] :
hahaha if she came downstairs and found me furiously benchmarking smartass in the middle of the night... divorce.
[Nexus S General] tchaari:
then you should have to invent some smart excuses to save your ass :D
[Nexus S General] krarvind:
I am a little fortunate my 21 month old only wakes me up at 5:30.
[Nexus S General] ballroom dru:
My 15 month old happily is also my 7:30 AM alarm clock. I do not. Miss the 2AM feedings.
[Nexus S General] tchaari:
Hi bedalus,

I hope you slept well after your late night benchmarks :)

I feel that the average score is a little bit confusing. What do you think about separating the stock kernel scores, morfic's T132-I kernel scores and average scores? I am wondering if it will be better to have the 3 rankings separated. It will give a better reading I think.
[Nexus S General] :
You're the first person to bring that up. When I first decided which way I would rank them, I excepted a lot of people to ask for the rank to be based on the T132-I kernel alone because it would appear to be a more fair test.

I considered it, but decided to rank using the average anyway. There's a more detailed explanation why in the second post: http://xdaforums.com/showpost.php?p=22200114&postcount=2

EDIT: Wait, do you just mean visually confusing? Not aesthetically pleasing enough?
[Nexus S General] tchaari:
effectively, that's what I wanted to say. Maybe I miss expressed it. My idea is to change the presentation of the spreadsheet by presenting the ROMs ranking based only on their stock kernel in a first part of the spreadsheet. In a second separate part, ranking based on T132-I kernel only and in a third the average ranking. It's just a visual change on the spreadsheet to have better reading in it :)

EDIT1: it remains a suggestion and you can let it go if it doesn't seem important for you ;)

EDIT2: what I have in mind next to this idea, is to have a more fair summary in the OP where you detail the top 5 ROMS according to only their stock kernel. Then, another TOP 5 based only on T132-I kernel and finally another top 5 presenting the average :)
[Nexus S General] :
I've retested a few ROMs due to updates, and apart from bugless beast (which apparently had some issues in the original version I tested) there haven't been any shifts in the ranking. I think this is because of the measures taken to remove variability, i.e. combining the two kernels' results. I want the ranking to act as a reliable indicator of general performance. I think it does that job well in its current form.

There isn't a huge difference between the best performing and worst performing anyway. Besides that, I think people are smart enough to look at only the default kernel, or only T132-I, if they have a different way of making a judgement.

I think I posted somewhere that everyone should try an underdog once in a while. And I do state that users should check features and other users' feedback first.

I think three different rankings might make the table more complex, not simpler! Sorry tchaari, not this time.
[Nexus S General] tchaari:
ok bedalus, it is aesthetic anyway... So, not so important ;)
[Nexus S General] tankkiez:
Wait for 4.0.4 on cm9.
[Nexus S General] :
I think all ROMs will get a little boost when the next source code is released.

kernels ; battery ; ROM ; gov/sched
[Nexus S General] thegtfusion:
did you bmark cm9 v3.0? (already :D)

btw, which rom are you using now?
[Nexus S General] :
The one I use most often is Slim ICS (for testing new kernels) but it depends on what ROM I last tested sometimes.

I prefer simpler ROMs, so when there's no testing to be done, Slim ICS suits me fine.

I've tested CM9 v3's kernel, but not done the ROM itself yet.
[Nexus S General] :
UPDATE: Brickmaker by Kong added to the chart!
[Nexus S General] Rose_of_Sharon:
bedalus' works for both devs and users :)
[Nexus S General] :
UPDATE: Cyanogenmod 9 v3

It's taken some sort of performance hit. Dropped out of the top 5, meaning evilisto's 4.0.4 ROM moves up into 5th place.
[Nexus S General] tchaari:
Bedalus, I saw that the scores have changed. Have you modified their formula?
[Nexus S General] :
No, the way they are worked out is centred around a mean generated from all ROMs, so if one score changes, they all shift slightly, but it is a uniform shift (excluding the one under test obviously!).
[Nexus S General] tchaari:
Ah ok, thanks bedalus. I am loving your benchmarks and I take a look at them almost every day. That's why I noticed the change and that's why I am bothering you by my frequent suggestions. It seems that benchmark addiction is spreading quickly here :D
[Nexus S General] :
Yes, well spotted. It's not something I expect everyone would notice!
[Nexus S General] :
UPDATE: Oodie pRiMe MIUI updated at dev request. Moved up a few places!