yashsingh1/xdaforums huggingface dataset:
[Nexus S General] :
Just a note on the new IO Scheduler Chart. I made a formula for comparing scores, but noticed the scores could change quite a lot for just a little change in one item of data. I have changed the way I worked it out, so it's more reliable. Some of the very close scores from the original chart have shifted around a little, but I think the ranking has remained essentially the same. Now follows technical detail, feel free to skip it!

Formula was geometric mean of all three mean scores over the ten runs, and all three standard deviations. I thought about this, and wondered whether the scores could be trusted. I decided to make just one database result up from 300 to 330 (10% increase) expecting a 1% increase in score. In fact, the score dropped by over 10% due to the standard deviation increasing. This is clearly wrong, one score shouldn't have a bigger impact on the magnitude of the overall score than the magnitude of the change in itself. I realised what I'd neglected to take into account was that the standard deviations are relative to the mean of the data they are calculated from. I have adjusted the formula for overall scores to accommodate this. Now, I take three standard deviations from the three means: database IO, read speed, and write speed. This is equivalent to finding the lower bound for 99.7% of all the results. I then calculate the geometric mean from these results.

I checked that this new formula remained accurate when an individual item of data changed, and it did, so I am happy the results can now be trusted.

I apologise if this caused anyone any confusion!

STILL TO DO: Do a partial comparison against a different kernel to see if the scheduler ranking matches for both kernels. (E.g. just check smartassv2 and all available io schedulers)
[Nexus S General] :
Okay, now I've had some sleep I'll try and address cspader's excellent questions properly.

I'm working on that right now. If you look at the I/O Scheduler Chart, and go way to the right, I'm re-testing Lazy, but using Steve Garon's kernel instead of Eugene's to see if the results come out in the same order. Results on that will be up later today.

No, the governor sheet measures CPU scores, RAM and Graphics in addition to I/O. The scheduler sheet only measures I/O. SmartassV2 is the best scoring overall when you take all these other measures into account.

I'm afraid I made a mistake on the scores in the I/O sheet (corrected now). Can you check this one again for me?
[Nexus S General] tchaari:
Hi bedalus,

Very good continuous benchmarks here :)

I just wanted to ask you about the conservative governor in the I/O schedulers comparison: is it the stock or the tweaked one?

cheers.
[Nexus S General] :
I was using an earlier test version of Speedy 6, which I believe did not have franco's tweaks.
[Nexus S General] tchaari:
Thanks that is better. Franco's conservative version is better performing but more hungry in battery. It only uses min frequency (700 Mhz) or max frequency (1200 Mhz) if I well understood Franco's explanation.
[Nexus S General] :
I think minimum frequency of 400 or 800 is more likely. There is no 700MHz step. I saw another report stating 800Mhz as the minimum.

I don't want to say anything in particular about battery saving until I do my study on that!
[Nexus S General] tchaari:
Yes thanks, you are right. The 700 and 1200 are for the GN :)
[Nexus S General] :
UPDATE: Study to see whether Scheduler/Governor combination results apply across other kernels.

Here is a comparison of the top three schedulers in both Steve Garon's kernel running Lazy and Eugene's running Lazy.

Lazy	Steve Garon's

1)	noop	35.96

2)	v(r)	34.01

3)	deadline	33.84

Lazy	Eugene's

1)	deadline	32.44

2)	v(r)	32.40

3)	noop	31.65

I'm not interested in CFQ SIO and BFQ because they had the same ranking for both kernels. But comparing Steve's to Eugene's, the ranking for v(r), deadline and noop did shift a little. If we round the figures, the results look like this:

Lazy	Steve Garon's

1)	noop	36

2)	v(r)+deadline	34

Lazy	Eugene's

1)	v(r)+deadline+noop	32

The reason for the generally higher scores in Steve's will be the OC. But while v(r) and deadline both rise by 2, noop rises by 4.

They are all very close scores anyway, but perhaps there is some reason why noop can benefit slightly more from the higher clock and bus speed. Or it might simply be that 10 runs is not enough to generate a completely accurate mean and standard deviation figure. If I retested the poorest noop result for Lazy on Eugene's and scored closer to the mean, that alone would be enough to improve it's overall score by the required 5%.

Summary: I'm satisfied that so long as you are using the top ranking I/O scheduler for your preferred governor, or one that gets a close score, you will get good results no matter what kernel you run.
[Nexus S General] :
UPDATE: Eugene373's Speedy 7 added!
[Nexus S General] kptnk:
First thank you Bedalus for your awesome work. Could it be possible to add this kernel in your study?

forum.oxygen.im/viewtopic.php?id=2160

you may know _thalamus's work, who's part of Oxygen team. At my opinion his hightly appreciated kernel has to be part of your benchmarks.

I know he doesn't post at all on xda, but i m sure his optimized stock kernel is very interesting for everybody and for the devs work especially.

This is just a suggestion of a greatful user though.
[Nexus S General] tchaari:
If the governors have the same code and the same parameters in the kernels, I don't see how they behave differently from a kernel to another :). I think that the comparison should be more fair if the clocks of the compared kernels were the same but it is a little bit surprising how Overclocking can influence I/O schedulers behavior :)

Nice findings man ;)

---------- Post added at 11:13 PM ---------- Previous post was at 11:09 PM ----------

Very nice suggestion. Thank you for this info :)

About oxygen, any news about the ICS ROM? :D
[Nexus S General] kptnk:
all i know is that Oxygen 3 will feature _thalamus 's kernel :-) ......(can't wait for it, but quite off topic here.....)
[Nexus S General] tchaari:
Ok merci bien and sorry for the off topic ;)
[Nexus S General] crabnebula:
This thread perfectly exemplifies (1) why I love science and (2) why the Internet is so great.

My gratitude for sharing your work and statistics bedalus.
[Nexus S General] :
It's a great suggestion and very much appreciated. I didn't know he had released for ICS. Will look into it!

EDIT: He's been in touch, and just to clarify, Thalamus' Kernel is an independent project, but Oxygen choose to integrate it.

For kernel benchmarks and more, see here: http://goo.gl/mpeHI
[Nexus S General] :
Copied from my inbox:

Thanks, I understand now.

These figures:

steveg/smartassv2/cfq(i/o only) -7%

steveg/lazy/cfq (i/o only) -16%

steveg/conservative/cfq(i/o only) -17%

...are difficult to compare to Eugene's.

The Governor results (using Steve's) are formed from an average I/O score from AnTuTu, Vellamo and SQL benchmarks, run twice.

The Scheduler results (using Eugene's) are formed just from AnTuTu, but then penalised by their variability, calculated over ten runs.

But I do think you have a point. I will need to do some extra maths to analyse this before I embark on further benchmarks! I will get back to you. I'm also going to copy this into the thread so other people can read it. (If that's not okay, please let me know and I'll remove it.)

EDIT:

Okay I've looked at the results for cfq using just conservative and smartass for both kernels, and I've used the variability to see if the results can be made closer by imagining they had the worst possible luck during the tests.

Eugene's kernel (with cfq)

SmartassV2 ~93% reliability boosts ~31 to ~33

Conservative ~98% reliability ~37 stays ~37

Steve's kernel (with cfq)

SmartassV2 -7% made worse with variability could be -9% (below the mean)

Conservative -17% made better with variability could be -15% (below the mean)

So yes, even if I massage the figures, it still looks obvious that cfq gets different results on these kernels. I still need to think more about this.
[Nexus S General] :
UPDATE: I/O Scheduler sheet now has reliability scores for each combo.

UPDATE: Franco's kernel added

UPDATE: Thalamus kernel added (thanks kptnk, Thalamus)
[Nexus S General] :
I decided it's too difficult to compare results that are calculated differently, so I went into Governor Spreadsheet and if you look from cell R78 you'll see I've added something.

I have worked out I/O scores from data produced using Steve's kernel in the same way that I work out all my I/O scores in the Scheduler Spreadsheet, when I used Eugene's. Here's the results (~rounded to the nearest whole)

Steve (CFQ)

Lazy - 12

Conservative - 21

SmartassV2 - 15

Eugene (CFQ)

Lazy - 31

Conservative - 37

SmartassV2 - 31

The reason Steve's scores are low is because they are more heavily penalised for variability, due to the fact the standard deviation is bigger (caused by the lack of passes: Only two passes for Steve's, compared to ten passes when I tested using Eugene's. More passes gives a more reliable standard deviation).

This will of course mean that the results above are less reliable, but they do now indicate a rough match in ranking compared to the full study I did with Eugene's kernel.

I think the only way to be 100% certain about the governor/IO scheduler pairing would be to repeat the study in full on another kernel. I will consider doing this with Air Kernel. It lacks BFQ (who cares? It performs so badly) and V(r) but the scores ought to come out about the same for the other four schedulers. But at fifteen minutes per scheduler per governor, this is a commitment of at least six hours. I'm wavering on whether I can manage it. Brainmaster told me to be careful not to burn out!

Edit: What I've decided to do, in the interests of saving time, is run SQLite to test just database performance using Steve Garon's kernel which has the fsync tweak to speed things up. I'll do five runs instead of ten, because there's less of an issue with SQLite producing variable results.

Another Edit: After testing all the schedulers (except bfq) on ondemand, performance, and conservative, the results were all coming out at 35s for SQLite, so if there's no effect, I can't produce a result. I suppose I will have to go back to my original plan using air kernel.

Yet another Edit: I've retested using Air Kernel. The results don't come out exactly the same, so I'm still confused as to the reason for different performance. I've seen posts about deadline being 'tweaked' for flash drives, but perhaps this means that other schedulers are tweaked too, and maybe they vary between kernels? Or perhaps there is some other reason for different combinations performing differently in different kernels. Need to think more.
[Nexus S General] Zorrich:
BLN and Voodoo Sound are both in his 1.2 kernel.
[Nexus S General] :
Thanks, I forgot about that. So did Franco I guess! I'll correct it now.

For kernel benchmarks and more, see here: http://goo.gl/mpeHI
[Nexus S General] :
UPDATE: I/O Schedulers Chart

Added Further results but using Air Kernel this time. I wanted to know if the I/O schedulers and governors combinations performed the same way with a different kernel. They are not quite the same, but it is easy enough to see which schedulers perform well for both kernels, so I have done an OVERALL ranking for 1st and 2nd place for each scheduler (within each governor).
[Nexus S General] phone_user:
The official thread of Air kernel opened

http://xdaforums.com/showthread.php?t=1469842
[Nexus S General] :
UPDATE: Air Kernel 2.5
[Nexus S General] mutantlx:
Thankyou for doing these :)
[Nexus S General] :
Not a problem! :D I was doing them for myself, and just thought, hey, why not share?

I won't be able to benchmark new kernels or updates for a few days, for the following reason... (Read next post...)
[Nexus S General] :
COMING SOON: Battery Benchmark

There's no acceptable app for this, so risked murdering my phone to test it the old fashioned way. See this VIDEO.

Test Details

Phone Preparation: Start up the phone, set everything to off position: GPS, sync, aeroplane mode active, no wifi, sound will be muted, nfc, auto brightness, everything! Set up the music app to play Time (Pink Floyd) on an infinite loop. Select a governor.

Metering Preparation: Crocodile clips will allow me to insert the amp meter directly into the battery circuit. The battery is suspended in my daughter's toy xylophone (see the video!). The amp meter is limited to a maximum of 250mA. The phone draws more than this at times, so I hook in the usb charger, and 3 AA batteries (rechargables in series, about 4 Volts) in parallel to Samsung battery, but not part of the amp meter circuit. This means the amps are drawn from both sets of batteries, which halves the amps drawn through the meter, preventing it from maxing out. (Maxing out causes the phone to power off. A better solution would be a higher rated amp meter.)

F.A.Q.s

(Q) How can you have a frequently asked question before you have even finished writing this post?

(A) I think it has something to do with playing Time by Pink Floyd on an infinite loop, that allows thoughts and ideas to find me from the future.

(Q) Yes, but why on mute?

(A) I'm attempting to eliminate what power the speaker is consuming. This will increase the run time of the test, so if there are differences between the governors, they will be more noticeable. If you watch the video, you can see how the amps increases every time there is a loud passage of music. With the sound off, all I'm measuring is processors amps.

(Q) What if a kernel updates, how will you benchmark it?

(A) I'm not going to be doing that until this test is complete.

(Q) What if someone is trying to call you?

(A) Good point. I'm going to be in hot water with the mother of my child.

(Q) What if someone has a serious question?

(A) Post it, I may add the response here.

Thanks everyone!
[Nexus S General] simms22:
try the battery monitor widget app from the market, lots of useful info in it https://market.android.com/details?id=ccc71.bmw
[Nexus S General] :
Hi Semeon! I know battery monitor widget (great free app!). That's the one I'm using to time the battery drain. I'm going to check the graph for any weird blips. Hopefully it'll just show a nice, long, smooth line all the way down.

The only reason that I'm not using it to measure mA/h is because the readings aren't accurate enough to compare the impact of the governor. I'm aiming to put the great "race-to-idle, truth or myth" issue to bed.

I can't believe I'm doing this. I'm essentially committing myself to a phoneless existence for what might be a couple of weeks!
[Nexus S General] simms22:
wussup!

dude, i soooooo feel your pain :eek:
[Nexus S General] tchaari:
This is the most important benchmark I think ;)

Wow, this is really long. Good luck man :D

Seriously, I don't think that you have to wait all this time for every governor test. I think that 6 hours test after full charge (with a check each 2 hours) will be enough.

I don't doubt that you love Pink Floyd songs but hearing the same music in a loop is not really pleasant. That is the main reason I think :D:D:D

---------- Post added at 10:34 PM ---------- Previous post was at 10:25 PM ----------

The only risk is that this app can significantly participate in battery draining if it is continuously running in background. Anyway, even with this risk, it would be fair enough because this app will be the same on each governor :)
[Nexus S General] :
Haha, I used to work as a sound engineer, listening to loops of a few seconds for hours and hours... I could probably take it, but I don't want to damage my daughter's mind! And I don't want my partner to defenestrate my phone!

Perhaps a few hours is all it takes to measure the drain on the battery. But I can't use an app to do it. The reading is not accurate enough. I have some equipment I can use. You have given me another idea, so thanks again tchaari!
[Nexus S General] :
Yes, the same testing conditions for each governor, so no unfair drain.

The battery monitor widget app can be set for different measuring intervals. From 60 seconds to something like every 4 hours. I have it set to take a reading every 10 minutes, which is not a big impact. The main processor activity is decoding the mp3 data, which will be the main load under which I am assessing governor activity.
[Nexus S General] tchaari:
I see, you are taking some risks with this new benchmark :D:D:D:D:D

You are welcome bedalus and as you already know, if I can help don't hesitate to pm me ;)

PS: I almost forgot bedalus, I am wondering about testing battery drain for the same governor (maybe the default one "ondemand") in different kernels... I personally think that this test will be (practically) even more significant for end users but the governors battery benchmark is more "scientifically" interesting ;)
[Nexus S General] :
I'll be investigating this after governors are done.
[Nexus S General] tchaari:
Take your time dear bedalus. Nothing urgent for this. Just try to take some rest when doing the long governors battery comparison ;)

I forgot to ask you about what kernel are you using for the governors test? I hope that it contains the stock conservative governor and not the tweaked one. The latter is not very battery friendly :)
[Nexus S General] :
I'm going to be using Air Kernel 2.2
[Nexus S General] cspader:
why the hell is this unstickied :mad::mad::mad:

mr b., i think the way you present all benchmarks is spot on

cant wait for the battery benchmarks
[Nexus S General] :
Yeah, that is a little curious. Maybe it's because AllGamer is going to move it back to development? Or maybe it's because it's going to be a little out of date while I do this battery study?

Check out my first post, I'm going to link to a video where I demonstrate how I am going to test out governor impact on power drain. (Embarrassing for me, but informative I hope!)
[Nexus S General] :
UPDATE: Battery Benchmark video! See first post for the link!
[Nexus S General] simms22:
morfic forwarded the video to me.. cute, lol. pink floyd was the way to go :)

what i wanted to know was if you found out why this thread was unstickied. anyways, it looks like they moved around a whole bunch of threads the past day or so. like, why is cwm touch recovery now in apps and theme, lol.
[Nexus S General] :
Nope. I've complained of course! What a big whinger. :o Still, I think I make a good case for reclassification as R&D, especially when I publish the astonishing battery results tomorrow... ;)
[Nexus S General] thegtfusion:
Oh you've got to be kidding me Allgamer:mad: !! (no offense :))

(why the unstickied?)

This is one of the MOST useful threads in general section!!
[Nexus S General] steve.garon:
Interesting video. Can't wait to see the results.

I was actually doing a lot of tests on my kernel to get the best battery life as possible. I was using the battery widget monitor comparing the mah drop on idle...

I get the best results using smartass with deadline IO sched.

I'm not sure what kind of test your are gonna be doing but this is hopefully gonna be really useful.
[Nexus S General] cspader:
hey bedalus, have you considered developing an app similar to quadrant benchmark or whatever that would give you another way to implement all the calculations and comparisons you did... i don't even remotely have a clue of how but maybe you do.. just an empty suggestion.. maybe it would keep score when you run it on a kernel then when you run it on another kernel it would compare for that particular phone.. don't even know if thats even possible.. might be interesting if you are into developing apps
[Nexus S General] :
I wish I could, but the last time I did code was a decade ago and things have changed a lot! I was accepted (but never enrolled) for a computer science degree at The University of Manchester, before software engineering courses even existed. I programmed in Turbo Pascal. I did one year of a maths degree, then switched to music production (I ended up working as a sound engineer in Hollywood/Burbank/Pasadena and the Melrose Media District, probably on the strength of my British accent ;)). When I moved back to the UK I did another crash course in Maths, loved it and then my teaching qualification. If I had the money, I'd probably quit that and try to get back into programming.
[Nexus S General] falconrulz:
I should have commented earlier, thanks a lot for all the work you've done, really helpful :D

And a Basic app with all your data would be amazing... Get some help from your fellow XDA people for making it!

(Even I can help, but I don't know too much :( I know the basics... I think... :cool: )
[Nexus S General] :
Hi Steve. I did every test I could imagine. Then the mother of my child got angry, so I had to stop. :eek:

I'm just working on my spreadsheet. I've had to re-learn some basic formulas, converting dBu to mW then from that to mA. I should have it up today.

BTW I may have ****ed up my phone or my battery! DO NOT TRY THIS AT HOME!

EDIT: Actually USB power seems to revive the phone, battery may just need a cycle. Wall charger may be ruined though. Oops!
[Nexus S General] :
UPDATE: Matr1x v15.5 and Air Kernel (Kernel Air?) 2.6 added!

MAJOR UPDATE: Battery benchmarks now available! See first post for the link! Feedback is essential!
[Nexus S General] bendirkss:
Thank you sir

:)
[Nexus S General] aLNG:
wOw~~~~

really really good discovery here!!!

it proves somehow my personal testing with gov :)

1 suggestion: how about this time test something with heavy usage

for example running "stability test" app

thank you very much with your good effort. really helping me.

edit: btw what do u mean by lulzactive2 min?
[Nexus S General] :
With music playing (a completely silent mp3 - just to keep the processor awake) and the screen off, lulzactiveV2 jumps around a lot in the power drain. It is spending about 50%/50% time at those minimum and maximum drain levels.

A value in between the two probably gives a good estimate.
[Nexus S General] grifforama:
So, if I read this right, it would appear that deep idle does nothing ?

I'm inferring that Matrix 15 ondemand battery ticks at 9.8 and deep idle 9.8, or is this because you were playing an .mp3 at the same time ?
[Nexus S General] :
Deep idle is supposed to have an effect while the processor is active, for instance, when listening to music. Clearly this is not the case. Deep idle appears to do nothing, and worse when screen off max frequency is set.

For kernel benchmarks and more, see here: http://goo.gl/mpeHI
[Nexus S General] jcolinzheng:
Are you sure Deep Idle is working properly (as reported by Deep Idle stats)? For many ROMs out there, there's a bluetooth fix that has to be applied (can be found in Ezekeel's original post).
[Nexus S General] nodstuff:
I agree completely! i don't think it's fully compatible with ICS kernel. maybe if "someone" ;) tried the same battery test on a gb rom and kernel with deep idle to see if deep idle works on gb and if it does then we can see that it doesn't work with ICS.

---------- Post added at 06:54 PM ---------- Previous post was at 06:53 PM ----------

I'm pretty sure bedalus thought of that! ;)
[Nexus S General] :
Idle stats reported as expected.

For kernel benchmarks and more, see here: http://goo.gl/mpeHI
[Nexus S General] simms22:
wonderful work.. extra credit earned ;)
[Nexus S General] jcolinzheng:
Thanks! This is a great job, most important benchmark result so far!
[Nexus S General] AeroEchelon:
It appears, Mathk1d's kernel has the least screen drain when screen is on with min brightness. And what I gather is that DEEP_IDLE is:

Either:

1.) Not working.

2.) Not saving power.

However this seems to contradict Ezekeel's find: http://xdaforums.com/showthread.php?t=1297535&highlight=deep+idle

In that thread he find's DEEP IDLE to save 17-18% more power than plain IDLE.

Interesting.