yashsingh1/xdaforums huggingface dataset:
[Nexus S General] tchaari:
Bedalus removed the stock kernels benchmarks to have fair scores. Benchmarking the ROMS with their stock kernels remains a very good idea but it should be better to keep their scores away and sperated from the current results. If bedalus has some spare time, he can do it and revive some kernel comparison after closing the kernel's thread :p

Wow, poll is already started. That's so quick lol

EDIT: I voted for both. I agree with Glenn. Stability is the first and last concern but many users flash other kernels too. Let's take two examples: Matrix is always cooked in Nexus MV but it is giving me some slight instabilities some times (my phone doesn't like mathrix so much lol). Second example, Slim ICS is given with Eugenee's kernel which wasn't updated since several months and users look for some more fresh kernels.
[Nexus S General] ljordan2:
Voted for continue benchmarking with T132

What I think is that it would be unfair for roms that uses the stock kernel because roms that have custom kernel will perform better than roms with stock kernel regardless of rom's performance. Just my 2¢.
[Nexus S General] thegtfusion:
Same here.

hey where did you get the T132 meauchassen (or however you write it)
[Nexus S General] ljordan2:
It's an test version of Trinity from few weeks ago. I don't use it anymore but I'm just too lazy to update my sig:)
[Nexus S General] bluelyd:
"Score of ICS Roms + T132" will be more helpful result for users, I think.

Maybe most of xda users use their favorite kernel, whatever they flash (rom).:D
[Nexus S General] thegtfusion:
Aha okay. but is there still a trinity kernel with 405 MB ram?
[Nexus S General] glennkaonang:
I see that the majority here agrees with T132 :D

I have another idea regarding the third option, though.

If possible, benching the ROM before flashing T132 would be great.

But remember that your work on this will double, bedalus :P

I think the result of benchmarks with the supplied kernel could be listed in another sheet so users can choose to see which they're interested in.

The best thing is actually let the poll continue and see the results :)
[Nexus S General] ljordan2:
I believe only muenchhausen kernel was the only kernel to have 405MB ram. If you want to try it, check this page out : http://morfic.euroskank.com/galinka/

This has all the old Trinity test kernels. Just choose T132-Muenchhausen.
[Nexus S General] thegtfusion:
thanks man :d
[Nexus S General] Oodie:
Voted for both . I think both should be tested & get a percentage out of it .

With trinity - test performance equal to every rom

With stock - test performance of the rom untouched ( most users use it this way )
[Nexus S General] Ibn Saeed:
Summary of Results need to be updated to reflect the latest rankings.
[Nexus S General] :
Thanks I'll do it now

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] :
I think that unless a very clear margin appears in the poll, I'll carry on as I have been. We'll give it a week or two to accumulate first.
[Nexus S General] RicardoOneri:
Bedabus, CyberGr CyberGR-MOD Huricanne v10 is out, when you can bench it please :D
[Nexus S General] :
Coming up today,

CyberGR MOD v10

Stock 4.0.4 (If I can get it to work)

and AOSP+
[Nexus S General] Maximilian Mary:
I think the most objective results would be produced by using the same kernel for all roms. Then you're not left wondering whether the performance is due to the rom itself, or its kernel. Though I have the deepest respect for morfic, I question whether T132 would be the best choice for this. The lastest Trinity kernels include their own ramdisk, rather than using anykernel to preserve the rom's ramdisk, which might lead to compatibility issues with some roms and result in an artificially low score. Likewise, flashing a different kernel without first clearing /etc/init.d/ could skew results in some cases because of conflicting startup scripts.

That being said, it would be insightful to see how a rom performed as the dev intended it to be used. If you did both, you'd essentially be resuming the kernel benchmarks, but using only built-in kernels rather than using only standalone kernels. The difference between the reference kernel and the built-in kernel would be its score.
[Nexus S General] :
I see what you are saying. About the ramdisk though, I chose this because it seemed to improve the scores (based on intuition, not a proper study). The only scores that seem to come out low are Cyanogenmod's, but that itself is strange as the ramdisk is taken from Cyanogenmod's kernel. Kalim said he would look into it.

I agree about how this would be a covert way of resuming kernel benchmarks. However, as I didn't find any really obvious differences between the performance of the kernels in any tests (apart from a slightly above average CPU performance for matr1x) I think the scores in both sets of benchmarks should be roughly equivalent (differences would have to be larger than the variability of the tests to be significant).

EDIT: I forgot to mention, there are no init.d conflicts, because I've removed the init.d tweaks from morfic's kernel. (Some ROMs provide their own init.d tweaks).
[Nexus S General] :
UPDATE: CyberGR-MOD ... v10 HURRICANE benched!

UPDATE: AOSP+ benched!

Will try to do stock now.

EDIT: Stock 4.0.4 installed no problem, so results up soon.
[Nexus S General] rrohanjs:
0.04 % difference between 1 & 2 ... niceeeeeeeeee ... :)
[Nexus S General] :
I think that we might start to see only very small differences. This is probably a good indication that things are getting properly optimised.
[Nexus S General] :
UPDATE: Stock Google ICS 4.0.4 benched!

Moves up to 3rd place.

This result is interesting. The stock ROM is within about 0.5% of the highest performing ROM (and that could be accounted for by the variability of the test).

Are the various performance tweaks and optimisations in custom ROMs really making any difference to performance? Where might those differences be if they can't be measured?
[Nexus S General] SirPrick:
I bet its partly placebo!
[Nexus S General] Harbb:
That spreadsheet looks better each time i look at it bedalus. Nice work as always.

Using Trinity is likely the most effective way of gauging, synthetically, what ROM tweaks do with regard to performance. Testing the ROMs kernel does little other than satisfy people's curiosity, assuming we are looking for the most efficient ROM and not the best ROM OOTB. The limit is, as always, that it's not very good at determining which one is best for any other situation nor other sorts of effects.

If anyone here has read about or played around with a Dual-GPU setup on your computer you might have come across microstuttering. Obviously this is not the same process which could occur on our specific phones but certain effects may certainly exist, whether it be OS/feature/mod-wise, which hinder in a similar fashion. We can see a bottleneck straight away by looking at our RAM and the huge impact it has on general usability in ICS when we up that to ~400mb, so anything which may play around with the RAM at the ROM or kernel level could result in a huge impact on overall performance which may have little to nothing to do with a benchmark.

I'd say that many people simply succumb to placebos, but this certainly doesn't mean that minor tweaks (or numerous tweaks) can't change large parts of the user experience, whether it be good or bad. That said, i still believe there is a very large amount in android which can be better optimised and controlled, possibly by the end user, but very little of this would be worth an independent dev attempting it considering the rate of technology moving forward. I've been seeing some of this being rolled out recently through both google and devs though, which is very nice.
[Nexus S General] :
thanks Harrb. Clear summary.

...and we can see from morfic's recent work, that RAM is high on his priorities. He's taken very slow and careful steps forward with his kernels and his first move after adding BLN/BLD has been the RAM.

I'm quite interested to see what the next batch of test releases Simms22 will unleash will bring.

Just for people who only look at the last page of a thread, I'll restate my question: If stock 4.0.4 performs within 0.5% of the highest custom ROM, what exactly do the numerous performance tweaks do?
[Nexus S General] rrohanjs:
what i feel is, if a developer puts all his tweaks and features and goodies which u dont get in stock and is still within +- 1% of stock rom then its a very good rom .. becaue google being so huge they should be a benchmark so if a rom is so close then its good after incoperating all the mods n features.

i just said the opposite way, ur comparing stock to customs roms, im comparing custom to stock :)

---------- Post added at 03:39 PM ---------- Previous post was at 03:37 PM ----------

btw new rom :)

http://xdaforums.com/showthread.php?t=1579422

its based on kongs miui only so see the changelog if its necessary to benchmark it or no
[Nexus S General] tchaari:
Well, maybe not noticeable in the benchmarks but they can have a great impact on user experience. Let's take the example of the launcher lag. It can't be always seen in benchmarks but when you use the phone you will easily notice it. A tweak that changes how the memory is allocated can be another example that can be seen in user experience and not in benchmarks.

All depends on how the benchmark tools have been developed and what they measure ;)

EDIT1: It would be nice to have a tool that measures user experience accurately (good research topic lol)

EDIT2: Nice summary by Harbb by the way but placebos can some times become solid science (that's what I learned in my research activities lol)
[Nexus S General] Oodie:
Are you using the latest Trinity kernels . I've read somewhere that they don't support init.d .

If you are using them , i think that's the reason that stock is almost identical to tweaked roms . Cause the tweaks are not working .

And i hope that you are using a trinity version zip without any system folders other than modules . Or else the rom tweaks will be replaced with trinity tweaks and the scores would be useless , without the tweaks comes with the rom .

Just suggestions ok . No offence :)
[Nexus S General] tchaari:
Thanks Oodie, I forgot about that. I always believed that not all kernels go with all ROMs. They should be complementary and compatible and that's the reason why I voted for an extra benchmark with the stock kernel.

@bedalus, to check what Oodie proposed, what about redoing the benchmark of pixel rom and stock rom using another recent kernel supporting init.d? (any suggestion Oodie?).
[Nexus S General] Harbb:
Might be wise to throw in a test script to check if init.d scripts are working on boot (ie, echo OK > /test).

Also, init.d scripts are just code executed on boot once in android. They are not ROM specific, just about all well known ones will work with any properly busy boxed ROM. Results of course vary with every other part of the system.

Tchaari, a placebo can certainly effect user experience but it wouldn't work in a blind test, much like comparing a coat hanger with monster cables in a hi-fi system. Slightly different to the sugar pill analogy where there is, likely, an objective difference and not a subjective one. Enough off topic talk for now :)
[Nexus S General] tchaari:
Lol, I like your hi-fi cables and coat hanger comparison :D

No offense buddy, I've just wanted to say that there is a too short time since 4.0.4 release and things aren't yet settled down. Therefore, we can still find some incompatibilities between some ROM tweaks and kernels. Besides, I highly encourage any RAM tweak (even if it seems useless) for devices having few memory for ICS like the Nexus S. These tweaks can have a great impact in phone responsiveness but not sure in benchmarks.
[Nexus S General] :
I think that's on-topic actually Harbb... Also, good thinking with the init.d test.
[Nexus S General] Oodie:
& also init must be activated from ramdisk also . Stock Miui ramdisk still doesn't support init.d ( as far as i know ) , so the ramdisk should be edited . Arguably , i think stock ics won't be much different . I think in stock ICS , init.d haven't been activated from the ramdisk ( again , as i think )

But coming again to trinity , I'm not certain that they fail with init.d . Just something i read from somewhere & older versions work well .

So , if latest versions have issues , let's use the older versions :D

Edit : Found out who told that they are not working . Its Kong . Let's kill him :D . lol
[Nexus S General] :
There's nothing in my init.d T132, I took it out to avoid conflicts.

I've just downloaded kong's version of T132 (he tweaked the cm ramdisk to support init.d)
[Nexus S General] Harbb:
Just remember, not all tweaks are beneficial. It works both ways, a shoddy tweak could initiate a bottleneck or other hindrance just as well as a performance boost. The new bigmem for example is great, but without heavy tweaking of certain apps and Android you lose the ability to record and play videos. This is an obvious con, there are many situations that are not quite so obvious.

I think there is one guy offering $1mil to anyone who can prove a difference between the two cables in a blind test. No success thus far. Good read.

Trial and error can answer all our questions ;)
[Nexus S General] :
here's my init.d script to test if it's working

#! /system/bin/sh
touch /sdcard/test.txt
echo "1" > /sdcard/test.txt

Does not produce a text file with a 1 in it. Not working?

Did I miss something obvious in my script or does this prove that init.d is not working? Do I need to wait for the next Trinity with init.d fixed, and then retest everything?

EDIT: Entering this...

touch /sdcard/test.txt
echo "1" > /sdcard/test.txt

...into terminal emulator has the desired effect. I assume init.d is not working in kong's version either, unless I made a mistake that I'm not understanding.
[Nexus S General] rasengan82:
I'v tried the stock 4.0.4 leaked for NS4G, and it's super smooth and snappy compared to many ROMs.
[Nexus S General] :
@tchaari

This gives me an idea for a "user experience" benchmark. I just need to export every ROM thread to a text file, and calculate the density of superlatives.

Like rasengan82 is saying here, his preferred ROM is "snappy" or "smooth" etc.

The ROM with the highest superlative density wins!
[Nexus S General] tchaari:
I think that the script should be given the right permissions using chown and chmod to be executed. I am not an expert in such thing but you can try a chmod 777 on your script to give it all the permissions if necessary.

By the way, why don't you try with an old trinity kernel that supports init.d?
[Nexus S General] Harbb:
Why didn't i think about finding the superlative densities! It's all so simple now.

I'll mess around with the init.d dilemma and let you know how i go. Thinking it might be trying to execute before the sdcard gets properly mounted, might be wise attempting to throw it in /data or something.
[Nexus S General] tchaari:
:D:D:D:D:D:D very funny bedalus :)

More seriously, user experience won't be by anyway as objective as scientific benchmarks but there is already a research domain called "Quality Of Experience"

http://en.wikipedia.org/wiki/Quality_of_experience
[Nexus S General] Oodie:
ha haa . Just when i was going to type it . lol . I also think txt should go to data or something , because with script manager script works fine . but device it all booted up when we try it with script manager . :rolleyes:

try this instead :rolleyes:

Or you can always test init.d when booting up using a logcat viewer . I always insert a banner in init.d folder & check the logcat to see if the banner content is showing @ boot . That's how i knew that init.ds are working properly in my rom
[Nexus S General] :
Good thinking again. I'll try write it to / as you first suggested.

Edit: no luck, didn't write.
[Nexus S General] :
and I'll try this too. Thanks for all the help everyone!
[Nexus S General] Harbb:
/ , /data, etc may also need to be mounted r/w before this might work. I believe /data/local/tmp seems to work. On a Stock Slim 3.1 (ie, krarvind building off Speedy kernel source i believe) i couldn't get init.d working. I tested this by making the same script and placing one in the init.d folder and a slightly modified script as the /system/etc/install-recovery.sh file (which is deleted in Slim by default). init.d would create a "test" file, install-recovery.sh would create a "test1" file. I only found test1 file in the /data/local/tmp folder, so init.d isn't working by default. Also, i made sure they were both executable by all users to save me any possible effort later. Script is below:

#!/system/bin/sh
touch /data/local/tmp/test

Touch isn't needed if you use echo, btw. It will create the file if it doesn't exist. Or you could just use touch itself as the identifier with no need for echo and so on.
[Nexus S General] :
Oodie, this worked!

#!/system/bin/sh
touch /data/local/tmp/init.d_log_test.txt
echo "done" >> /data/local/tmp/init.d_log_test.txt

So, now I just have to redo all the benchmarks I've done over the last few days! Extra benchmarking for me!
[Nexus S General] :
Ah, noted.

Oodie's idea worked. I ended up with the file in /data/local/tmp. I'm satisfied this means init.d works with kong's modified T132.

But will Slim ICS 3.1 init.d work with this T132? (Sorry I didn't get whether it was a krarvind issue or a Eugene issue)
[Nexus S General] Oodie:
why do you have to redo all ? did you check that the kernel you used doesn't support init.d ? maybe its working :rolleyes:

yep . Init.ds will work in any rom if you use the whole boot.img ( zimage + init enabled ramdisk ) with init.d enabled , If there are init.d folder in the rom . lol
[Nexus S General] :
Good point, better check that first. (I quote kong here: http://xdaforums.com/showpost.php?p=24390132&postcount=9890)
[Nexus S General] Harbb:
I don't know whether it's inherent to Speedy or something krarvind changed, i'd give it a test but it seems you have it sorted anyway and i don't need init.d scripts. I just assumed it would be working since there is an option to enable startup scripts buried deep within the ASS :rolleyes: (i tried to resist.. i failed)

Just now after trying to enable it there seems to be a problem though, so maybe it wasn't expected to. As for trinity (old or new) i have no clue if it'd work. If you want me to give it a shot let me know ;)
[Nexus S General] :
krarvind is a comedy genius for that menu :D
[Nexus S General] tchaari:
So bedalus, you will replace boot.img in every ROM before benchmarking it with the tweaked trinity kernel by Kong?
[Nexus S General] :
Yeah, I checked it, kong was right, the support for init.d was broken in the original T132 from 23 March.

I'll use kong's version and retest all the ROMs I've done so far.
[Nexus S General] Harbb:
So he gets an extra density point for the extra CO2 we all laugh out i'm assuming? This might put him over the 0.5% edge. Easily deserves it if you ask me.
[Nexus S General] :
Not exactly, the kernel zip includes the boot.img

No need for me to do anything except start all over again!
[Nexus S General] tchaari:
Do you think that this can affect the scores?
[Nexus S General] Oodie:
:D Let's kill Kong instead . That's easier :rolleyes:

He's the one who figured this out ...
[Nexus S General] :
HA! Ultra-violence. I like it. :D
[Nexus S General] glennkaonang:
And by starting all over again, you would get: quicker phone aging, abandoned C++ learning, and more wife's hate to your already hated phone.

Just a joke, man :D

I was wondering the same as tchaari here, will getting the scripts working well actually affect benchmark scores?

I thought they have someting to do more with user experience, though.
[Nexus S General] Oodie:
It should be , accoring to The Book :rolleyes: . Or all the chatting about "tweaked" would be useless . ( As i think )
[Nexus S General] :
Exactly. How many ROMs do you see with "PERFORMANCE TWEAK X;Y; and not forgetting Z!"? Those tweaks better do something!
[Nexus S General] :
I put C++ on the back burner for a few days while I read a book on E=mc^2

I have two weeks break right now also, so lots of free time.
[Nexus S General] glennkaonang:
Bedalus, I don't know if I am right about this, but AFAIK Pete's Bugless Beast wouldn't get init.d scripts working properly unless you re-installed Busybox using the installer from the Play Store.

I got this from my experience of flashing Brainmaster's tweaks over Bugless Beast, and apparently the tweaks weren't working well (as seen from dalvik-cache not moved to /cache and the unmodified build.prop).

But after I re-installed Busybox using the installer, the tweaks were working properly.

As an additional info, in many ROMs other than BB, I saw "Busybox is installed to /system/xbin" in Busybox app by Stericson.

In BB instead, I saw "Location of Busybox couldn't be determined" if I didn't reinstall it using the app.

I think this is the reason why I had to reinstall Busybox in BB.
[Nexus S General] :
Does Pete provide his own init.d tweaks?
[Nexus S General] glennkaonang:
No, he doesn't.

I've checked this right after clean wipe and flashing the ROM, using Root Explorer app I didn't see init.d folder inside /etc.

Perhaps you could check first later when you're re-benching Pete's whether the scripts work or not.
[Nexus S General] :
So the correct formula is

Individual ROM UX mean % = (Superlative Density + Hilarity Factor ) / Global mean.

I'll do a feasibility study. May have to factor in LOLs.
[Nexus S General] :
His score won't have been affected be lack of init.d support since he has no init.d

You can flash kong's kernel if you like to check if init.d is restored. Kong's T132 with fixed init.d: http://www.mediafire.com/?bbub5x9cai9xr9c
[Nexus S General] Oodie:
Don't forget . Using same ramdisk in every rom will cause issues . It will boot fine most probably , but will give fcs , camera not working issues , theming issues ( i got when i used a different ramdisk with MIUI ) . etc etc .

But it won't affect benchmarking .. So just do your thing bedalus :D ..

Or just use a anykernel version . So the original ramdisk would be intact . & then it will be fair with both init.d enabled roms & one's that are not .

Edit : Oh ( sh*t ) dear !! why didn't i come up with this solution before . Use anykernel version , so you won't lose ramdisk settings .
[Nexus S General] :
I think the current CM9 ramdisk is compatible with all ROMS (?)

I flashed it with the recent 4.0.4 stuffed crust and had no fcs or any other issues. Have you tried it?

EDIT: Oops. Posted something about a Trinity kernel in post number #666. Isn't that a bit like the planets aligning? Things may happen :eek:
[Nexus S General] Oodie:
TRINITY :eek: lol .. May be the End Of The World :rolleyes: :D

Yeah ! there should be no issues . Just test out the results again & see if the scores are changed :rolleyes:
[Nexus S General] RicardoOneri:
He's allready testing all at this moment, and results changing , this will become interesting :D
[Nexus S General] simms22:
it is compatible with all roms at the moment. Trinity just merged the code from 4.0.4 and is in testing.
[Nexus S General] :
Okay, I checked all the ROMs, the only one's I needed to re-test were Pixel ROM and CyberGR-MOD v10

(the only ones with init.d scripts included)

CyberGR-MOD v10 went up to first place,

Wildest Pixel dropped about 4% due to SD Card Write speed dropping about 1 MB/s
[Nexus S General] tchaari:
oh ****, changes again :D

Well, thank you bedalus for all these efforts to improve the scores.

About Pixel ROM, do you mean that the modified trinity kernel by kong causes SD card write speed issues?
[Nexus S General] tycruickshank:
Does Dario's ICS have init.d scripts as well? I looked in /etc/init.d and there seems to be some startup scripts. Some are from air kernel but some seem to be specific to the from itself.