yashsingh1/xdaforums huggingface dataset:
[Nexus S General] Oodie:
;) That's what i said. Now tweaks are in full effect ( or bad effect in some cases )

Too many tweaks would cause issues most of the time .

Another question :rolleyes: Will the Devs gonna kill me now for what i did :(

I think the tweaks are causing those , not the kernel
[Nexus S General] wildestpixel:
Time to fix the tweaks ;)
[Nexus S General] kong:
Mine also has init.d script T_T

Lol, don't bother... I don't think the results will change anyway :D
[Nexus S General] RicardoOneri:
I think he will do it anyway to all :D
[Nexus S General] Tordp:
Are you sure they're the only one with init.d scripts? I use Brainmaster's 4.0.4 and it has 4 scripts in init.d by default.
[Nexus S General] :
Yes. I forgot to check that one!

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] :
Yes. I forgot that one too!!

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] :
Damn, I actually remembered yours. But then I had to give my daughter her lunch and got sidetracked. I'll redo all three ROMs asap!

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] :
Every single time, if there's a chance for me to **** something up, I do!

kernels ; battery ; ROM ; gov/sched

Now with summaries in the first posts. Convenient for XDA app users!
[Nexus S General] glennkaonang:
If you haven't known, an update for Bugless Beast has just released, though.

It includes Pete's own kernel (if you're going to benchmark it both ways, which I think you'll do based on the poll results :D :D)
[Nexus S General] :
Again, dario's results drop due to tweaks. Bad news for certain tweaks.
[Nexus S General] :
As others have said, it's the tweaks that are the problem
[Nexus S General] :
Think I will leave Stuffed Crust! if kong is putting out a new one in the next few days. I'll check with him. Brainmaster's ROM will have to wait until the morning.
[Nexus S General] tchaari:
I am wondering which are the tweaks that can decrease SD card write speeds :confused:

bedalus, if you have some spare time, can you please redo the benchmark of the stock 4.0.4 with the new kernel? (just to be sure about the tweaks)
[Nexus S General] :
Sounds like a good idea.
[Nexus S General] tchaari:
thank you bedalus. It is also to be sure about the scores fairness. If the stock 4.0.4 results will be different, I am afraid the benchmarks of the remaining ROMs should be redone with the new kernel :o
[Nexus S General] :
Yeah, checked. Stock survived the test. Looks like it's definitely the tweaks, or just some bad combination of tweaks, or something.
[Nexus S General] RicardoOneri:
What is important is that is not your way fault :D

Good job mate !
[Nexus S General] Harbb:
So we're slowly coming to the conclusion that broken init.d functionality effectively provides higher performance. Sometimes being skeptical has it's benefits ;)

It is still only very general though and it is hard, if not impossible, to systematically benchmark sqlite optimisations, tcp tweaks, ram tweaks, lowmem and so on, especially when some are affected by others. It would be nice to know if there is anything useful to keep, but this would likely require formulating some very specific tests. Did we, maybe, just find something that you can't bench :O
[Nexus S General] :
UPDATE: All ROMs tested without init.d now RETESTED with init.d

Final one this morning was Brainmasters, his results went up about 4% so some tweaks are definitely helping.

kong says he will soon provide the next version, at which point I will re-test.
[Nexus S General] :
I had to go back like 8 pages to find this. Lots of action on the thread last night! I do claim that I test ALL ICS ROMs, so I suppose I should do it, but as far as I can see, this is a themed version of what kong provides, so whether that would have an impact on performance, I can't say.

Uh. Yeah, in fact I'm certain the performance of this ROM will be identical to kongs.
[Nexus S General] Tordp:
Thanks for your time and effort bedalus!
[Nexus S General] motorator:
Just want to check: Is the spreadsheet still valid? It does not appear to correlate with the rankings in the OP? It says CyberGR V10 is the top dog.
[Nexus S General] :
The spread sheet is always more up to date than the op.

Just updated the op now!
[Nexus S General] motorator:
*Now* it makes sense. Almost sprained my brain trying to match the two up :D
[Nexus S General] Oodie:
Looks like my mission ( Mission " frustate bedalus over init.d" ) is over.

I will rest in peace now :D

..Generated Using Tapatalk..
[Nexus S General] :
I'm glad that it was discovered early, before I'd done too many. Cheers Oodie. What's your next mission?
[Nexus S General] stojshic:
Oodie, can you make new trinity based on 4.0.4 with init.d support? Thanks
[Nexus S General] Oodie:
No mission as for yet :D

Surely I'm in killing mood lately :D

Is there a trinity release based on 4.0.4 ? If it comes in a boot.img and if it doesn't support init.d , i can edit the ramdisk to make init.d work .

Plz give me a link .

And if it comes in anykernel , then no need to make changes , just flash it and your rom will do the rest .

And now it's my gym time . Will be online after about 3 hours .
[Nexus S General] tchaari:
nice work bedalus ;)

CyberGR have been also retested?

BTW, Bugless Beast has been updated two times last night. Peter Alfonso has built a new Kernel and I think that the ROM has been tweaked to go with the new Kernel ;)
[Nexus S General] :
Well, it's certainly looking like I will have to benchmark both ways, judging by the poll. (Maybe this won't be too much extra work anyway.) I'll give it a bit more time to see if it definitively pulls ahead.

I was thinking of using the joke idea of the UX benchmark as an excuse to program something in C++ to analyse the data. I wonder how difficult it would be to pull all the text from posts in a ROM's thread?
[Nexus S General] :
Here they are Oodie:

TEUV-ns-cm9-201209401 http://www.box.com/s/0a53a9482d55a07592e2

T132-ns-cm9-201209402 http://www.box.com/s/976183c8f9970d3cf646

... rebased on the updated 3.0.8 kernel
[Nexus S General] rasengan82:
New buglessbeast release!!
[Nexus S General] Oodie:
OK .. Done ! Didn't change anything in the zip other than the init enabled boot.img .

& Now init works . I checked :)

TEUV-ns-cm9-201209401_Fixed.Init - Download Here

T132-ns-cm9-201209402_Fixed.Init - Download Here
[Nexus S General] Oogway13:
New ROM has gone out MIUIXperience 2.3.30:

http://xdaforums.com/showthread.php?t=1579422

Using it and working pretty damn good. Maybe you can add it to your benchmarks. They say that with Gaming Kernel + Script boot @ 1.1ghz (By r_data) it's flying!!!
[Nexus S General] tchaari:
Nice idea especially that new interesting kernels come up (like the ones included in Bugless Beast and MIUIXPERIENCE)

Are you serious? :D

Surely, a difficult work without guaranteed results :)

I have an idea about how to do it in Java if you want ;)
[Nexus S General] :
I've already written the program to pull the text from the threads, next I just need to quick sort it, and throw away any trivial words.
[Nexus S General] :
Okay, I'll bow to public demand. But it'll have to wait. I have to download stuff early in the morning when it's free. (I'm close to my download limit this month - six days until it resets.)
[Nexus S General] tchaari:
wow, that was quick. How do you plan analyzing the retreived text?
[Nexus S General] :
Combine all text from all threads

Place all words longer than two characters into an array

Count the number of occurrences of each word (case insensitive and watch out for spelling mistakes)

Decide which words have some unambiguous significance

Combine totals for synonyms. This figure over the total number of words will be the mean UX density for each group of synonyms (i.e. each aspect of the UX).

Then do a similar thing for individual threads. Then calculate (individual UX density over global UX density (per aspect) ) -1 = a positive or negative percentage relative to the mean UX density (per aspect).

...

It'll make sense when it's in a spreadsheet.

The tricky thing will be making sure that the words selected are unambiguous. Like two people someone say

1) The battery drain is good!

2) The battery drain is fast!

So I probably can't use battery as an indicator.

In fact it might be quite tricky. I wonder what words can be exclusively used in a positive or negative way? It's hard to say, because you can always throw the word 'not' in. However, I suppose most people are direct and would prefer to say "The ROM is fast" compared to "The ROM is not slow".

This is probably more your field of expertise any, right tchaari?

EDIT: I'm quite please with my first ever C++ program so I've uploaded it here. (You will first need to download and unzip this file into your system32 http://xoomer.virgilio.it/hherold/wget-1.10.2b.zip) - and when you run my program, be aware, it will start to download all XDA ROM threads into the folder where the executable is.

EDIT2: Here's where you can get a program that batch converts html docs to plain .txt files: http://www.nirsoft.net/utils/htmlastext.html

EDIT3: Here's how to aggregate the text files: > copy *.txt aggregate.txt
[Nexus S General] tchaari:
It is not really my expertise domain but, as I said, this is a huge amount of work without guarantees in the results. Anyway, you are trying to build a corpus and then analyze its semantics. This is an entire research domain called "knowledge retrieval". Anyway, if you really want to do it, I can point you to:

1) Wordnet which is an ontology of english words

2) Wordnet::Similarity which is a tool for measuring semantic similarity between words.

Maybe counting the given "thanks" for the ROM builder in the thread would be simpler :). However, the results won't be too realistic. For example, Peter Alfonso isn't very active in XDA and he hasn't a big number of "thanks" compared to other more active devs. Despite that, he has one of the most appreciated ROMs for Nexus S.
[Nexus S General] kong:
Thanks both of you for sharing your insights. This could be a very interesting project :)
[Nexus S General] rrohanjs:
hey bedalus i think u may wanna benchmark MIUIXPERIENCE 2.3.30 .. hes 402 mb ram and some add on tweaks on kongs miui ..

its still your call if u think its still similar to kongs, i dnt knw the technical part of it
[Nexus S General] dinos3:
cm9 5.0 is out

.based 4.0.4 , and karnel 4.0.4 ;)
[Nexus S General] :
I've already agreed to do it!

Excellent news! Links always useful, save me a few clicks...
[Nexus S General] thegtfusion:
CM9 v5.0:

http://xdaforums.com/showthread.php?t=1399035

MIUIXperience:

http://xdaforums.com/showthread.php?t=1579422

:)
[Nexus S General] biergaizi:
awesome work!
[Nexus S General] Oodie:
Spread sheet changing bit by bit .. Would be interesting when completed :D
[Nexus S General] AeroEchelon:
Regarding the test, using the same kernel is optimal as it really shows a ROMs statistics under the same management.

If you can do both that is good too; just don't want to make you work too hard bedalus!
[Nexus S General] kong:
Nice figure you got there :D
[Nexus S General] Oodie:
Don't thank bedalus !! Or else he'll lose that figure . lol :D :D
[Nexus S General] :
Nice, binary!
[Nexus S General] :
Okay I managed to download the modded stuffed crust and cyanogenmod nightly. But should I do v5.0 instead of the nightly?

I'm a bit busy at the moment, I've got two weeks break for easter, so I'm benchmarking like crazy, and have a couple of side projects going on with other devs and also my bizarre UX experiment (excuse to learn C++).

Currently coding an app to take my downloaded threads and do a count how many of each word there are. Eg. battery=70 lol=324 smooth=97 etc...

Need to work out how to add stuff to an array but I got up at 4:30 am today and had to take the car to be serviced (twice)...
[Nexus S General] glennkaonang:
Looking at the spreadsheet now, it seems that you are still busy re-benchmarking the ROMs both ways, bedalus?
[Nexus S General] :
Yeah

I need to finish Dario 6.1

Slim ICS 3.1

Then do both default and t132 for:

Bugless Beast

Cyanogenmod (either nightly or kalim's v5.0)

EDIT: I like the new avatar!
[Nexus S General] tchaari:
wow good luck in all this stuff bedalus :)

About the benchmark, just an old suggestion :) I think that it should be better to separate the T132 ranking and stock kernel rankings. I don't find the average scores signficant :) users intrested in stock can directely go to stock kernel ranking and users who want oveclocking and extra features can go to T132 ranking. This separation can make the spreadsheet more readable and less confusing :)

Anyway, it is just a suggestion busy man ;)

EDIT: About your C++ programs, what do you think about ranking ROMs according to the least negative feedbacks count. I think that couting negative feedbacks is easier than postive ones. For example you can count "problem" occurences that are not preceded with "no" or "any". You can also count "don't work" too...
[Nexus S General] :
I think it's quite interesting to average the two out. All I'm doing is looking at performance. I've got two perspectives on each ROM, and I'm giving them both equal significance.

EDIT:

How about this,

I continue to rank based on the T132 score, but in the next column show how much performance was gained over the default kernel.
[Nexus S General] glennkaonang:
Yeah, just stole this avatar from another user I spotted :D :D

Anyway, I think you'd better do CM v5.0 instead of the latest nightly.

I saw a user here telling that the latest nightly doesn't include the latest kernel, while v5.0 does (weird).

As for Bugless Beast, grab the latest build from Pete's usual download site, it's 04-04-12, which includes Pete's latest kernel that fixed the bootloop issue.
[Nexus S General] tchaari:
And what can I do if my head is exhausted by 4 hours teaching, 2 hours meeting on a project and 2 hours of a 12 pages scientific document correction? :p

EDIT: I forgot to mention that I can't sleep without seeing what's new in XDA :)
[Nexus S General] :
OK, will download in the morning!
[Nexus S General] :
Did you see my edit?
[Nexus S General] tchaari:
Very good idea bedlaus ;)

EDIT: Did you see me last edit about your C++ program? :)
[Nexus S General] :
Good idea, measure "anti-superlatives"!

I've implemented the change on the spreadsheet. Not sure if it makes proper sense. It shows if the score improved more than it should have done or less than it should have done.

Scores that improve more than they should have done get a negative %, because the performance of their packaged kernel was deficient.
[Nexus S General] tchaari:
I understood the T132 scores but not stock kernel ones. Let's take this one T132:6.77%	stock:14.54%. 6.77% is the percentage above the average global T132 mean.

Is the 14.54% is the performance gain of T132 compared to the stock kernel?

And what about this one: T132:8.17%	stock:-15.75%?
[Nexus S General] rrohanjs:
+1 ..,.............
[Nexus S General] :
I think i need to sleep on it
[Nexus S General] tchaari:
Well, thank you again bedalus for your continuous improvements to the benchmarks. I think that I have to sleep too. My neurons battery is too low. Needs some charging through the night :D

Maybe I'll understand better the scores tomorrow ;)
[Nexus S General] :
woke up... having a think
[Nexus S General] :
Okay, slept adequately last night, and had a think about the 'Supplied Kernel' scoring system. I think it does make sense.

The 'Supplied Kernel' is tested using the same parameters as T132, but limited to 1000MHz.

In an ideal world, the 'Supplied Kernel' performance should be directly proportional to the T132 performance.

However, differences exist in the 'Supplied Kernels'.

Using the Average Proportional Improvement [aka API - now I pretend to be a dev :D] I can compare the ROM Specific Proportional Improvement [RSPI] using the formula RSPI/API.

(1 - RSPI/API) yields a percentage representing the proportional surplus or deficiency in performance of the 'Supplied Kernel' based on whether the jump in performance with T132 was smaller or bigger than average respectively. This is what you see in column F.
[Nexus S General] glennkaonang:
Noob language please :D :D
[Nexus S General] Oodie:
lol . i was gonna say the same thing :D Reading bedalus's posts are like " Hitting our own heads with a big hammer " :D

& hate to say , the forum is too boring these days :| Inventions seems to be fade away & this thread is one of the exciting threads exist here .
[Nexus S General] :
I think it's too difficult to explain the maths by text alone. Perhaps I should upload a video of myself explaining it on a whiteboard at work!

Thanks for the praise.

EDIT: Does it make more sense if I say that the column F score should be 0% if the jump up from 1000MHz (Supplied Kernel) to 1320MHz (T132 kernel) was exactly what was expected?

It gets a negative % if it jumped up MORE than expected, because it shows the Supplied Kernel was performing weakly.

A score close to 0% or any positive % is good, because the score with the Supplied Kernel AT LEAST as good as it should be.
[Nexus S General] Ibn Saeed:
@beladus, can you add the link to T132 which you are using for the kernels tests. Can you provide the link it in the Header of the Table of T132
[Nexus S General] Oodie:
I can't get this 100% , but i like the idea you used in the beginning . Running 2 passes with T132 & 1 pass with supplied kernel ( passes can change ) . Then get a percentage out of 3 passes . Should be unreasonable in somebody's view , but It should be reasonable who are using the rom without changing the supplied kernel . :rolleyes:

Edited :::

http://xdaforums.com/showpost.php?p=24445378&postcount=706
[Nexus S General] :
I'm using the one kong built that has init.d support, then I flash morfic's first 4.0.4 build (any kernel version) over the top. See http://xdaforums.com/showpost.php?p=24209684&postcount=9744 for kongs.

I remove the contents of morfics init.d folder and re-zip (I'm testing ROM tweaks here, not kernel tweaks).

You can get kongs, then flash morfics latest any kernel version over the top if you want a CM9 ramdisk version with init.d support.
[Nexus S General] :
I think that is too much of a compromise.

I'm not using those Oodie. I already started testing before you produced them, so I'm sticking to the same method for fairness. (It was stojshic who originally asked you for them.)
[Nexus S General] Oodie:
:D Just thoughts .

Oh ! Post edited . I saw that you mentioned them in the spread sheet :rolleyes:
[Nexus S General] :
Keep thinking. I usually cave in at some point.

Thanks, I tried yours (to save me flashing two kernels), then realised it was inconsistent. I need to edit that on the spreadsheet. Sorry for confusion!
[Nexus S General] tchaari:
I think I got it now after a little of rest :)

The idea of measuring the performance boost from stock kernel to T132 is very very interesting to me. However its results are a little bit difficult for common users :). Maybe it should be better to present them in an easier way so they can be better understood.

Besides, I have only a little reserve on this method. You are basing your expected percentage difference only on frequency (from 1000 to 1320 Mhz). I think that there is more than frequency in kernels and ROMs. This proved by the results you get. It can be also proved by user experience. You can find a ROM with a super overclocked kernel but it is slower than another with a stock kernel. It depends on what the devs put in the ROM and the Kernel. I remember you said that you left kernel benchmarking as it is based only on the frequency difference but I feel that there is more than frequency in the global measured performance.

Oh and I have to say I love your threads bedalus. they are a precious mine of information ;)

cheers.
[Nexus S General] Oodie:
Hey guys , Don't Thank tchaari , or else he'll lose his binary figures :D :D
[Nexus S General] :
Yes, sort of. There wasn't much difference between the kernels apart from the MHz speed difference.

The ROM dev can remove incompatible tweaks. There remains a selection of tweaks chosen by the ROM dev, and whatever was integral to the kernel.

OS level tweaks remain the same for both kernels.

The only remaining differences are the 'Supplied Kernel' tweaks (as T132 remains a constant).

So I feel I can produce a reliable statistic on the 'Supplied Kernel'

No problem!
[Nexus S General] tchaari:
Oodie = smooth binary detective :p
[Nexus S General] Oodie:
Ok . Now I ruined your pleasure & Gave you a Thanks . lol :D

& @ Bedalus .. I think its better to Benchmark CM9 stables . V5 is released & it'll be a waste of time doing nightlies :rolleyes: because Kalim always release a Complete version in RC , not in nightlies . Mostly nightlies lack some items .

But this is not true in Pete's rom . well , his rom comes in nightly always :D
[Nexus S General] :
For posterity...
[Nexus S General] :
The one I just added was the v5!

I'll avoid the nightlies from now on. Kalim updates regularly anyway.
[Nexus S General] Bandis710:
I am very bad at statistics and math so couold somebody explain what the second % means I see a t132 % and another % ( from included kernel ??) eg BB has 16% in the second column , does that mean that with the included kernel BB performs 16% better than average and with T132 it performs just 4,5 % better ? And the red % mean they perform a lot worse ?
[Nexus S General] Oodie:
:confused: wonder why that's sitting @ the bottom of the table . Feels really smooth to the hand :rolleyes:
[Nexus S General] :
They have some issue with the RAM. Don't know what it is, but their RAM throughput is very low compared to other AOSP.
[Nexus S General] :
I'm a maths teacher, and this spreadsheet is a covert way of engaging learners.

Green % means that included kernel is good

Red % means that included kernel is bad.

(but remember performance isn't everything)
[Nexus S General] Ibn Saeed:
Slim ICS 3.2 Released

http://xdaforums.com/showthread.php?t=1390118

http://slimics.haxer.org/Slim_ICS/404/3.2/
[Nexus S General] alen1901:
What does it actually means, ram throughput is very low?

Sent from my Nexus S
[Nexus S General] Maximilian Mary:
If you measured the kernel score based on its improvement over teuv, noone would complain. But people don't like seeing performance scores based on how much "less worse" they are than the control. So you just need a baseline adjustment. Say the average performance decrease is 50. Then DefaultKernel - (t132 - 50) is your score.

Perhaps people would be less confused if you simply changed the column headings to "Rom" and "Kernel," perhaps adding a seperate column for "Rom + Built-in Kernel."
[Nexus S General] :
I need some time to think about this.
[Nexus S General] Oodie:
REPOST !!!! :D

Bedalus just won't listen . Too stubborn lad :p
[Nexus S General] panther-'black':
Cybergr is first?

Ummmm......
[Nexus S General] wildestpixel:
PixelRom updated to v1.72 now officially available on the mirror for download courtesy of xdafilerserver.nl.
[Nexus S General] :
I'll get right on it asap.
[Nexus S General] :
As part of my ongoing C++ learning, I've now developed another app that looks at any input.txt file and generates an output.txt file that you can copy and paste into excel to analyse. It performs a word count and outputs the words that were used in alphabetical order and how many times they appeared.

This is part of my 'joke' study of ROM UX. Anyway, here is the program for anyone who is interested.

Also see my previous work on this side project here
[Nexus S General] :
Here's the TOP 30 words used in all the ROM threads:

the	50525

and	24870

post	17654

from	16037

you	15170

this	14746

posted	14535

for	14492

quote	14483

originally	14230

rom	13233

with	11610

nexus	11561

that	11303

using	10832

sent	9582

have	9247

but	9079

xda	8461

can	8086

not	7803

february	6831

just	6051

kernel	5988

march	5955

app	5826

are	5251

all	4992

one	4951

your	4775

...how informative :p

[words like originally and posted come up a lot for obvious reasons]