yashsingh1/xdaforums huggingface dataset:
[Touch HD Windows Mobile ROM Development] michoob:
If only daeron would delete the last column and the classification, all these rude talks would disappear.

To my mind, as I said previousely, the problem is not this benchmark and its reliability, but the classification based on some percentages issued from the benchmark...

Please daeron, delete the percentage column and the classification, and leave only benchmark results as is! This will surely stops all rude words in this thread.

Let each user make his own classification from benchmark results.

Don't you agre with me?
[Touch HD Windows Mobile ROM Development] squirreleater:
That's exactly right, it would just leave a load of results that people can view and decide on themselves.
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
No matter what daeron does, your always gonna get a minority that want something else, you cannot keep all happy. If daeron is happy with spreadsheet as it stands, then who is anyone, to tell him differently. People can interpret the results anyway they want with or without the %'s. All experienced users will know that these are benchmarks only and the real testing and comparisons are done by actually using the roms........then they can make there own judgement!!
[Touch HD Windows Mobile ROM Development] KukurikU:
+1

u r finally sober...:p
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
I'm glad, took a few whiskey's to get me there!!
[Touch HD Windows Mobile ROM Development] KukurikU:
c'mon, keep testing...please:D
[Touch HD Windows Mobile ROM Development] sdalleave:
+1 too.. i prefer to look at a quick % then to go thru all the test results.. although i want all the numbers to be there if i need to look into a particular area.. nice work, keep it this way..
[Touch HD Windows Mobile ROM Development] daeron.be:
Yup, new benchmarks commin'. By houms, long live to him. :)

M-Amine v3 Radio PF	M-Amine

PhamQuang v42 21816 2009.06.29	PhamQuang

Miri v21.0 Premium	Miri

Black Pearl v3.2 Light WWE	panosha

TEMPEST V2.0 OS5.2.21051	PiLoT

EnergyROM 2009.06.30 Standard	NRGZ28

Topix v2.5 blue	Kwbr

Enjoy.

BTW: If any of you doesn't like last "overall" column, there's a way to not see it. Really, you should try. No kiddin' ;p.
[Touch HD Windows Mobile ROM Development] Matthes42:
sorting by "overall performance"

I really honor your work, because you are messure all those "damned fast ROM" messages. Go on benchmarking new ROMs so that a couple of us can choose a really fast ROM to test.

But please sort the spreadsheet by "overall performance" as it has been all the time. So we can see the fastest ROM on top and the slowest tested ones at the botton. Thanks
[Touch HD Windows Mobile ROM Development] bobsbbq:
Only to prove a point I created a quick 6.1 ROM from a Factory 1.59.841.5 Base and added tweaks and adjusted caches and pagepool, Took out some programs etc. As you can see from the results it is not hard to make a ROM that will perform well in these test. I spent all of 2 hours in this ROM and could have tweaked it some more to better some other results but I don't like WM 6.1. The fact is these test don't really show or prove anything but how fast your device can read and write a file. The graphics test do show some benefit though. With all the software disabled how can you ever test a device for stability, and reliability.
[Touch HD Windows Mobile ROM Development] KukurikU:
friendly suggestion: open a thread with yr own benchmark. ;)

just live this one alone.
[Touch HD Windows Mobile ROM Development] bobsbbq:
Thanks but no thanks, going to stay here for a while.
[Touch HD Windows Mobile ROM Development] KukurikU:
no problem. knock yourself out! u r finally doing something useful: u are bumping the thread:D after blowing off 2 hours of yr life trying to prove once again how mature u are, bumping the thread is a real progress.

come more often u are amusing me:rolleyes:
[Touch HD Windows Mobile ROM Development] bobsbbq:
Ha Ha your funny, The real progress is to show people how false and fake this thread is and I'm glad to keep bumping it for you. Your welcome
[Touch HD Windows Mobile ROM Development] Matthes42:
Hi Bobsbbq,

I´m respecting you opinion, but do you know the meaning of benchmarks?

They are no functional tests but only messured facts in this case of ROMs.

It´s true, there is much more than speed to make a good ROM but it´s one for some ppl important point. so, if it´s important for you, you´re free to ignore all benchmarks.

I for my own have decided to have a look at this thread every some days to see what´s new in fast ROMs or not so fast ones. The fast ones I´m going to test for stability and reliability.
[Touch HD Windows Mobile ROM Development] daeron.be:
Ekhm, Today I felt like testing new Laurentius26 rom. And I think it was a good idea. I guess even he will like the results. :) I wonder if they are real. ;p

Added:

Laurentius26 6.1 v3.2
[Touch HD Windows Mobile ROM Development] michoob:
Isn't there an error in the K2 cell?

Shouldn't it be 1254.70 instead of 254.70?:eek:

I know Leo is a great cooker, but here, there is an obvious error in the benchmark...:rolleyes:
[Touch HD Windows Mobile ROM Development] daeron.be:
That's not an error. It can be felt when you enter directories. I assure you, that's a correct value. This rom opens directories and lists files there 5 times faster than others. :)

BUT: it has one of the most lagging manila (tf3d) I've ever seen. Really. I'm happy I don't use it, because I'd be really pissed off.

Laurentius posted video few pages ago. Look at that video and pay attention to manila bar sliding lag. But anyway... who cares about manila?

EDIT:

Added Onkolog's 2009.06.03 lite (6.1 rom). Actually, this rom felt really great for me. But it's subjective, and in terms of that benchmarks, there was no breakthrough. I flash L26 v3.2 for further testing. :)
[Touch HD Windows Mobile ROM Development] SatScan:
SALUUT to all GREAT COOKS !!

Hi NR1 sorry i mean Leo,

Can I ask, do you specific (only for this matter) Patching/Hexediting to get better benchmark results?

Grtz,

SatScan
[Touch HD Windows Mobile ROM Development] SatScan:
Fieldtest haha

I HAVE A BIG SMILE !!!!

What a joke are that benchmarks! It tells nothing about how the rom works in the FIELD!

The only thing you can do with it is... cleanup some server room en delete this thread!

Have a nice live,

SatScan

Beta TESTER for (far out) NR1 in this bench!
[Touch HD Windows Mobile ROM Development] daeron.be:
I'm glad you like my work.

BTW: After a longer while of using, sliding got smooth. Really, that's the best rom so far I've seen.
[Touch HD Windows Mobile ROM Development] Timmy_01:
Hhhhmmmmm... Laurentius26 V3.2 with an Overall performance of 95,36% ????

That's impossible! There must have been made a fault in testing!!! Yes, it's a very nice version to work with (I do it since 24 hours), but such a high score is unbelieveble!!! :confused: :confused: :confused:

Regards,

Timmy
[Touch HD Windows Mobile ROM Development] SatScan:
Testing and Testing ... YEAH real FIELD Testing!

Hi There,

As you know i'm testing all the (Pre-)Beta rom builds allmost on daily base.

This ROM is fast and is very smooth... but that fast... haha what some benchpatches all can do...

So time to move over on WM65. to get the fastest overthere 2

Grtz,
[Touch HD Windows Mobile ROM Development] daeron.be:
Naah, they aren't unbelieveble. If there was a 80% difference to another rom, that would be a weird thing. I have to better explain how is it scored, because I feel like many people don't understand that.
[Touch HD Windows Mobile ROM Development] Laurentius26:
No comment! :D
[Touch HD Windows Mobile ROM Development] KukurikU:
oh, how nice...some ppl do come back;) maybe they even changed their opinion about this benchmark and the one that started this thread. suddenly, smiles are everywhere:D

daeron, i think this is the right moment to delete some of the chefs that begged to be deleted. :D
[Touch HD Windows Mobile ROM Development] pzucchel:
Daeron, let me go back to the point.

I tried Laurentius ROM, and i found it much less effective and snappy than Dutty's, or kwbr. I am a bit surprised, since i got convinced that what we call "snappy behaviour" was related to the I/O, and therefore i was weigthing these columns more in order to choose what to use. Laurentius Rom is the exception: in my hands, it FELT much slower than Dutty's.

I doubt that the problem is TF3D. Did you try to execute your tests with TF3D active? if so, did you notice a significant degradation of the performances? if this is confirmed, are some roms behaving better than others with TF3D installed?
[Touch HD Windows Mobile ROM Development] vorenius:
Well... I am sitting on top(ix) 2.2.1 for a while and could some smart head be so smart as to explain me how the hell that exact rom was scoring ninety something percent and now it plummeted to the miserable eighties?:D A-ha, may be the chef himself did a rebuild in order to keep low... ahahahhaha.

Guys, i am not an ally nor a foe to anyone but i will keep saying that these benchmarks are just benches in da park.:D

Nobody up to now explained to the poor illiterate audience the magic abrakadabra of the final output. Or may be the recipe is as secret as the coca-cola ingredients.:D

I keep seeing dudes sayng that some smart asses know what these figures are all about. But i've never seen any smartie expalin...;) Aiudooooooooooooo... please, please...

Well, again... i think the author of this should give us a short free of charge lesson of what is what and where the lions lie. :cool:

Salute
[Touch HD Windows Mobile ROM Development] Laurentius26:
Dear gentleman,

In the 8 years of experience of rom cooking at XDA I never had so much insults as in this thread, some people here should really be ashamed of their language used.

In the past I have written that this thread is misleading and I have demonstrated this with a video.

I kindly requested if my results could be removed because they indirectly affect my work and I think I made my point a second time now.

Daeron decided to ignore this.

Maybe you understand now what is bothering me.

I like to advise you to respect the requests from my fellow chefs in future.

If you like you can remove all of my results.

I thank the people who have supported me:

Bobsbbq

Cerato

Epiphany

Satscan

Timmy_01

My apologies if I forget someone.

Sincerely, Leo
[Touch HD Windows Mobile ROM Development] daeron.be:
Counting Overall score is basically doing this sort of math:

=(E82/E2+F82/F2+G82/G2+H82/H2+I82/I2+J82/J2+K82/K2+L82/L2+M82/M2+N82/N2+O82/O2+P82/P2+Q82/Q2+R82/R2+S82/S2+T82/T2+U82/U2+V82/V2+W2/W82+X2/X82+Y2/Y82+Z2/Z82)/22

then

=(E93/E3+F93/F3+G93/G3+H93/H3+I93/I3+J93/J3+K93/K3+L93/L3+M93/M3+N93/N3+O93/O3+P93/P3+Q93/Q3+R93/R3+S93/S3+T93/T3+U93/U3+V93/V3+W3/W93+X3/X93+Y3/Y93+Z3/Z93)/22

etc.
[Touch HD Windows Mobile ROM Development] liamhere:
note to all

i am now cleaning all the comments that are off topic and causing trouble.

behave....or this thread will be closed....and i will take action if i feel fit.

you have be warned.
[Touch HD Windows Mobile ROM Development] yung_vu:
from what I can tell, this is how it's calculated:

it looks at a roms result in each individual test vs the top result in that test and coverts it into a %. it then takes an average of each of these 22 individual tests to come up with the overall %.

confused? here's an example (assume for the example below that higher scores are better):

- ROM ABC scores 75 in test 1 but the top scoring ROM scored 100. so for test 1, ROM ABC scores 75%

- ROM ABC scores 30 in test 2 but the top scoring ROM scored 60. so for test 2, ROM ABC scores 50%

- ROM ABC scores 234 in test 3 and is the top score for this test. so for test 3, ROM ABC scores 100%

- based on these 3 tests, ROM ABC's overall score would be 75% (the average of 75%, 50% and 100%)

so a ROM's overall score % will change as new ROMs are tested and new "best times" are achieved in each test. we saw this happening recently as Laurentius' ROM produced new best times in numerous tests
[Touch HD Windows Mobile ROM Development] KukurikU:
i totally dislike the selective cleaning! where were u when the troublemakers posted again and again all kind of nonsense insulting the thread opener?

too much of their posts are still here. if u decide to cleanup than do it properly. please clean this: http://xdaforums.com/showpost.php?p=4071797&postcount=353

and the next after it.
[Touch HD Windows Mobile ROM Development] SatScan:
MOB benching versus PC

Modurator,

What I am concerned, you may close this thread. I will explain why I find this.

The design of this benchmarking ROM is no account of what the makers of these ROMs are there. There is not initially indicated how the results come.

By later to ask is there any insight into data. This has led to fierce debates / discussions that do not contribute to the fine open and friendly atmosphere here at XDA.

A proper design would have a good disclaimer about how and what.

The possibility to makers of ROMs data not in the list to be included.

These benches are very subjective look at the history and current discussions on PC on the Internet.

If all this for me enough reason to close this thread.

Thank you for your attention,

Grtz,

SatScan​
[Touch HD Windows Mobile ROM Development] pzucchel:
Dear Vorenius, the answer is very simple - even without mathematics: the way the score is build up - arbitrary but objective - depends on the other roms. If all roms would become suddenly great, a ROM having 100% could go to 0%. The percentage value you are looking at is a RELATIVE ranking, not an absolute ranking. As Daeron explained, the ranking is totally transparent to anybody that knows how to use a spreadsheet. I agree with you that a relative ranking could be misleading for those who do not take the time to understand it by opening the spreadsheet.
[Touch HD Windows Mobile ROM Development] pzucchel:
Dear Leo (how are you, BTW?)

I probably didn't see the insulting comments you are referring to, i fully agree that setting up a realistic benchmark should not be the place of criticizing one or the other cook - i know how much time is needed and how difficult it is!

As you know, i was also historically not convinced about benchmarks. However, it is probably not by principle that benchmarks are not good enough in representing reality. Probably, the benchmarks have never been optimized in order to give a realistic response, comparable to user experience. In fact, it looks to me that having a thread on this purpose could be of interest to everybody, cooks and users.

I realized just now i also commented about your ROM. Far from me to judge the overall quality of your ROM, good as usual - and very rich of interesting features. I was just commenting on the fact that you scored extremely high on some columns, i particular the I/O benchmarks, but TF3D looked - to my subjective test - slower than in other ROMS that performed less good. I got convinced, from previous comparisons, that I/O was the most important parameter and almost representing what i define subjectively "snappy", but i probably got it wrong. I don't know if you agree, please try installing Dutty's 3.9 Turbo and feel the TF3D speed: I know you are an intellectually honest person...and as you know, I am definitely not a Dutty's fan!

Since apparently there is a lot of people which is taking all possible opportunities to criticize and offend others, i will refrain from now on to post comments on specific ROMS that could be misenterpreted. It is a pity, since ROM progress could come only by "sharing our knowledge".

ciao,P :)
[Touch HD Windows Mobile ROM Development] KukurikU:
thank u for the very deep analysis. this thread was very quiet until u and bobsbbq came. so quiet that it was most of the time in the second page! everybody was reading and getting his own conclusions. that was before laurentius's ROM hit the bottom. from that moment, all hell broke lose.

i also think that daeron deserves apologies for the fact that after taking all the s*** from you, he benchmarked the new laurentius ROM and posted the flattering results. btw, congrats to laurentius!

as for u, every possible explanation is in the thread. start reading it and only after that prepare a new deep analysis.
[Touch HD Windows Mobile ROM Development] daeron.be:
Update

Very regular update of roms - all 6.5 ones. So regular it hurts. All by houms.

Added:

M-Amine_V3_PF_update2

Miri_WM65_23003_V22.0_Premium

EnergyROM_BLACKSTONE_20090628_Standard_Magic

Miri_WM65_23003_V22.0_Premium_Manila_2.5
[Touch HD Windows Mobile ROM Development] devil1591:
hi! thanks for this good work!

Could you add the release date of the rom? this would be more accurate... :p

thanks!
[Touch HD Windows Mobile ROM Development] Vinfer:
With all due respect, I disagree. Sincerely i appreciate the daeron's work. I think he has explained quite well the criteria of the benchmark. You can accept this or not, you can remove a rom or not, you can read or do not open the topic. You choose, no one will force you.

Benchmarks not are the Law, we can read and we can ignore them. But I also realize that they may be an strong advertising and so the author of them must be a very careful person above the parties, this is perhaps the most important thing.

However at the end of everything ... the important thing is...take it easy...:)
[Touch HD Windows Mobile ROM Development] pzucchel:
Great words, i also appreciate Daeron&Houm work since I am assuming that who spends the time in testing and publishing their results is above the parties. This Thread, simply, is very useful. It gives some metrics and allows you to choose with some guidance, different from the number of reads of a thread or from the voice of supporters of one or the other cook.

It is not perfect of course...like our touch HD! :p
[Touch HD Windows Mobile ROM Development] pzucchel:
Mr Sonydesouza,

Please do the following: once you have opened the google spreadsheet, click on the third item on the pale blue menu list, the one after File and after Edit called "View". In the pulldown menu, there should be five items, the last of which is "List View". Please select LIST VIEW, then click below OS and select 6.5 or 6.1 or whatever you like to separate. You can also choose your favourite cook, and see how his/her ROM has evolved over time. A spreadsheet is a tool allowing to see the data you like and to hide the data you don't like, you don't need to delete or to make a modification for any request from any user of the same file. And a spreadsheet is designed to be usable also by people and children without technical or scientific background.

;)
[Touch HD Windows Mobile ROM Development] rehmaau:
Excellent Job

Well done!!! An excellent comparision between the ROMs and best guide to choose the best ROM.:)
[Touch HD Windows Mobile ROM Development] bobsbbq:
Pzucchel, I wish you would have brought this up a long time before. Now I have to admit I'm not a spreadsheet expert but do know some general formulas and excel etc. but I have never used google doc's until this thread started. I did not know this was possible and answers one of mine and many others request about separating the type of ROM's. So if daeron would have nicely said this before it would have helped to stop some of the problems.

Now I still have some issues with the way the test and methods are done, but I guess he does not want to listen to some of the cooks advise to make things more credible and accurate. That's ok I'm done fighting in this thread unless attacked again, I have sent daeron a polite PM with a request and apology and he still has shown an arrogant attitude. With some people they are just way too arrogant and short sided, or know it all and it is a shame because here on XDA everybody learns from one another. If you read the thread from the beginning there was many people including myself making friendly suggestions about the 6.1 & 6.5 separation, and other creative suggestions. Daeron finally listened and put in the 6.1/6.5 column. Things could have been much better if he would have kept a little more open mind when listening to suggestions, especially when coming from cooks.

Another request which has been ignored is batt measurement. This can be done and I offered to help but it was ignored. There is a app you can install which will measure battery current or drain in mA. You can do a test with BL set to manual and for a few min and record the results. Is it a perfect test? Know but it does give some reference.

In the end I wish daeron would be more receptive to change, but that may not be and is where my problem lies.
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
I wish people would stop bickering and accept this spreadsheet for what it is....... a spreadsheet!!!

Read it, view it, sort it out whatever way you want.....and in the end interpret it how you like. Then go try the bloody roms out and make you own conclusions!!

Simple isn't it!!
[Touch HD Windows Mobile ROM Development] EpiphanyNL:
Well said!
[Touch HD Windows Mobile ROM Development] corwin_de_amber:
I can understand the difference in graphics performance. It depends on drivers ect. But how CPU test can be different? IMHO, it seems more like the inaccuracy of the benchmark program.
[Touch HD Windows Mobile ROM Development] s1rl4ncel0t:
Dont forget to include the Energy roms in your benchmarks;)

HastaSSSS:D
[Touch HD Windows Mobile ROM Development] daeron.be:
Well, that's pretty obvious to me. How well a rom will score in such CPU benchmark depends heavily on drivers in system's core and other pieces of software from system itself.

In a same way there are differences in speed (and benchmarks) in Windows 98, Windows XP, Windows Vista, Windows 7, MacOSes and linux kernels, there're such differences in our roms.

These benchmars are for users to help them choose best tuned/cooked roms and for cooks to check which rom's the best, so they'd be able to see what parts of system that rom using and make theirs even tastier.

But I'm talking about raw performance! Nutritious soup is good, but tasty and nutritious soup is even better!And everyone have different tastes... :)
[Touch HD Windows Mobile ROM Development] djcedric:
interested to see energyrom results to,thanks.
[Touch HD Windows Mobile ROM Development] ergo911:
Dutty's 3.4 is here :)

Please test it ;)
[Touch HD Windows Mobile ROM Development] daeron.be:
I've tested another rom and yes, we've got another King :)

I wonder if how Dutty's 3.4 will do, benchmarks will be here in 9 hours. Stay tuned...
[Touch HD Windows Mobile ROM Development] agroote:
What do you mean. Users of your rom still need to tweak ur rom with advanced config? Why isn't this cooked in when this is faster?
[Touch HD Windows Mobile ROM Development] KukurikU:
he is setting them average to protect our devices from overloading:D:D

this way we have even more choices. not only what progs to install but also to have a fast device or a slower one:rolleyes:
[Touch HD Windows Mobile ROM Development] kwbr:
Thanks Daeron.
[Touch HD Windows Mobile ROM Development] styler_RO:
pls add davideuck ROMs to compare.
[Touch HD Windows Mobile ROM Development] zuadao:
would also like to see topix 1.6.1

could you include it pls??

i find it very fast and stable

thx
[Touch HD Windows Mobile ROM Development] KukurikU:
wow! kwbr's ROMs are smokin' fast!:p i have to add that they are also completely bug free!:D
[Touch HD Windows Mobile ROM Development] hellonasty:
And MaryOne Wm6.5?

:D