yashsingh1/xdaforums huggingface dataset:
[Touch HD Windows Mobile ROM Development] daeron.be:
Ehem. Laurentius, I think you're missing a point. I've said as far as _I_ know number of files in a rom doesn't have to impact speed of a rom itself. Unless system is performing a listing of files in a certain directory (it's senseless for every app to continuously scan /windows directory or any else - maybe there're some applications do such listings, but I don't believe they do this all the time to really affect anything) or A FILE IS RUNNING IN A BACKGROUND/FOREGROUND! Therefore, if I install 1000 apps with exe and dll files, but none of them will be loaded/executed on a boot or by other programs. As I mentioned in my previous post, installing so many files may cause disc (rom) fragmentation.

I'm trying to give them to you. But you don't listen. I've PMed you an answer to your accusations once.

1. not always true. not running apps don't use anything, only ROM space. Running apps use memory (RAM) space and if they're running CPU time too.

2. that's true

1. I'm not testing manila, but a rom without manila nor other "TODAY" apps/plug-ins on. Some roms don't have manila so I couldn't compare them equally. Yeah, you can say that. But it's like arguing that all speed tests of cars should be done with 4 people in a car, because you'll be driving this car with your wife and children. Don't be over-demanding!

Huuh! I've tested (with help of Houms) pretty a lot of roms. I'm sorry to see that your roms are about 10% slower than the rest. I've already said that people might not see even 20% difference in speed.

And even if you say that your rom is performing best, it doesn't mean it performs that good. If you say that it's most liked rom here, or most bug-free, or most user-friendly then say so, I won't test it. But in terms of performance, IN MY TEST it doesn't score 1st.

If you like, then go and open your own thread, test all roms around and show me I'm wrong. Or create a _better_ rom. Look at JohnPatcher's barebone rom - it matched speed with Kwbr's! You can do it too if you can! I'm cheering on you ;)

Well, if I understand you correctly, you're right. Maybe. :)

See my post in this thread - http://xdaforums.com/showpost.php?p=4024242&postcount=236 (first part). I've already stated that.
[Touch HD Windows Mobile ROM Development] cerato:
Dear Daeron,

To be honest (no offense) this benchmarking is a total stupidity. As i stated previously in this real thread, roms ARE NOT PROPERLY evaluated based on OS, latest updates used, drivers etc. This so called "benchmarking" ( btw you never explained to public which value means WHAT just "less" or "more" better ;) ) is as useless as an umbrella after rain passed and sun went out... hehe. I've made my own benching... wow nothing to do wit so called results here... no offence once again. AND... what is speed... pls explain... And what about A MOUNTAIN OF BUGS some of the roms contain in ( no offence to chefs just my greatest appreciations of their HARD works) and we all have to understand that to a certain point NO CHEF could do NOTHING untill an official release of some part is released. There are THOUSANDS OF MOUNTAINS of workarounds, tweaks, tricks, cheats and... name it to make our roms better and alll these are part of mutual work and co-operation by chefs, users, testers, and... name it again.

If you take a look at latest editions of roms.... they are all as kwbr said "frankenstein" builds... topaz, mopaz, rhodium, mhodium... a lotta sleeples hours by all chefs involved. AND PLEASE ONCE AGAIN EXPLAIN how on earth you guatrantee that these figures represent real world... i hardly doubt you can. :)

p.s. Please there is a REQUEST by Laurentius26 to remove data from yr list. PLEASE REMOVE>

p.S of P.S========= HAte to be personal but why this skinny rom of John Patcher (who btw blamed onkolog of being incapable of making good FULL roms) is taking part in this so called "classification" Daeron, note that the road to HELL is paved wit good intentions ;)

EnD OF QUOTE.
[Touch HD Windows Mobile ROM Development] cerato:
Hey dude,

I've made alll tests AS PER YOUR SO CALLED METHOD... and believe me... NOTHING REAL ON A RAW... 10 tests 20 results...

I have two HD's and currently running L26 V3.1 and Topix 2.2.1... BELIEVE ME... your results are WRONG... ;)... BUT WHAT IS MORE IMPORTANT... they are both as rock stable as OSAMA BEN LADEN AGAINST FBI AND NSA... Sorry for the black humour hehe

I will not take part in this "scientific" ;) thread anymore... hehe
[Touch HD Windows Mobile ROM Development] Laurentius26:
Daeron it's pointless to talk to you.

You continue and continue and even if I ask you to remove my results you are ignoring.

Have fun with your benchmarks.

As surprise I've made a video of that worse then others performing rom so everyone can see how misleading benchmark results can be.;)

Have a nice show. :D

http://www.youtube.com/watch?v=j2IrDM0ngQM

Bye bye.
[Touch HD Windows Mobile ROM Development] cerato:
Leo,

It's obviously really no sense to continue. The guy just doesn't take care. That is sad but... Lets give him his minute of glory.:D

My finish for real ;)
[Touch HD Windows Mobile ROM Development] Laurentius26:
Haha.. thanks Cerato. :)
[Touch HD Windows Mobile ROM Development] daeron.be:
Well, I've watched this "show", and I'm not impressed. Sorry to say that. Your manila has a significant lag when it's scrolling. And that's a pity.

Laurentius26, you're acting like a child. If you don't like this thread, me, benchmarking then don't post there. Forbid all your "followers" than looking into this thread. If somebody want to take these into consideration, he will. And really, your manila is laggy. No offense.
[Touch HD Windows Mobile ROM Development] andes83:
Sorry to see such an argue here between rom cooker and benchmarker but in my opinion it really would help if ROM cooks put a video of their ROM in action (like Laurentius did) and not only screens.I know that they change a lot of things too frequently but all in all they follow their own "style" and it would be really usefull if we see how they perform or look before flashing them on our devices.I'm not taking any side here just making a sugestion "raw" numbers sometimes don't mean anything against usability and looks and I'm really gonna point this out again BATTERY LIFE DOES MEAN A LOT and if there is a benchmark about this it's gonna be the most popular thread in the forum cause there is no point in having speed without the time to use it.
[Touch HD Windows Mobile ROM Development] squirreleater:
Hope that wasn't aimed at me?

I've only run leos Rom once and didn't like it, but thats not the point, it's about giviing the Rom cookers some respect- this whole section wouldn't exist without them. If someone politly asks for their Roms to be excluded then you should-end of.
[Touch HD Windows Mobile ROM Development] MrHarm:
The real question here is of course why it was so hard for daeron to just remove the results as requested in the first place.

You may not realize it yourself, but by your rather obvious campaign against Laurentius26 over your last posts, you cast serious doubts on both your desire and ability to show even what limited objectivity a benchmark test can achieve.

As for the benchmarks themselves...I guess they can be nice enough for some users. They obviously don't reflect my usage patterns, as for instance Dutty's ROMs (which always score highly) always seem very sluggidh on my HD, while the newest 6.5 Energy ROM is by far the fastest in everyday usage of the ones I have tried.

Just goes to show that speed is most of all a question of individual usage patterns beyond a certain minimum point, and that the only way to actually find out the speed of a ROM is to flash it and try.
[Touch HD Windows Mobile ROM Development] daeron.be:
Well, he "attacked" me, because his rom didn't score good enough. I was trying to discuss it. Is it really a campaign? Somebody already else already requested their rom to be excluded from this comparison. I've turned this person down. "No quarter".

If people can cook roms using part of operating systems that wasn't developed by them and credit them, as if they were their own work, I can test them. Have you ever seen Microsoft wanting to ban somebody from testing the speed or anything of their OSes? That's ridiculous.
[Touch HD Windows Mobile ROM Development] MrHarm:
Obviously nobody can force you not to test a ROM. But it wouldn't hurt you to show some respect to those who actually do the work that makes this site what it is.
[Touch HD Windows Mobile ROM Development] daeron.be:
I've shown respect to them. I answer their accusation with everything I got. But I won't remove these results, because that would be unfair to those, who read them. And I'm positive, there are people who do it. Right now, 10 people have this spreadsheet open. That's enough to keep me going.
[Touch HD Windows Mobile ROM Development] houms:
4ill stop testing your rom

In my point of view, the good response would be to understand why our benchmarck show your rom as beeing slower. Perhaps it needs some optimization, perhaps it's an error in our method.

Eventhought, if you don't want me to continue with your rom, I'll stop.

houms
[Touch HD Windows Mobile ROM Development] johnpatcher:
Where have you this from? I haven't blaimed onkolog for anything at all. I've just claimed that I don't like the idea of protecting ROMs you release, I haven't said anything about his skills as a cook nor have I shown disrespect to him in any other way.

Far from it! I've even tried to tell him that he has ported his XIP bad, as the OEM part of the XIP is missing. This was supposed to be constructive critic, but, I guess, he got it wrong, because after telling him that he accused me of stealing parts of other ROMs, what isn't actually true.

I agree with that, the "speed" of a ROM can't be tested at all, but these benchmarks are the best we have so far. If anyone is aware of new objective tests, I bet that daeron will try to implement them!

I don't understand this silly conflict between Laurentius and daeron :(.

Laurentius has cooked a ROM, which may be great, but objective tests show that it isn't that good at all. He has asked for reasons, and I've given some. But instead of trying to get them working, he is accusing the benchmarks to be bad, and that his ROM is just great and best. I think Laurentius is mixing up subjective impressions of his ROM and objective results of a benchmark. They both haven't to be the same, but I think looking at the (more or less) objective results of the benchmark is more reliable.

And trying to get (bad) results deleted isn't that nice also, these are independent tests, so it doesn't make sense to delete or modify them in any way. Every producer has to live with that, be it a carmaker or an artist. If any cook is understanding this as disrespect, he should quit ;). I think its a kind of challenge to achieve better results with the next version you release.

So don't be that closed-minded Laurentius. If your ROM is that great, although benchmark say something different, you'll have enough users anyway :).
[Touch HD Windows Mobile ROM Development] squirreleater:
Microsoft earn millions of $$$$ from their operating systems, as do HTC.

The Rom cookers ask for and get next to nothing.When you pay for something you have a right to criticise, if it's free STFU.

Of course the cooked roms are their own work? As I've said, without them this entire section wouldn't exist-along with most of XDA dev.

Carry on with your arrogant attitude and it wouldn't surprise me to find a chef stop making Roms just to spite you.

Loose the attitude, show some respect to the people who make this happen and improve all of our phones by doing as they ask.

Not much to ask is it?
[Touch HD Windows Mobile ROM Development] johnpatcher:
So you say it is not allowed to compare things that are, more or less, for free? I think it does make sense. I want to know which ROM performs best on the benchmarks. Neither the benchmarks nor daeron are saying that the best performing ROM is actually the best one. You don't have to install or try it. That would be arrogant, but comparing them doesn't say anything. Its up to the user to interprete this results and to choose the ROM afterwards. These benchmarks can only assist you while choosing a ROM.

I think this is a claim not that easy to proof. As a cook I know you have to put in much time (and a cook of a "heavy" ROM certainly puts in even more time ...), but in the end you are using available packages, which isn't even totally legal at all. As far as I know there were even claims of Microsoft to remove all ROMs, so the proprietary is still on side of Microsoft or HTC.

This is certainly true, nothing to argue about.

If a cook can't live with that, he shouldn't be releasing ROMs here, because xda-developers is about testing, understanding and improving our devices. These benchmarks contribute to this goals.

daeron has invested some time and effort to test the ROMs. Now a cook is asking to remove the results, because he doesn't like them. I don't think thats fair at all. As stated above these results are independent, and therefore nobody has a right to add, remove or modify anything at all. If you don't like daeron's work, then just ignore it?

A carproducer also can't ask a testing facility to remove their results, because they are too bad. So is the testing facility disrespecting the efforts of the carproducer? I don't think so! And, for me, it doesn't matter whether you pay for something or not. Its just a comparison, not any form of recommandation.
[Touch HD Windows Mobile ROM Development] bobsbbq:
John I'm not taking a shot at you but just a statement. I disagree with the above statement and they are not even a close comparison. The testing methods for cars are Government ISO World testing standards and are equal to all Auto Manufacturers. And just as a point you would never see official test results being published that were sent in by the manufacture themselves.

This Whole testing and benchmark thread has lost all credibility as soon as you start letting more than one person do the testing and especially the cooks send in their own results.
[Touch HD Windows Mobile ROM Development] bobsbbq:
daeron.be, This type of resource is supposed to be just a resource. IMHO your methods and the way you gather results are a Joke and in know way are credible. As previously mentioned how a device runs with everything turned off or disabled does not provide any real world user benefit. I mean would you trust the purchase of a car because someone said it was the fastest from 0-60 and had the fastest stopping distance? I think you and most others would test the stereo and AC, lights, electric seats steering handling etc. as well.

You should take a more polite approach to the cooks on this forum. As without them you would have nothing to test.

Please see the following thread where I had a similar idea in the past and think about my quoted text below from this thread.

"My goal is not to cause any hard feelings which is why I will upon request of any cook review again or change any thing which is brought to my attention which may be in error. I will also at any cooks request remove their ROM. This is meant to be a resource to help, not to hurt anyone."

http://xdaforums.com/showthread.php?t=492868&highlight=detailed+rom+information

If someone doesn't want their ROM in your test then give them the respect and remove it. You will still have plenty to test so why make problems for yourself and get this thread closed over 1 or 2 request not to participate in your testing.
[Touch HD Windows Mobile ROM Development] KukurikU:
yr ego's got the best of you. what a mess!! this was supposed to be a benchmark, a simple sum of tests to help users choose a ROM and u turned it in a war zone. a few unsatisfied "test subjects" and their fans :p think they can beat the numbers by flaming the tester. the numbers are there and i see no point in arguing. how well those numbers reflects the reality...let the users decide!

i PMed and warned daeron.be in the first day after opening the thread:

i was right even if daeron.be was and remain impartial.

for all others: look at onkolog for a fine example of correctly using this thread.

instead of whining, kicking, screaming and flaming he did the right thing, he used the results from here and really improved his ROMs. he won all my respect for his attitude!
[Touch HD Windows Mobile ROM Development] henrik.dk:
The whole XDA is making stuff non-officially and probably not perfectly.

Should we get rid off XDA? The answer applies to this thread also.

the benchmark may not be perfect and somewhat might be misleading, but it is a kind of information. whether the information is from the cooker or others, there is no cheating. The user should just judge it by himself.
[Touch HD Windows Mobile ROM Development] daeron.be:
I've added results produced by chefs, but I'm going to verify them. If you were looking into this thread regularly, you'd notice that JohnPatcher sent me his results, but he done this test wrongly, so the results were 'hyped'. When houms tested this rom and seen the difference, we've deleted JohnPatcher's results as they weren't reliable.

I'm not going to fight here anymore with anyone, because I won't accomplish anything that way. Let's let people decide.

Kukuriku - well, I thought for a second that _dialog_ would be possible with some people. I guess I was over-demanding. ;)

:)
[Touch HD Windows Mobile ROM Development] vorenius:
Hi,

It is not a matter of someone's EGO, but a pure objectivity, which imho is missing neverteless figures and measures listed. For example I use SKTools and when benchmarking and then, corelating SK benches to SPB, results are either better or worse than those published. So which benchmarks are more reliable - that of SK or these from SPB?

Let me briefly give you another illustration. One of roms tested loads an application for 2.5 seconds, while the second does this operation for 3.0 seconds, In terms of figures the first one should be 20% faster but... the supposedly quicker rom has so terrible memory management that it often hangs still and needs a soft reset. Moreover, this "smoking fast" rom eats my battery as a hungry dinosaur. As for the "slow" one i can open some dozens of apps and it works like a charm, no hickups, lags and soft resets every hour - when i close respective apps nothing stays running in the bg and memory is always around 42-44%.

To continue my opinion i will point that some cookers experiment with different drivers, apps, manilas and etc. This is the way to further develop software but often these new software slows or speeds the rom, or makes it show some buggies, but do you think the rom is better or not?

I see that almost all users including chefs stress on speed. But for roms sake, this is not a Formula 1 race, it is a rom development. In my personal noob opinion ;) stability and flawless usage are all that is important. And to give you my final illustration i will say " Who gives a sh*t if dialer rotates like a merry-go-round when I constantly scream in the receiver: alo, alo, alo-o-o-o" :D
[Touch HD Windows Mobile ROM Development] johnpatcher:
But thats nothing you can test at all ...

Its not wrong to give a **** about the benchmarks, you just shouldn't take them too serious.
[Touch HD Windows Mobile ROM Development] MAMeingast:
NO ONE EVER SAID THESE BENCHMARKS ARE THE ONLY CRITERIA TO JUDGE A ROM BY!

That's the problem with this discussion: All those critizizing the benchmarks emphasize that there are many more aspects to be taken into account when deciding for a ROM - and that is absolutely correct. But that has nothing to with this thread: It doesn't claim to be the final and complete ranking system for the (overall) best ROM - it just compares one aspect: the benchmark results. And the user has to decide how much emphasis he is going to put on them.

To demand the removal of these benchmarking results is like complaining to a poster writing about the awesome battery life of ROM X because he's only talking about the battery life and not about the bugs the ROM might have. Just because you don't mention every information about a ROM doesn't make the partial information wrong or unnecessary!
[Touch HD Windows Mobile ROM Development] vorenius:
The problem is that there is a discussion over imaginative data. ;)

Ok, here I say it: BENCHMARKS ARE NOT CRITERIA AT ALL. Because and mainly:

Whenever you perform benchmarking at always same conditions and with whatever tool, the results are always different! To explain more picture-like, it is as if measuring a liter of milk three or four times and results showing 1l, 1/2l 0.75 l and etc. :D If you do not believe me just carry a test with SPB three times in a consecutive and then post "results".

When there comes any tool capable of showing equal info, no matter how many tests are being done, then outputs would be trustworthy and not misleading.
[Touch HD Windows Mobile ROM Development] bobsbbq:
How do you know there is no Cheating? Are you sitting next to the multiple testers and cooks to make sure they are following the method? TestOpenGL is a good example, the longer you let it run the higher the results are.

I'm sorry you can have honest and credible results with multiple people and cooks sending in their own results.

I'm not worried about my results they have been good within the WM 6.5 group. It is just the way the testing is being done and data collected and the lack of respect to cooks.
[Touch HD Windows Mobile ROM Development] sonydesouza:
I already said that these benchmarks don't make sense and some people laughed at me...

Let them with their stupids benchmarks...

Moreover, since this thread is open, there is a bad vibe between some of cookers, some of peeps...It's really sad...

So, i would like admins close this ****ing thread...It would be better for eveybody...
[Touch HD Windows Mobile ROM Development] KukurikU:
i am not laughing, this is yr opinion and u are entitled to express it.

being an engineer i know exactly what the benchmarks worth...imo this thread is useful so please try and refrain from hasty conclusions and do not decide what is "better for everybody". as for the bad vibe u know as well i do who is to blame.:o
[Touch HD Windows Mobile ROM Development] sonydesouza:
Well Mr engineer,

Then you know that whenever you perform benchmarking at always same conditions and with whatever tool, the results are always different...

Moreover, benchmarks are not the only measures...

In addition, bad vibe is a fact...So...

See ya Mr engineer ;)
[Touch HD Windows Mobile ROM Development] pzucchel:
I am sorry, - if people don't understand what a measurement is - it is their problem! any measurement, from the speed of your car to time, takes some measurement error with it. If people or cooks think that a rom is better since it scored 0.00001 better than another one, this is not a good reason to close a thread about objective measurements!

The proposal of closing this thread is nothing more than censorship. Where did you see the bad vibe? may i say that your opinion is at least subjective? may i say that this thread improved many roms (see for example Dutty 3.9) and state "that this is a fact"?

Actually, i tried most of the roms in the list, and i have to agree that actual use impressions in some cases differ from the test ranking. However, if you compare a ROM in the top and lower tier, you will also feel a significant difference yourself! Of course, if the difference is on 3D graphics and you don't try 3D graphics, you will never realize it! but try roms with significantly different read/write performances and you will agree with me.

Essentially, not all the tests performed in this benchmark have the same WEIGHT factor, weight that differs according to the use you make.

If you generate your own profile - e.g. a list of weight for each individual test according to the way you use it, and you re-rank the roms accordingly, you will see that the data in this thread are very indicative....this is what i did in order to try out different roms.