yashsingh1/xdaforums huggingface dataset:
[Touch HD Windows Mobile ROM Development] kemsou:
Barebone v2.5 performance explication

Barebone v2.5	test by JohnPatcher show that his rom is very great.

The column that makes a huge difference in result is "L" "Graphics test: DDB Bitblt (LESS IS BETTER)".

Explication: The overall performance egal the average of each column performance. Here, column “L”, all the other ROM has "50%" of Barebone v2.5.

Note: Without this column Barebone v2.5 would be top 2. Barebone v2.5 would be anyway the best after "Topaz vanilla final 2009.06.13". Congrats!!

Conclusion: There are no cheat. This ROM is globally better; especially on "Graphics test: DDB Bitblt"

Please, JohnPatcher, Barebone v2.4 & Barebone v2.5 have a very good grade on "L", how does a cook can reach this grade there too?

Kemsou
[Touch HD Windows Mobile ROM Development] johnpatcher:
To be honest, I don't know ;). I've just deleted every crap that I don't need, and added the grahpic drivers available here and increased/enabled all kind of caches.

As my ROMs are unprotected, you are free to look it up ;).
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
@ Johnpatcher - Congrats on your rom going top of the poll.......oddly enough you tested and created it.....hmmm....LOL

Nah, only teasing mate
[Touch HD Windows Mobile ROM Development] johnpatcher:
I've told this in the very beginning, and I know that is not reliable at all. Therefor I invite everyone to redo this benchmarks.

:D, ok ;).
[Touch HD Windows Mobile ROM Development] daeron.be:
New houms benchmarks, enjoy!

It seems that John's benchmarks weren't compatible with houms and mine. I don't want them to confuse anyone so I've deleted them. Nevetherless, Barebone v2.5 does good -> more than 90% is always really good. :)
[Touch HD Windows Mobile ROM Development] daeron.be:
Another small up, M-Amine v3 b2 and newest Miri 6.5 tests.
[Touch HD Windows Mobile ROM Development] purec0de:
6.1 >>>>> 6.5

liking the benchmarks, makes ROM choosing easy (if you're only interested in pure performance).

quick question for the serious tech heads

why do almost all 6.5 ROMs suck compared to 6.1 ROMs. (in terms of benchmarks)

is it simply due to device drivers not up to spec and the fact that 6.5 isn't final or is there really a bucket more payload in there and if so what on earth does 6.5 do with it lol:D
[Touch HD Windows Mobile ROM Development] Laurentius26:
Hi guys,

I do have a question.

Is this test only done on 3D performance?

What about speed of the rom?

My latest 3.1 is very fast but why does it scores so low in the bench?

Screenrotate, opening apps, browsing true the win dir goes very fast, I can't imagine how it could be faster. :confused:

Edit; even avi movies play so smooth, I've been never so happy with a rom and my users confirm.

Grtz, Leo :)
[Touch HD Windows Mobile ROM Development] daeron.be:
Yeah, it's funny that cooks make so many 6.5 roms - they are much slower and on top of that is there really ANY new feature there? And honestly, don't tell me new gui/shell is some unique feature, because there're tons of programs that add new, cool guis for 6.1 and older.

Maybe WM6.1 cooks use is more optimised for our Blackstones than all WM6.5s?

I've added PDAviet rom by PhamQuang, v40 - it's 6.5 and it's at the bottom... Bench by houms ;)
[Touch HD Windows Mobile ROM Development] Urokinase:
Hi.

My only reason for using 6.5 is that 6.5 is much finger friendlier than 6.1 :)

MfG Uro
[Touch HD Windows Mobile ROM Development] johnpatcher:
Not only, you can guess what is done in a test by looking at the name of the column.

There is nothing like "speed of the ROM". This are subjective impressions, which, in best case, can meet the (objective) benchmark results.

That doesn't mean anything, does it :D?

Looking at your benchmark results, you can clearly see that your ROM performs worse in any test. There are plenty of ways to make your ROM performing better. It seems that you aren't using the latest graphic drivers, as you results are a little bit worse than the other ones, although the difference isn't that great at all.

Furthermore you should consider to enable/increase all kinds of file system caches, as your ROM performs really bad on this tests (write, read, copy). This should do the job and make your ROM reach the 90% level.

Using it because it is new could also be a reason, althoug not a very rational one ;). In general I don't think its that bad, but the bad results are holding me off of using it.
[Touch HD Windows Mobile ROM Development] daeron.be:
There're even better guis - spb mobile shell - there's an unlimited trial version and people here usually re-flash their roms really often, I guess always sooner than trial ends. And point ui - there's free version.

Last 4 tests are measuring OpenGL (!!) performance. See http://xdaforums.com/showthread.php?t=511363
[Touch HD Windows Mobile ROM Development] Laurentius26:
Thank you for the explanation, now I'm happy. :)

Did you actualy feel the rom?

Your words are great, so you like to look at numbers instead of objective results. :D

I have to admit I like numbers too but the results are the most important that count for me. ;)
[Touch HD Windows Mobile ROM Development] johnpatcher:
Results like "xyz ROM runs smoother than any other, its best so far" isn't very objective at all. I don't look at numbers only, but I'm very careful when it comes to subjective impressions ...
[Touch HD Windows Mobile ROM Development] Laurentius26:
Me too, but opening a thread on june 22 and currently having 20.000 hits on it with happy users I can not call subjective anymore. ;)

The benchmark is giving people a totally different impression.
[Touch HD Windows Mobile ROM Development] MAMeingast:
If you don't like the benchmarks and think they detere people from using your ROM, you can ask the author of the thread to remove your results - see post #1.

And to say "Your words are great, so you like to look at numbers instead of objective results." is obviously backwards: If there are ANY objective results, they are achieved by the benchmarks used here and definitely not statements about the way the ROM "feels". Don't get me wrong: Benchmarks are not everything - but they are the most objective criteria we have.

Also, to rely on the number of hits on a thread as a criteria is not objective at all... The number might reflect the popularity of your ROM, but surely not it's speed - just because the majority of people THINK something is fast doesn't mean it actually IS fast...
[Touch HD Windows Mobile ROM Development] duttythroy:
My first post regarding benchmarks here;

I always do benchmarks since cooking with kaiser ,and although its a number and sometimes doesnt reflect on the speed of the rom, I use it as a guide to build all my roms.

My v3.9 release I knew was fast and knew it would also do well on benchmarks although I got a better results posted, My v4 is also faster than v3.9 carried out by same benchmarks.

Regarding benchmarking Roms here on XDA, we have to give thanks to these guys who does them but to make it more concise and results reflecting to respective roms e.g lite, barebone, full would need to be categorised seperately including 6.1 AND 6.5 ROMS.

A lighter rom with most stuff remove should be faster on paper but not at all times e,.g benchmarks on first post,

But to have a fair benchmark which is unlikely we would have to test every rom with same amount of apps included which is unlikely.

So respect to the guys doing the benchmarks, I always do my own and most times its reflected by results shown above but if thier is a posibilty of categorising the results for wm6.1 and wm6.5 and full roms and lighter rom might reflect better on all chefs.

my 2cents:)
[Touch HD Windows Mobile ROM Development] Laurentius26:
Hi Dutty,

Thank you for clearing this up.

I was looking for some answers and I felt like defending my rom because a lot of time trial and effort goes into this and the results are a bit misleading for public to my opinion.

What you are actualy saying is that these benchmarks are inaccurate because light and full roms are cooked and it's impossible to measure because of these differences?

My apologies Dearon for opening this up, you are doing great work but if you like you can remove my results.

Regards, Leo
[Touch HD Windows Mobile ROM Development] michoob:
Two previous posts = two great cookers

It's good to have great cookers discussing on this thread friendly, because it can help to understand how we can interprete these benchmarks.

If all goes friendly, it can only go to the best!

Thank you Dutty and Leo, I enjoy your ROMs very much!

And thank you to all the cookers of ROMs I have not tested, because thanks to you, step by step, our Blackstone is becoming greater and greater without any hardware change (you see what I mean...).
[Touch HD Windows Mobile ROM Development] SatScan:
Yes we can

I AGREE !!!

Count me in!

Grtz,

SatScan
[Touch HD Windows Mobile ROM Development] daeron.be:
Woops, double post =X
[Touch HD Windows Mobile ROM Development] lynne02191:
Actually, I think the benchmarks imo is really helpful,

I agree that sometimes that the benchmarks may not that accurate due to some reasons,

however, I've tried several roms on the top of the list,

and the results are indeed relatively better,

so, thanks to daeron.be and houms and those who are contributing to the benchmarks :)
[Touch HD Windows Mobile ROM Development] binhk88:
Can you bechmark for this Rom:|BLAC|6.5|ROM|WWE| ★ EnergyROM HD (21820) ★ || Build date 2009/6/25 |?
[Touch HD Windows Mobile ROM Development] bobsbbq:
Finally others are seeing what I have been stating since the beginning of this thread. Very well Stated Dutty. These are the reasons I asked for my ROM's to no longer be included, Poor organization and lack of accurate ROM information.

Debaeron Sorry, but if you think WM 6.1 and WM 6.5 are the same you need to quit. The very fact the WM 6.5 is still a beta should put it in a different catagory. WM 6.1 has been around for a long time and is much more optimized to the drivers and hardware and is naturally going to perform differently.
[Touch HD Windows Mobile ROM Development] johnpatcher:
Thanks, but I'm just wondering why my "alpha texture" test is that bad :(. Running it myself I get something around 37k, just as the others do. But anyway, I'm happy with the results ;).

Btw: The results of benchmarks of light ROMs are much more realistic than the ones of "heavy" ROMs. While they might perform very similiar with all today plugins disabled (and airplane mode), its quite obvious that they will perform different when all of the included stuff is activated. So while light and heavy ROMs are mostly the same in the benchmarks, they will certainly be different in "normal" use.

He never said anything like this.

The results just show what you stated above. But that isn't any reason to put it in an own category, because the results should be as objective as possible. Its just a matter of fact that WM 6.5 isn't that fast as WM 6.1 yet, and the results just show that.
[Touch HD Windows Mobile ROM Development] KukurikU:
what a BS!! yes, 6.1 is older, yes 6.5 is beta but why not compare?

so, the older 6.1 is faster than the beta 6.5!! a cow is bigger than a sheep, a cougar is faster than a dog, an elephant is bigger than a giraffe but the giraffe is taller than the elephant...(watched too much National Geographic recently:D) why not compare???!!
[Touch HD Windows Mobile ROM Development] onkolog:
daeron.be

I give you my results from my new WM 6.5 FULL ROM (version 1.2,June 26);)...

You can add it if you want in your list...

And please tell me how do you know overall performance for each ROM?

Thanks...
[Touch HD Windows Mobile ROM Development] bobsbbq:
Sorry John he did say there was no difference between the 2 platforms and that simply is not true.

Ok I think people are not understanding mine and others point of view. When I said different categories I mean they should all be listed as WM 6.5 ROM maybe by means of a additional column to designate 6.1 or 6.5 and or Placed all WM 6.5 in their own section say at the bottom and arrange them as per performance in their section. Again this is My opinion and just as their are many different ROM's there are different opinions.
[Touch HD Windows Mobile ROM Development] daeron.be:
Fine, I've added your results. And yeah, they changed the table a bit. And your rom is in very top. If your test is accurate it seems, that 'core' you used for your rom is the best around. But your opengl performance seems poor - what graphics drivers have you used?

Please, give me any difference between 6.1 and 6.5. Apart from gui and peformance, because different 6.1 can differ in gui, performance, other drivers and compatibility too. I don't see a reason for separating those OSes. 6.5 is like 6.1 with different name.
[Touch HD Windows Mobile ROM Development] bobsbbq:
To start with it is a Different build SYS (OS) and XIP. Yes they look the same other than the gui but so does a Camaro, one with a 327, and one with a 454 looks is not all where it is. All we are saying is that your list contains a mixture of both and you don't specify which are which. Take it as some constructive criticism to better your project. With them separated you can have the best of both, There are on the same sheet so people can compare the results, you also can have ROM's in the WM 6.5 section arranged according to their performance so a person could see if they are looking for a WM 6.5 ROM which one might interest them.
[Touch HD Windows Mobile ROM Development] Laurentius26:
Yes that's my opinion, obviously you didn't read post #1 because the author doesn't like to remove my results.
[Touch HD Windows Mobile ROM Development] daeron.be:
As requested by many, I've added a new column with 6.1 or 6.5. Some of roms have this cell empty, because I don't remember which version it was. Correct me if I've left some places empty or version is wrong.

But anyway, I think there's no significant difference between 6.1 and 6.5. There's difference between 5 and 6, that's true...

BTW: One rom (EnergyROM 20090625 Standard) have failed one test. It doesn't HAVE to mean this rom is worse than the rest because of them. It shows there's some incompatibility between SPB Benchmark which failed and this rom. But maybe you won't experience such incompatibilities in your daily usage. Cheers. :)
[Touch HD Windows Mobile ROM Development] Major_Sarcasm:
I'm curious as to why onkolog's 6.5 ROM (v1.2) appears so much quicker (on paper) than Miri's 6.5 ROM (v20) with the same 21820 WM build...
[Touch HD Windows Mobile ROM Development] daeron.be:
If I'm not wrong Onkolog has written it's 21820 & 21051. And that's your answer. And that's really misleading. :)
[Touch HD Windows Mobile ROM Development] bobsbbq:
The Arkball test is 3.78 is less than half or the result is 50%+ better than any other ROM even 6.1 this is likely why it's placed higher. I'm sure debaeron will re confirm the results so there is no speculation in cooks sending in their own results.
[Touch HD Windows Mobile ROM Development] MAMeingast:
Sorry, I mixed it up... I actually was thinking of this post by houms: http://xdaforums.com/showpost.php?p=3943827&postcount=177

"[...] Then, yes you are right, if some cookers do not want to be measured, I respect that so I'll no more publish measurement about your rom. [...]"

Of course this was not stated by daeron - but in this case I'm on your side: If you don't want results posted, daeron should respect that wish and remove the results!
[Touch HD Windows Mobile ROM Development] daeron.be:
Ekhm, It's Daeron. :)

Users have right to test roms and compare them with each other in any way they see it fit. I'm using this right and if you think your rom should be 1st, go and polish it more. I'm talking to you, Laurentius26. My test is there for people to see a difference between all roms available for Blackstone on xda-dev.

You say that they MISLEAD people. That's not true. They do the exact opposite, because they show people this rom's performance in true colors. Choice is theirs, anyway.
[Touch HD Windows Mobile ROM Development] Laurentius26:
Hi Daeron,

Great words but not objective at all, it's your own personal opinion.

You totaly ignore the cooks who have some experience as well don't you think so?

Again you are misleading the public here.

Applications don't they use dlls exes Daeron?

My apologies I have to bring this up one more time but you are ignoring my request to remove my results so I do have the right to defent my rom isn't?

Regards, Leo
[Touch HD Windows Mobile ROM Development] daeron.be:
Ekhm, where I wrote apps don't use dll and exe files? What's my personal opinion again, which mislead people? Point out what isn't true.

It's obvious that applications in Windows are generally executables (exe) files, unless they're plugins to other apps, etc.
[Touch HD Windows Mobile ROM Development] kwbr:
Ok guys, relax. Leo, nobody is bashing you or your rom. So I see no point why you should defend yourself or your rom. I agree that you can add categories like win6.1 and 6.5 since we all know 6.5 has a base which is slower than 6.1.

A barebone rom does not always have to be faster. See my topaz port which runs faster than damn small topix at the moment. And it still has a lot of apps included.

Furthermore, let's keep opinions objective. I don't see what pagehits have to do in anyway with the quality or speed of a rom. My experience is that the more a rom is discussed the more, let's say features or improvements, have to be discussed. Let's see this positive. You can approach it in a way what can be improved in your own rom. Yes every rom can be improved :)

Some comments on speed, it is well known that caches, the number of files you keep in \windows, pagepool size and a lot of other things will impact the speed of your roms. Challenge yourself to get the most out of it. And above all, don't take things too serious and have fun in what you are doing :)
[Touch HD Windows Mobile ROM Development] daeron.be:
I agree with you. But hey, you say that number of files on /windows impacts speed. Impacts what? And why? Let's be concrete. And it can be tested in a scientific way, not by chit-chatting.
[Touch HD Windows Mobile ROM Development] kwbr:
Nice tone of voice in this thread :)

This is already proven. Use Google, search xda or cook a rom and test and you will see :)
[Touch HD Windows Mobile ROM Development] johnpatcher:
Since I never tested it, I can't say this for sure, but I have experienced this also. I think that any file in the \Windows directory has to be cached on the one hand, and on the other hand it takes away some space of flash, and as far as I know flash storage gets slower by putting more data on it. But, this isn't scientific at all, so if anyone has some links, or explanations, it would be great to hear them.

I don't know who you addressed with this, but my point made above was that the benchmark of a "light" ROM is more realistic than the ones of "heavy" ROMs, as the benchmark is made with quite the same preconditions, but in reality there is much more stuff running in the background in a heavy ROM (Manila, Icons, Today Plugins, whatever).
[Touch HD Windows Mobile ROM Development] Laurentius26:
You want me to point out?

First you say the number of apps do not influence the speed of rom.

Second you say dlls and exes do influence.

So one more time tell me do apps use dlls and exes or not?

Edit; so in that mather it does influence the benchmark results.

I will stop it now, but I'm not happy what is happening here.
[Touch HD Windows Mobile ROM Development] kwbr:
Right John, running processes and the way you organize your filestructure (where files and programs are installed) influence the performance of a rom. Golden rule is to keep as much files out of \windows because some (system)programs use this folder and access (read/writes) slows down when this directory is packed.
[Touch HD Windows Mobile ROM Development] michoob:
Why are you getting mad Laurentius?

Of course those benchmarks can't show everything, and of course the worse benchmarked ROM may even be the best on a daily common use.

I don't think that daeron pretends to be an expert in ROM benchmarking, and he is certainly opened to include other benchmark criteria in the list, if they can be measured somehow.

It will be more constructive to find other ways to benchmark a ROM than to attack daeron for not benchmarking all parameters of a ROM.

And perhaps you will come with an another ROM that will be in the top without any real performance improvements on the user side, but perhaps it would help to investigate some better benchmarking methods :)

EDIT : by the way, I agree with donzx6's post below.
[Touch HD Windows Mobile ROM Development] donzx6:
Some interesting views.

Having flashed several roms (both 6.5 and 6.1) I love the work that the chefs do and they should be thoroughly thanked (and donated to if you love the ROM).

I also respect the benchmarking exercise, but the reader (not the benchmarkers) need to realise that benchmarks are only that.......

To make an analogy with the auto industry, you may have a car which comes top of the benchmarks for power, reliability, fuel consumption, tyre wear etc, but if the chassis is poor, the car cannot get the power down and therefore may be quite slow on the track.

A balanced car whereby the chassis is matched to the power, tyres yada yada would prove to be quicker on the track.

So, please everyone, realise that these benchmarks are just that.

ROM's are more than just the sum of the parts, they need to flow and be balanced.

I think that's what Leo was getting at.

Don't forget (and I'm sure I'm not alone here), I load up lots of apps after flashing and lots of customising.

That is also bound to affect the ROM's performance.
[Touch HD Windows Mobile ROM Development] Laurentius26:
It's not that I'm getting mad or attacking Dearon but I'm only looking for some answers.

I've asked Dearon friendly to remove my results because to me these results are misleading and causing people to think the rom performs bad while it actualy is oposit to my opinion but he ignores my request.

Daeron is doing these benchmarks but I do have the right to disagree isn't, same as people have the right to complain about the bugs in rom in my rom thread.
[Touch HD Windows Mobile ROM Development] Laurentius26:
Ok thanks, for example the more apps in rom the more cache it uses isn't?

Also more services running in the background affects the performance.

If you do a benchmark with Manila and radio off it doesn't say anything of the actual performance to my opinion.

Your benchmarks asuming the rom performs very good but when you use the rom it can perform a lot different.
[Touch HD Windows Mobile ROM Development] squirreleater:
My thoughts.

Nice idea to do some benchmark tests, but the second a Rom cooker asks for his Rom to be removed it should- no questions.

They spend a lot of time doing these roms for us to use for free, and we shouldn't be upsetting them at all.

I also agree that Roms that are slow according to the tests actually don't feel any slower than some 'fast' roms.

If they want the Roms removed-remove them, don't argue and upset them....... they may just stop cooking roms for us!