yashsingh1/xdaforums huggingface dataset:
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
TestOpenGL could be a nice addition to my benchmarks, I'll look further in this. Is it only a graphics test? I don't want too much graphics test as Coreplyer also is about grpahics.

I have a question about SPB Benchmark: how often do you perform this test? Because SPB often gives very very different results under the same conditions. Therefore I always test at least 6 times and get the best results out of them. One test is absolutely unreliable...

Check back later about this...
[HD2 Windows Mobile 6.5 ROM Development] houms:
I do SPB Benchmark only once. So for sure, your method is more stable.
[HD2 Windows Mobile 6.5 ROM Development] Xannytech:
You got impossible results for my roms...:D and the same with my turbo rom with blackstone... maybe yours devices does not like my rom? or is something personal? :D:D

I cant belivieve that whaen i did test got similar and also a little better result than Pietrucci's bench and with yours i had the slower rom of the planet :D about OpenGL i use chainfire patch... :D :D and in my hand i got very very different results...

No problem it's the last time i write in a bench thread but it's really impossible :D but i will do an experiment about it in next release .... just to understand something about tests and then i will post my results ;)
[HD2 Windows Mobile 6.5 ROM Development] houms:
sorry and I'll reflash and retry.

Sorry if my results are inconsistent. I'll retry. I'm also surprised about openGL bench results.

Don't yet understood why my method of test is the one described on Daeron site.

Let me retry.
[HD2 Windows Mobile 6.5 ROM Development] Xannytech:
about opengl sometimes happened also to me i did many experimentation... so could be, but the others are really strange, if you want i can send you a beta of my 6.5.3.3 release ;)
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
you could send me one...:p
[HD2 Windows Mobile 6.5 ROM Development] Xannytech:
ok ;) uploading now

i understand why HOUMS had so different results... it's about his sd card.
[HD2 Windows Mobile 6.5 ROM Development] tonband:
thanks again!

Watching this thread closely! very intresting to compare RO benchmarks with your own experience.

Thanx again for your effort!

:D
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
New ROM

Benchmarked new ROM:

maatiROM V3 premium (28014, Sense 2.5.1922)

Also added 3 extra benchmarks in Coreplayer. I take the average result as new Coreplayer index.

See post #1.
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Thank you!
[HD2 Windows Mobile 6.5 ROM Development] Caste23:
Hi,

It's possible to do the same with Miri's ROM v1.1 ?

thanks a lot for your great work !

Caste23
[HD2 Windows Mobile 6.5 ROM Development] houms:
Redone my tests

Xannytech, I've redone my tests and you can find here after, the results.

All my tests was done with the same SD card which is a sandisk class 2 SD card.

I still have the opengl bizar results but globaly I'm very suspicious regarding opengl tests with HD2:

1- For instance for the 1.61 HTC rom I tested, results of opengl test are very good with the pure rom. But as soon as I installed my applications and put my settings, the results are going down and are like the one I mesure on your rom. I didn't succeded to identify which application or parameter bring this slowness in results.

2- For some rom like the Kwbr 1.1.2, I'm unable to run OpenGl correctly because it measures 0 Kpixel/s. If the result is correct, it means that the Graphic performance are so slow that openGl test are unble to measure .

So globaly, I think that your rom don't have any performance issue but that there is a slowness perhaps due to some Energy saving parameters?

Hope all this can help to progress.

About the SDcard, here are the bench (FDBENCH) done on my SD card directly in my computer(in the other zip file). µIs it a low result? Should I change my SDCard? Any recommendation?
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Above all I think you shouldn't do

any

tests on sdcard. I perform all my tests on the device itself after unpredictable results on my sdcards.
[HD2 Windows Mobile 6.5 ROM Development] houms:
thanks

Thanks, I suppose, you do your SPB tests without the Sdcard in the phone. Right?
[HD2 Windows Mobile 6.5 ROM Development] peter7:
Great what and how you are doing it !!! Congratulations !

Don`t stop ...

When will you bench Duty ?
[HD2 Windows Mobile 6.5 ROM Development] mightymn:
just wanted to say your work is great pietrucci! thanks
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Right now...
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
New ROM

Benchmarked Dutty's first Leo: Dutty's Leo Holy Grail V.01 (21887, Sense 2.5.1922).

Fásst, almost (almost...) as fast as XannyTech (99,49 vs. 100%).
[HD2 Windows Mobile 6.5 ROM Development] loomx:
any chance of making this a sticky?
[HD2 Windows Mobile 6.5 ROM Development] killerjohn:
when I look the number of new ROM releases/day I'm sure that you'll be pretty busy in the next... hmm... 2 maybe 3 years? :)
[HD2 Windows Mobile 6.5 ROM Development] giodesi1965:
Please, do you test Lynx WWE edition?
[HD2 Windows Mobile 6.5 ROM Development] :
I think you really only need to do this for a while, until users discover their favourite "Leo Chef", then thay'll probably just stick to that...

Good job on the benchmarks! I think I'll stick to Topix as it seems the best mix of speed/free memory/apps available.
[HD2 Windows Mobile 6.5 ROM Development] nutmegy:
Make it Sticky!
[HD2 Windows Mobile 6.5 ROM Development] jrmccann:
Hi,

Maybe...but people always like 2-3 great chefs and like to compare them, so personally I love benchmarking because...

1. It helps me decide which ROM

2. It helps me decide if the improvement between ROM versions is worth all the time flashing

3. Where the strengths of the ROM lie..

So, personally, thanks for doing this benchmarking...and thanks for doing it often because I look at this page (like I did with the HD Rom benchmarks) more regularly than anything!

I appreciate the chefs and the benchmarkers work!!
[HD2 Windows Mobile 6.5 ROM Development] secoseco:
+1

Hello pietrucci

thanks you, keep this thread always updating as you can
[HD2 Windows Mobile 6.5 ROM Development] Jack E:
Any chance of benchmarking the latest Miri 1.2 ROM?
[HD2 Windows Mobile 6.5 ROM Development] killerjohn:
it seems Xannytech's ROM is updated too. Maybe we can count on an updated benchmark ? :)
[HD2 Windows Mobile 6.5 ROM Development] ctman:
Jup... would be greate

and... ADMIN PLEASE MAKE THIS STICKY!
[HD2 Windows Mobile 6.5 ROM Development] fuchikei:
great work!!!

it's very very helpful!!!
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Wil test a few tonight
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
A few days ago I did a quick benchmark of a beta version of XannyTech EVOLUTION ROM 28205. Although it had some bugs it was rather quick, 98,56% of his Pandora 1.0 (the fastest ROM till now).
[HD2 Windows Mobile 6.5 ROM Development] Jibreil:
Can we have the new V6 PDAViet benchmark test as well? The word is that it's very fast + stable.
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
I'll test it right after EnergyROM 28011 (beta).
[HD2 Windows Mobile 6.5 ROM Development] Jibreil:
Thanks mate :)
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Sorry I mean alpha...
[HD2 Windows Mobile 6.5 ROM Development] chris_ah1:
Would it be more interesting from a bencher's perspective to look at the differing WM builds? Like 21... and 28.... and 23...??

I know Freyberry has a 23... and 28... variant.

sorry i can't remember the full build numbers. And I'm not demanding benches, but perhaps you are curious to see the differing WM builds and how they effect the benches, especially the free memory stats.
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Yes that would be interseting too. But there are so much different perspectives, I can't test them all. In my chart you can see (and compare) the different builds, but I won't test all the variants from every single cook, that's takes too much time from me.

I personally like the 23xxx and 28xxx the most, I would like to only test these ones, but I can't do that because the 21xxx variants still are all over the place.
[HD2 Windows Mobile 6.5 ROM Development] Mikulec:
pietrucci, u should get some helping hands, a few tester that would help you. This benchmarking is really time consuming.

Keep it up.
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
New ROMs

Benchmarked two more ROMs:

EnergyROM 28011 Std M2.5 alpha 06-01 (28011, Sense 2.5.1922)

PDAVIET 2.01.531.06 V06 (23517, SENSE 2.5.1922)

See first post.
[HD2 Windows Mobile 6.5 ROM Development] Ibb1982:
great thread, glad to see the Xanny tech has a top mark :)
[HD2 Windows Mobile 6.5 ROM Development] chris_ah1:
p.s. STICKY STICKY!!!

Oh yeah, could you please explain why Spb is sooo weighted towards the file-system part of the test and why it is so polarised between 1,000 and 2,000? Devices fall into one or the other and 1,000 makes the difference between a 70% and 99%?

I know you like 23xxx but it hasnt come up well because of the storage portion of the test.

21xx has more leads, followed by very few 28xx.
[HD2 Windows Mobile 6.5 ROM Development] freyberry:
In real life, 23xxx is often much faster than all other builds. Benchmark results hardly tell anything about the real life experience.

There are tweaks to boost the file system benchmarks, and some ROMs are using Extendir, which speeds up the file system and gives those high results around 2000 points.

But often it also comes at the price of reduced stability. If I only use some tweaks that don't reduce stability I get a result of 1500-1600 points for my ROM (see my ROM thread), but I can also be sure that there are no bad side effects.

Hope you see my point - benchmarks really can't replace a good real life test, because they can't tell you anything about possible lag or stability issues.
[HD2 Windows Mobile 6.5 ROM Development] tregba:
Seems to make sense as I've just tested the pdaviet v06 rom and it does seem faster than some of the other custom roms i;ve tried. Also, it seems to bench coreplayer much quicker than the other roms going by your comparison despite having the lower spb scores.
[HD2 Windows Mobile 6.5 ROM Development] chris_ah1:
Yes, that is what I suspected - I knew there must be a reason why the storage scores were so polarised. I realise that benchmarks are not the be-all and end-all but it is a good place to help direct the start of a cooked rom search.

the weighting towards the file system DID make me realise that a ROM that scored 70% might NOT necessarily be any slower in real use though.

So as you are a cooker freyberry, I should look to window build first, then to manilla version, then to included apps/tweaks in deciding the cooked rom to try?

I might go to the WM forum now and see which builds are preferred. I had read on here that 23xxx was not very stable albeit quick though.

20xxx ---> 2187 --> 28xxx --> 23xx for slower to faster (more to less stable?) in a nutshell?
[HD2 Windows Mobile 6.5 ROM Development] freyberry:
First, you have to decide if you want WM6.5 (= 218xx) or WM6.5.3 (= 23xxx and 28xxx).

If looks are important, check the screenshots. If certain Manila versins are important, check the Manila version included.

Then you should try it youself. Don't ever dismiss a ROM because of benchmark results, it might turn out to be the fastest ROM you've tried. And of course, speed is worth nothing without stability.
[HD2 Windows Mobile 6.5 ROM Development] band27:
Hey Pietrucci,

I know that page pooling (PP) can play a somewhat role in performance. Is there any way you can ad the PP that chefs used in your spread in the Rom description section. No biggy if it's too much as I know it's quite of bit of work to get this all organized and processed.
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
I guess I could explain the weights. But that's something from SPB, they made this benchmark, they decided to choose these weights.

To give you some more insight look at this page: http://www.spbsoftwarehouse.com/pocketpc-software/benchmark/indices_descriptions.html. I assure you, it took me a lot of time and thinking to get these calculations in my sheet!

I guess you could experiment a bit yourself bij changing the weights in my sheet (for example: set every weight to 1). Then you will see the ranking will dramatically change).
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Yes, sticky would be nice, for all my other devices it always was sticky. But it has been asked a a lot of times before, mods don't seem to like this...
[HD2 Windows Mobile 6.5 ROM Development] appelflap:
On the other hand, there are so many stickies in this forum... Maybe they should consider a sticky subfolder, with of course your benchmarking thread in it!
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
You're right, like in the other threads
[HD2 Windows Mobile 6.5 ROM Development] giodesi1965:
Why not testing?

http://xdaforums.com/showthread.php?t=611405

Bye
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
New ROM

Benchmarked mriri's latest: Miri HD2 wm6.5.X v1.0 (28205/23518, Sense 2.5.1922). I like the ROM, nice interface, not many bugs.

See post #1.
[HD2 Windows Mobile 6.5 ROM Development] Jibreil:
Seven new ROMs for your benchmarking test:

- Laurentius26 v1.0 http://xdaforums.com/showpost.php?p=5312057&postcount=1

- Eugene Beta 3 http://xdaforums.com/showpost.php?p=4809597&postcount=1

- PDAViet v7 http://xdaforums.com/showpost.php?p=5224088&postcount=1

- Karmba_a v1.1 http://xdaforums.com/showpost.php?p=5310694&postcount=1

- Topix 1.1.3 http://xdaforums.com/showpost.php?p=5088602&postcount=1

- Touch-IT Black Leo http://xdaforums.com/showpost.php?p=5222372&postcount=1

- HD2O 1.3 http://xdaforums.com/showpost.php?p=4810416&postcount=1
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
I like you too...
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
I will test itje's and Laurentius for starters.
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
As soon as I can find the time I will make a description how to do my benchmarks exactly. But as I explained before these are very very sensitive tests so I have to describe it very detailed. And that will take some time.

When this works and people want to join we could form a (fixed?) group of testers so we are able to perform more tests. I will add the tester's name in the benchmark results so we know who tested what.

Anyone interested in this idea?
[HD2 Windows Mobile 6.5 ROM Development] axl30:
ok I am interested
[HD2 Windows Mobile 6.5 ROM Development] pathologo:
Thanks for this ROMs comparison!

A few suggestions for the benchmarking:

Take ALL custom ROMS into account. Seems a few are missing.

Benchmark without any today items activated, TURN OFF the slow Manila!

With a task manager close all other processes which are not needed!

This way the benchmarks will be much more accurate!!

Re-Benchmark ALL ROMs as soon as there are new ROM builds out.
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Hello what planet are you from???

First of all if you would read a bit through this thread you would know I simply CAN'T test them all because it takes over 2 hours to test one single ROM. Reason of that is that is have to do several tests, always under the same conditions, to get accurate results.

Then you suggest to disable Manila and close all programs, well that are exactly the things I do. Amongs many others.

Furthermore you suggest these benchmarks aren't accurate, how do you jump to that conclusion considering you obviously didn't read any of this stuff?
[HD2 Windows Mobile 6.5 ROM Development] mracket:
Thank you for all the benchmarks :)

An accurate benchmark is a series of tests that is run the same way on all roms/devices and then compare the results, so I really don't get why some people just can't see that what you do is really the right way to do it. I have been following the thread from the start, and I appreciate the work you do!
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Thnx,

I shouldn't react to posts like pathologo's but hey, sometimes I can't stop myself! :D
[HD2 Windows Mobile 6.5 ROM Development] hmota78:
This is really funny dude :D

Telling a senior member with almost 2.000 posts how he should do his stuff :D (no offense intended :))

But seriously, thanks for your time pietrucci. I know it doesn't only take 2 hours to do the tests, but you also have to download and flash each of the ROMs, which means after you're done with the tests you probably also have to flash back to your favorite one. Thanks again for this. I would also like to ask the mods to sticky this thread.
[HD2 Windows Mobile 6.5 ROM Development] Toss3:
+1 Great job! :)
[HD2 Windows Mobile 6.5 ROM Development] puremind:
Could you indicate in your comparison which ROM' include chainfire drivers? Because if they improve the ebchnarks on some ROMs maybe some lower ranked ROM's are worth considering, as Chainfire drivers can be added.

I haven't read through the full thread, so my apologie if this was already answered.
[HD2 Windows Mobile 6.5 ROM Development] freyberry:
I thought Chainfire doesn't allow cooking his drivers in?

And by the way, new versions of my ROM have been out for a while, with much better benchmark results.
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
I will test a new version of your ROMs soon...
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
New ROMs

Benchmarked two more ROMs:

TouchIT BL 23518 V08-01 (23518, Sense 2.5.1922)

Laurentius26 HD2 V1 Ultimate Series (21877, Sense 2.5.1921)

Laurentius really sets a new reference. His ROM is really really rocking, it has by far the best benchmark results ever. Previous nr. 1 XannyTech is only at 82%.

I wonder when and how this record will be broken... Maybe by Laurentius26 himself hihi.

See post #1.
[HD2 Windows Mobile 6.5 ROM Development] mracket:
Nice work. I do wonder about how it can be that much faster. I had the ROM installed, but it felt slower in sense than both topix and xannytech.
[HD2 Windows Mobile 6.5 ROM Development] OrionBG:
About that!

In Laurentus's ROM thread the benchmarks are a lttle different! Any ideas why?

The biggest difference is the CPU Index!
[HD2 Windows Mobile 6.5 ROM Development] Lokiller:
What about the TOM HD2 v1....

That ROM look very fast on my Leo!!

Thks for your benchmarks^^
[HD2 Windows Mobile 6.5 ROM Development] puremind:
Yes, someone should open a thread on the interface customizations and applications that each ROM includes, because the features and theme are also important aspects.

The overview should contain cover the following items:

Chainfire drivers

12, 16 or 20 shortcurts

BG4all enabled (Y/N)

Main applications: Youtube, FIM, GoogleMaps, Windows Live, etc...

NET 3.5

Number of custom wallpapers installed (Y/N)

Custom taskbar icons installed (Y/N)

Custom start menu icons installed (Y/N)

Extra Tabs enabled (Manilla TV, Navigation Tab)

Hotfixes

Lanscape / rotation coverage

Registry Tweaks covered

Installed Utilities (file manager, registry editor,...)

Overall Theme coolness (give a mark from 1 to 10)

I guess this is too much work for one person to do this and the benchmarks. Perhaps I could create an Excel questionnaire for the chefs to fill in, and then we could publish the results

Anyway it is already great to have some quantitative benchmarks.
[HD2 Windows Mobile 6.5 ROM Development] band27:
WOW! quite a bit going on in this thread...

Any who,

Folks the benchmarks certainly have an important place here as a starting point, when selecting ROMs, but by no means should it end here.

As benchmarks are great, they don't take into account the overall feel when it comes to the ROMS stability.
[HD2 Windows Mobile 6.5 ROM Development] chris_ah1:
Thing is Laurentius canes the benchmarks because of the CPU and mostly the storage bench.

Graphics could be higher...but im sure we can install chainfire ourselves and get more out of it.

Imagine what it would do with nippy 23xx build.
[HD2 Windows Mobile 6.5 ROM Development] derwalde:
guys this benchmarking is awesome. thanks for your work and help! keep benchin'
[HD2 Windows Mobile 6.5 ROM Development] BobaFeTT_NL:
I'm on duttys v01 right now and I'm very curious if anyone can test/benchmark Laurentius v1 with the Chainfire drivers....
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Don't get your hopen to high on Chainfire. Sure it wil be faster but I already did some tests a while ago and it didn't make much difference in benchmarks.

But I'm willing to do a comparison test again with some help from a cook and some time...

Right now there are some other challenges: the new tom, the new dutty, freyberry. And many many others!
[HD2 Windows Mobile 6.5 ROM Development] puremind:
Yes, thanks for keeping up to date. It is the only way the benchmarks can have good significance.
[HD2 Windows Mobile 6.5 ROM Development] chris_ah1:
Lol, easy on puremind. We dont want to burn the poor chap out!!!

I definitely think you should create a benchers group though pietrucci - how is that coming along?

thing with this WM world it all seems so open and development occurs on a daily basis so you are always going to have someone clamouring for one review or another.

There is so much going on - between different radios, XIPs, drivers, ROMs, apps, sense versions, WM builds that this can never be complete enough.
[HD2 Windows Mobile 6.5 ROM Development] wilcovh:
pietrucci

it it worth trying this?

http://xdaforums.com/showthread.php?t=615398

as you are the big speed tester and by me and the community highly appreciated.
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
New ROMs

Two more new ROMs:

freyberry ROM.V5SE 28014 (28014, Sense 2.5.1922)

Tom HD2 WM6.5 v1.0 21887 (21887, Sense 2.5.1922)

See post #1
[HD2 Windows Mobile 6.5 ROM Development] puremind:
Job well done! Tom's ROM ontains best marks almost everyhwere. The file system index is where Laurentius' ROM gets the edge.

Could the file system advantage be due to the SD card being removed during flashing in that particular instance? He is the only one with those kind of benchmark values.
[HD2 Windows Mobile 6.5 ROM Development] SatScan:
SD has nothing to do with bench testing!

Hey There,

SD card has nothing to do with bench speed.

How better the read / write speed, how faster the rom works in pratice.

So the L26 is the fasted around jet for the HD2, on this moment :P

Come on cooks you can do better... can you...

Grtz,

SatScan
[HD2 Windows Mobile 6.5 ROM Development] tom_codon:
Well , Laurentius' ROM used manila 1921xx :)

If he use 1922xx i don't think benchmark can high like in first result
[HD2 Windows Mobile 6.5 ROM Development] freyberry:
Not sure, SatScan. Actually, ROMs with high benchmark scores can be very slow in real life.
[HD2 Windows Mobile 6.5 ROM Development] Laurentius26:
You think and I know. ;):D
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
I agree with satscan that SD has nothing to do with file system speed of SPB Benchmark. Tested is a few times with and without, no differerence. And no, I didn't remove the sd-card on Laurentius ROM.

And Satscan, I wouldn't use that screaming font in my posts. It scares people...
[HD2 Windows Mobile 6.5 ROM Development] pietrucci:
Could be. But as always I also perform my benchmark a few times WITHOUT Manila activated...
[Touch HD Windows Mobile ROM Development] richardnagrant:
I have a problems with my 6.5 Energy Rom, Blackstone is running really slowly(there are no running apps in background). Is there any options how to set factory 6.1 ROM? If yes could you insert link where to download it?. Than you all..
[Touch HD Windows Mobile ROM Development] KayK:
honestly the wiki is there for a reason, you know, and the wiki thread is even a sticky...not to mention there's a search function too...since you has flashed to a custom ROM, i believe you should know how and where to get those resources that you need to restore to a stock ROM...

Blackstone wiki