yashsingh1/xdaforums huggingface dataset:
[Touch HD Windows Mobile ROM Development] fredphoesh:
Hi guys,

I have only been trying out these ROMS for a couple of weeks, but I am surprised at the results. I tried two recent versions of Dutty's and find the current v25 of Miri noticeably faster than both the Dutty roms... particularly in navigating around SPB Mobile Shell's interface... it animates more smoothly.

So not to put the tests down, but based on my limited experience, I would not make any rom choices based on these tests.

tx

Mark.
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
As previously stated in this thread these results are only based on certain requirements/tests and can be easy manipulated by chef so his rom comes out top. Best advice would be to try rom and make own conclusions. These benchmarks are only a starter for guidance to help steer you towards certain roms
[Touch HD Windows Mobile ROM Development] daeron.be:
Sorry... You sounded like results here can be manipulated. They aren't manipulated and they are accurate (look at Miri roms, they all have similar performance. Look at Kwbr's, Dutty's or Laurentius'es, they most of them score similar because they didn't change in some version much).

These tests simply test how fast rom/phone does some operations and measure time. That's all. For example, how fast it would list files in a certain directory or calculate some floating point stuff. ;p

If by "manipulated" you meant for example disabling something like active sync, which might usually lead to a great improvement in listing tests, you are right. But some people, like me, doesn't use active sync, so I don't need it to be active...

Well, cooks manipulate roms anyway, it's called "cooking" by people around xda-devs. :) But I believe they don't manipulate results. Especially, if they are made by me or Houms. ;)

Nobody ever paid me for this job, but if you want, you're welcome ;D.

Well, we don't have any test measuring SPB Mobile Shell's interface speed... That might be possible, this performance thing is a complex issue. I'd say it's a placebo effect... But it's not a bad thing, as long as it works. Miri's roms have something in them. Even if it's just popularity of Miri's thread, that's pretty much anyway.
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
I don't mean results where manipulated, I mean a chef can modify his rom to perform well in these benchmarks (like you said with activesync) but when tested in every day use may not be as good as others which came beneath it in the benchmarks. I'm not saying they do this though.

I think what your doing is great as it gives people a guide they can work from ;)
[Touch HD Windows Mobile ROM Development] daeron.be:
Thank you. :)
[Touch HD Windows Mobile ROM Development] !Aman!:
i don't cook for HD, but this might of high interest to some of the chefs, please check my post here
[Touch HD Windows Mobile ROM Development] tnyynt:
If my logic is right, this means that cabbing the selected values and installing might fix ROMs that suffer from Sync failure issue?
[Touch HD Windows Mobile ROM Development] !Aman!:
cab won't work, it is something related to XIP and boot.hv. if someone can make a Hotfix like HTC do, then it should work.

PS: i hope some of the unanswered questions in this very thread are answered now ;)
[Touch HD Windows Mobile ROM Development] daeron.be:
I've added new tests by Houms:

Dutty's 6.5 R8 Turbo

Black_Pearl_v3.4_WWE

EnergyROM 2009.07.23 Standard

PhamQuang v49 b23009 2009.07.22 BLACK
[Touch HD Windows Mobile ROM Development] daeron.be:
I've added a poll to this thread, please take part in it. :)

And maybe this thread should be sticky? I wonder...
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
Good poll. Would be interested to see how results end up!

Keep up your good work mate ;)
[Touch HD Windows Mobile ROM Development] KukurikU:
please remove my sticky and replace it with this thread. my sticky is not actual anymore!
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
I've made this thread sticky and removed KukurikU's thread '[FiXES and tweaks for the new ROMs] 22 fixes and tweaks' as a lot of these have become obsolete or are already embedded into newer roms. Thread is still available to those that care to search!!
[Touch HD Windows Mobile ROM Development] bobsbbq:
Now that is one of the smartest things I have seen in this thread. Kujuriku not taking anything away from what you have done as it has served and will still serve as good information, if people search. But the fact is as newer builds are emerging the some of the fixes are getting fixed.

Now I still don't agree with some of the things in this thread, but everybody does not think the same, and we all differ on opinion. All I was trying to do was better the process as a cook and maybe share some knowledge, but it didn't work out that way.

I'm going to share one more:

Daeron, there is a problem with many of the test results with the Arball game test. And I feel it is important to address the issue. Some devices especially the ones with 6.5 platform ( I can only speak for what I know) have a problem running the test. When the test is started there is just a white screen and the game never actually runs and it gives a false result of less than half of what it normally would (I'm estimating). I have found out through investigating that one of the causes for this is modified XIP, in paticular to the OEMXIPKernel. A original unmodified XIP will run the test with proper results which fall in line with others. The ones modified especially with removed packages and or SpinLock Patched and Nue gxdma will cause the test to not run properly.

Now this might be ok if everyone used the same mods in there XIP but as we all are different and cook differently that is not the case. How you would fix it? I don't know it is up to you.
[Touch HD Windows Mobile ROM Development] daeron.be:
Honestly, I don't know what to do about such problems. It's obvious that those tests aren't ideal. Testing each rom better would take a lot of resources, which I don't have. Lately, I don't have time to mess with my phone, and as you can see Houms is doing all the testing work (thx).

There are even more problems out there, for example, current first rom - Laurentius26's 3.5, is causing problems graphics in NFS:Undercover game. What should we do about that? Disable all tests? Add "problems&bugs" into our spreadsheet and ask people to send bugs they experience with certain roms to note them all?

There's a way to 'solve' Arkball test problem. We can put "fail" into each Arkball cell of each rom, that's affected by this problem, and delete Arkball test from overall scoring cell of that rom, like I've done in some other rom, which had some incompatible test too... But I won't do this unless you give me names of roms affected, which is a tough indeed...
[Touch HD Windows Mobile ROM Development] bobsbbq:
That's a fair statement. nothing against Leo but the drivers he is using are creating faster result but with problems as you said. I tried the same drivers and got much better results on test but NFS would not hardly run. So this is another reason these test can be some what misleading, but I guess people can find out for their self. Any way threads like this require time to be done right, I myself have had little time lately.
[Touch HD Windows Mobile ROM Development] Fedron:
Battery benchmark

Hey Daeron, nice work man, I really think this thread is a good thing. As said in the poll, I check the results of the benchs but after that I do what I want to do, but hey I like to check :)

Anyway I did not posted to talk about that, I just wanted to ask if it would be possible to add a battery bench. Speed is nice but lasting is even better. Battery life is an important data when you want to choose your rom.

Here is a nice example done by Ikkari:

http://xdaforums.com/showpost.php?p=4149733&postcount=23

I did not searched this thread to check if somebody already asked for that, if it's the case just count my post as a +1

Thanks for your job, it's very much appreciate.

Cheers.

:)
[Touch HD Windows Mobile ROM Development] cadenza:
yeah, sticky^^ thats great, I love this thread, although I voted also for the second option:) But, I know its been asked, I just wanted to ask once more, isn't it possible to create two different results, one one for 6.1 ROMs and one for 6.5 ROMs? Isn't it just copy&paste "work"? Sorry for this question..

Anyway, keep up, its great to have this benchmarks!!

Many greets, cad^^
[Touch HD Windows Mobile ROM Development] daeron.be:
No. :)

You can sort OS column or export spreadsheet and modify it as you like

As about battery tests - sorry, but I don't have enough time to play with that. I'm afraid I'll even have to stop working on this project much and only add results from Houms or other volunteers. Sorry for that.

Added:

Epiphany HD TouchXP v1.0 Extendir (first Extendir rom here, very promising)
[Touch HD Windows Mobile ROM Development] mrobo:
Benchmark Dutty's HD WM6.5 R8 Turbo ROM + MSVC

Posting my benchmark test results I have flashed Dutty's HD WM6.5 R8 Turbo ROM + MSVC and here are the results

SbpBenchmark:

</fs-largefilewrite>

<fs-largefileread>

32.900000

</fs-largefileread>

<fs-largefilecopy>

634.300000

</fs-largefilecopy>

<fs-manyfileswrite>

4432.800000

</fs-manyfileswrite>

<fs-manyfilesread>

245.100000

</fs-manyfilesread>

<fs-manyfilescopy>

2850.700000

</fs-manyfilescopy>

<fs-dirlisting>

322.800000

</fs-dirlisting>

<fs-db>

768.200000

</fs-db>

<grf-ddb>

13.375000

</grf-ddb>

<grf-dib>

34.700000

</grf-dib>

<grf-gapi>

1.245000

</grf-gapi>

<misc-zip>

2714.600000

</misc-zip>

<misc-jpeg>

410.700000

</misc-jpeg>

<misc-arkaball>

3.673500

</misc-arkaball>

<misc-mflops>

3476.400000

</misc-mflops>

<misc-mops>

1007.600000

</misc-mops>

<misc-mwips>

7138.400000

</misc-mwips>

<misc-memcpy>

8.517000

</misc-memcpy>

</main-tests>

</benchmark-results>

TestOpenGL:

Results

--------------------------------

alpha texture (128x256 rgba)

17806 kPixels/s

torus (3k triangles) one light

6380 kPixels/s

color blending

18765 kPixels/s

sub texture (128x128 15x15)

8694 kPixels/s

OpengGL ES info

--------------------------------

Vendor

QUALCOMM, Inc.

Version

OpenGL ES-CM 1.0

Renderer

Q3Dimension MSM7500W 01.02.03 0 5.1.2

Extensions

GL_OES_byte_coordinates

GL_OES_read_format

GL_OES_fixed_point

GL_OES_single_precision_commands

GL_OES_compressed_paletted_texture

GL_OES_matrix_palette

GL_OES_point_sprite

GL_OES_point_size_array

GL_ARB_vertex_buffer_object

GL_ARB_texture_env_combine

GL_ARB_texture_env_dot3

GL_ARB_texture_env_crossbar

GL_ARB_texture_mirrored_repeat

GL_EXT_stencil_wrap

GL_EXT_blend_minmax

GL_EXT_blend_subtract

GL_EXT_blend_func_separate

GL_EXT_blend_equation_separate

GL_ATI_texture_compression_atitc

GL_ATI_extended_texture_coordinate_data_formats

GL_OES_draw_texture
[Touch HD Windows Mobile ROM Development] daeron.be:
Please, send them through email. Believe me, I'll give you credits in a spreadsheet as a person who ran this test. :) Posting full files here pollutes this thread a lot.
[Touch HD Windows Mobile ROM Development] KukurikU:
anxious to see the KWBR's 2.8 benchmark results. after using it for a few hours i think we have another "top one".:)
[Touch HD Windows Mobile ROM Development] daeron.be:
Folks, there's some nice update for you - tests of

26 Miri roms Premium and Premium with Manila 2.5,

Final 6.1 rom from Dutty (4.1)

Newest Energyrom

all done by Houms, and because I was as anxious as KukurikU I've tested quickly newest Topix 2.8 and yes, there's improvement over 2.7, see for yourself! :)

To mrobo: I was trying to add your results anyway, but you've forgotten to send us <fs-largefilewrite> value. Send them via email again plz!

EDIT: Epiphany send me results of one of his roms. I've added them, and you can see them for yourself. If you feel that's unfair to include cooks benchmark, do one yourself and check, if he has cheated or not. I believe he didn't, that's why I've included those.

EDIT2: Updated OP slightly.
[Touch HD Windows Mobile ROM Development] tossero:
Hi! Could you test Duttys 6.5 v9 Turbo / MSVC ? Thanks ! :)
[Touch HD Windows Mobile ROM Development] EpiphanyNL:
Hi daeron,

Thanks for adding the results I send you. Thanks for the trust in me that I didn't cheat, I kept to the procedure you described in the OP.

If anybody wants to verify the results they are free to do so.
[Touch HD Windows Mobile ROM Development] daeron.be:
Update from Houms:

Miri 6.5 Premium b23017 Manila 2.5 REV6.1

Miri 6.5 v26.1 Premium b23017

Topix 2.8.1 regular

Dutty's 6.5 R9 TURBO

Enjoy.

BTW: In 3 days I'm going on a holidays and I won't update this thread for about a week. Sorry for that.
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
No worries, enjoy your rest, I'm sure everyone will be able to survive!!
[Touch HD Windows Mobile ROM Development] KukurikU:
i don't remember seeing yr request for a holiday on my desk. now is too late so consider yr request denied !!:D
[Touch HD Windows Mobile ROM Development] aminsa:
Hi everybody,

To me the most important thing when selecting a ROM is -- look for a less buggy ROM. Better performance is nothing/useless if it got a lot of buggs. Soooo I think we should have another index/table that show/list the bugg that certain ROM have that is already confirmed by the cooker/users. This is just my opinion, if no body is interested than just forget it. Right now I'm using onkolog ROM 6.1 because I don't need HTC interface or Microsoft 6.5 I like spb software.
[Touch HD Windows Mobile ROM Development] Oemie:
In the poll i answered "i use the rom even if it is not on top of the benchmark list". After two weeks of using one of the highest ranked 6.5 rom in the list Dutty r9 i would make the statement the ranking is really of use. To have a faster rom (and the highest ranked rom is much faster in daily use) is really great. Everything is just snappier and smoother. Copying a file is so fast it is a real joy to see such a high kb/sec...

So to everyone try the highest ranked rom and enjoy the difference..

To bad i will never go back to windows 6.1 again, because if you are used to 6.5 you just can't go back. When you are used to it, it is like going back from windows7 to windows98 and i don't want that...

Cheers and have fun flashing:D you addicts;)
[Touch HD Windows Mobile ROM Development] MAMeingast:
Please enlighten me: What is so great about 6.5? I tried a 6.5 ROM a few weeks ago and it didn't impress me that much... But maybe I missed the great improvements that have convinced you! ;)
[Touch HD Windows Mobile ROM Development] tossero:
Hi all ! Can anybody test Dutty`s 6.5 R10 and Leo 2.5 test 1 ? This will be just great .. :D

Thank you in advance !
[Touch HD Windows Mobile ROM Development] Oemie:
It's the fingerfriendlyness (if that is a word at all). I have never used the styles (never from for me day one with windows mobile 2003).

For instance when you copy or move stuff to another place in 6.5 you have all the space for your finger to choose the right location to put it..it is a matter of a second because you don't have to am like in 6.1..then there are for instance

All the menu, the startmenu..the scrolling..and

I hate the old 6.1 calender that is still in the 6.5 roms. Also in windows 2003 an 2003se i never used the stylus, and used my fingers instead.

So maybe i nevber go back because i hate the stupid styles:D

It's a matter of taste and what you are used to, but i definitely never go back.

In a month or so i hope manila 2.5 sms wil be bugfree or it least workable and then i will be the happiest nerd on earth because from that time i am rid of the stupid stylescalender also:rolleyes:

That's all.
[Touch HD Windows Mobile ROM Development] MAMeingast:
Thanks for your notes on 6.5!

Since you're pretty much only talking about the fingerfriendlyness (nice word-creation there...) 6.1 is still fine for me... Using the right apps, 6.1 is just as fingerfriendly! ;) I'll wait for a bug free release... :D
[Touch HD Windows Mobile ROM Development] fredphoesh:
its a lot more finger friendly than 6.1
[Touch HD Windows Mobile ROM Development] tk_venkatesh:
On Updates.....????

Looks like this section is not updated time to time ..... can anyone please update the benchmark of duttys 6.5 9 and 10 ......
[Touch HD Windows Mobile ROM Development] EpiphanyNL:
Daeron is enjoying his vacation
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
Quote from daeron one page back. Not to hard to search :rolleyes:
[Touch HD Windows Mobile ROM Development] MAMeingast:
And by the way... No one is keeping you from doing those benchmarks yourself! ;)
[Touch HD Windows Mobile ROM Development] aminsa:
To me friendly means 'less number of touch/click to run an applications or to do certain things' and of cause the responsiveness. By running spb pocket plus I could reduce the number of touch/click to accomplish certain task.
[Touch HD Windows Mobile ROM Development] daeron.be:
MAMeingast send me results of his test of Topix 2.8.6 Regular by Kwbr. I've included it into the spreadsheet. If you want to see any rom here, consider testing it by yourself and sending me results.

Thx MAMeingast! :)
[Touch HD Windows Mobile ROM Development] MAMeingast:
No prob! The least we can do is contribute if we're flashing a new ROM anyway... ;)
[Touch HD Windows Mobile ROM Development] mcurotto:
Hello daeron

Nice work with the benchmarks. It helps me a lot deciding wich rom to use.

I only have 2 suggestions:

- Remove the last 4 SPB Benchmark indexes (CPU and MEM) as they are software independent (if I´m not mistaken), so you reduce normal variations.

- Include a battery index, (don´t known wich one is the best), even not making part of the final grade.

Tks
[Touch HD Windows Mobile ROM Development] daeron.be:
Also, I'd like to thank Houms for his new tests of

Topix 2.8.6

Miri b21054 Manila 2.5 v1.1

Miri b23017 Premium Manila 2.5 REV6.2

And right now I'm checking performance of Dutty's 6.5 R10 Leo rom. Well, I don't think results will be any good, but rom alone is really awesome. :)

EDIT: After a few seconds of consideration, I decided to make some slight changes to a spreadsheet and a whole testing project (aka revolution). Stay tuned if you want.
[Touch HD Windows Mobile ROM Development] andes83:
Hey that's interesting is it a new way of testing ? I can't quite understand the reults but it's somekind of overall value of the ROM or is it not?
[Touch HD Windows Mobile ROM Development] daeron.be:
Sorry, but I've messed up overall score index for a few hours, but now it's working again. I've done some changes:

- added 'build' column, because there were quite a few people requesting that in this thread and because there are already builds noted in many rom's names, I've added that. There might be problem with completing this column though... :)

- rom creation time column (look above)

- running today programs during test column, no disabling manila anymore, if you like one! :)

- comments of a tester (if there are any...) column

- rom archive password and rom archive links (mirrors) (I'm leaving this feature for discussion here. What do YOU think about that?)

Well, there's a lot to edit right now, because the spreadsheet seems very empty :). Therefore I've (finally) given permission to edit the spreadsheet to houms and I'd like to offer the same thing to MAMeingast, if he wanted to contribute to this project. Also, if you want and prove yourself worthy, I would give you the same permission too ;).

And yes, sevene PMed me an idea to add coreplayer benchmarks of matrix trailer (to test 3 different versions):

http://shintak.info/ppc/download/betaplayer/sample/RL_XQ_640x480_1500_128.avi

http://shintak.info/ppc/download/betaplayer/sample/RL_HQ_640x352_1024_128.avi

http://shintak.info/ppc/download/betaplayer/sample/RL_MQB_320x240_512_128.avi

But I won't do that unless somebody volunteers to help me with it...
[Touch HD Windows Mobile ROM Development] MAMeingast:
I'd love to help! :)

EDIT: I also think that adding a download link and password is a great idea! But I don't know if all the chefs will like this... ;)
[Touch HD Windows Mobile ROM Development] daeron.be:
Hm... if we are that complex we could add rom's thread link too.

EDIT2: I've added weights used to count how much the test is important when calculating overall index.

EDIT3: Dutty's R12 :)
[Touch HD Windows Mobile ROM Development] MAMeingast:
Update: Just added a lot of information in the new columns...

Although it was easy to find the info for the newer ROMs (still available on the forum) and the ROMs by Miri (good documentation in thread) and Dutty (all versions on his website), it'll be harder to find the info for older ROMs... ;)

EDIT: I also cleaned up the naming of the ROMs since they were sometimes very inconsistant... Although it's not perfect yet, it's a lot better now! :D
[Touch HD Windows Mobile ROM Development] Niksie:
I see that you've added the coreplayer results for Dutty's R12.

Correct me if i'm wrong, but the results of the Matrix High quality is according to your test: 141.60%.

I've tested several roms (Miri's, Topix's) but they never exceeded about 109%

Are your results correct and is Dutty's R12 REALY that fast? Or did you make a tiny mistake??

Realy appreciate your work.
[Touch HD Windows Mobile ROM Development] Oemie:
I don't get it anymore, why is Dutty's R12 on 3th place? The roms above him and beneath have much better scores in 90% of the tests??

How do i read this benchmark ranking from now on?:confused:

I like to know cause i am on Dutty's R9, and when R12 is that fast i am going to flash right away:D
[Touch HD Windows Mobile ROM Development] MAMeingast:
Daeron has introduced weights to differentiate the importance of different scores... Tests with a low weight have a low importance, tests with a high weight are very important. If you compare two ROMs, one ROM can easily be better than the other 90 % of times and still come in second, because the remaining 10 % are more important!
[Touch HD Windows Mobile ROM Development] Oemie:
So in the end Dutty R12 will be experienced in daily use as really fast i guess?
[Touch HD Windows Mobile ROM Development] MAMeingast:
Dutty's ROM will definitely be fast - it's on third place! ;)

If it will be faster than Laurentius - I don't know! Depends on whether Daeron chose the "right" weighting! :D
[Touch HD Windows Mobile ROM Development] daeron.be:
err, Guys. I've just forgotten to sort overall performance column after adding a new rom. sorry for a misunderstanding... It's now fine. But you can sort all columns if by yourselves too, just click on a button between first and second cells in a column.

As I said already, these weights need some deeper thought. And I've asked (in an OP, if not, I ask here) to discuss it or to export this spreadsheet and change them by yourself.

About coreplayer results, they aren't counted into overall score yet, because there aren't enough rom tested. And about Miri performance -> we won't know unless somebody tests some Miri rom with coreplayer (using the method, I mean without polluting rom, after a reset, etc).

ATTENTION!

I'm experiencing a difficulties with a spreadsheet and the only reason for that I can came up with is that the size of that 'thing' has surpassed Google Spreadsheets abilities. The spreadsheet will be offline until further notice. Sorry for inconvenience.
[Touch HD Windows Mobile ROM Development] ckoeln:
Offline
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
To make spreadsheet smaller old roms no longer used could possibly be removed.....just a thought