yashsingh1/xdaforums huggingface dataset:
[Touch HD Windows Mobile ROM Development] ceevee369:
I bow my sword upon you oh lancelot.

Euh, but were the Diamond Roms neither tested "as are" with the radio's included? . I remember that the nice guy made this huge spreadsheet comparing OS build - ROM and used radio by the Chef no?

I can't remember that he tested the ROM's with different Radios, but I might be wrong :o

If we want to make it fair - why indeed not testing in Airplane mode?

You are right that Radio interfers with speed- CPU power etc, so if one is testing -benchmarking his Rom under perfect condition - no network instability, it might be - and will be different vs. one which Radio is still searching for best signal (if not in locked Operator mode)

Something to consider indeed my lord.:D
[Touch HD Windows Mobile ROM Development] daeron.be:
Every benchmark was made in Airplane mode.
[Touch HD Windows Mobile ROM Development] chrisque1:
;)

Thxs for the benchmarking work friend. Just try to ignore the radical positions and keep the good work.

:)
[Touch HD Windows Mobile ROM Development] parnunu:
if u compare 6.1 with 6.1

and 6.5 and 6.5

(all developer)

be good
[Touch HD Windows Mobile ROM Development] daeron.be:
I've added

Damn Small Topix 2009.05.20

Topix 2.1 HD 2009.05.24

benchmarks, done by Houms. They're fully compatible with my benchmarks, they're done in the exact same way.

Thank for your hard work!

Enjoy!
[Touch HD Windows Mobile ROM Development] ceevee369:
Sorry my fellow country man. oversaw that one :o

Not a request, but a recco is to include now Dutty's 6.5 release.

Majority of users will stick either to Miri's releases or upcoming / future Dutty's - which is guaranteed seen the history of TytnII and Diamond Rom users.

Especially comparing Miri's and Dutty's would be the suggestion :)

(and thanks for this great contribution vriend/ami)
[Touch HD Windows Mobile ROM Development] dusty_nz:
JohnPatcher

Please add the johnPatcher rom.

Blew me away how fast everything runs. Key id manilla removed.

Good indication for the overhead of manilla.

http://xdaforums.com/showthread.php?t=472972&highlight=johnpatcher

http://xdaforums.com/showthread.php?p=3774227#post3774227
[Touch HD Windows Mobile ROM Development] sna:
Dutty's WM6.5 rom

Any benchmarks for Dutty's HD WM6.5 R1 rom yet?
[Touch HD Windows Mobile ROM Development] wind79:
Is there going to be one for Onkolog's Superlite?
[Touch HD Windows Mobile ROM Development] The Whistler:
If everyone posts his wishes here is mine: ;)

Panoshas Black Pearl 2.9 Full and

Panoshas Black Pearl V2 Clean...

Thanks for your work,

The Whistler
[Touch HD Windows Mobile ROM Development] dany20mh:
No New Update ????

Dutty's WM6.5 R1

Miri V14

...
[Touch HD Windows Mobile ROM Development] daeron.be:
I've added newest Dutty's 6.5 release candidate 1 rom.

Unluckly, it sucks in terms of performance, especially 3d one.

Miri's 14 comming really soon. :)
[Touch HD Windows Mobile ROM Development] SWYRx:
Would like to see Topix 1.6.1

Appreciate your work very much!
[Touch HD Windows Mobile ROM Development] ceevee369:
Thanks for including Miri and Dutty's my dear compatriote.

Early conclusion looking at Spreadsheet :

Average of all WM6.5 benchmarks combined are definitly slower than 6.1 releases....:mad:

rather surprising. Increasing 3D scores (new drivers) won't solve the GAP.

Start wondering if 6.5 is as "good" as we think :confused:
[Touch HD Windows Mobile ROM Development] dany20mh:
Hi , Thanks For Add Dutty's 6.5 R1 , And Now I Have Some Idea For You ,

1. Add Windows Version In Front Of Every ROM You Test , With This Option , We Can Select Win 6.1 Or Win 6.5 An See Result For Them And Compare .

2. Add Manila Include Option ( Yes Or No ) , Some User Want To Compare ROM With Or Without Manila .

Thanks Again , And Sorry For My English .
[Touch HD Windows Mobile ROM Development] sephiroth_vg:
dude miri is upto ver 14 nw :S version 12 sucked a lot
[Touch HD Windows Mobile ROM Development] daeron.be:
Frankly speaking, I wonder what what they're tinkering in 6.5 so it almost doesn't have either good, new features nor any performance boost. It's pretty weird.

btw: I've added Miri 6.5 v14.0. And yeah, it performs lot better than v12.0

Take a look :)

Edit: I've added latest WWE shipped rom and latest DamnSmall Topix...
[Touch HD Windows Mobile ROM Development] freskile:
Thanks @daeron, grteat initiative adding these benchmark results.

Could you add LIA Prem v5.0 pls.
[Touch HD Windows Mobile ROM Development] pzucchel:
Hi,

i would be very curious to see the results from an "archaic" ROM like this one or from stock roms. In other words, quantifying the actual improvements in the last months....;)

http://xdaforums.com/showthread.php?t=462974
[Touch HD Windows Mobile ROM Development] bobsbbq:
Daeron, I'm not trying to take a shot at you but I do have some comments.

#1 Your details about the ROM's are not complete, which is vague and not fair to everyone. Ex. You said you included Miri's V14. What you did not state is which one? He has 3 versions.

#2 Why do you still insist on having WM 6.5 in the same category as 6.1? It is obvious that 6.1 does better in results and has been around for a long time where 6.5 is still a beta? IMHO if you are going to have a challenge or are going to give congratulation on the #1 ROM then you should separate them by 6.1/6.5, then Lite/Standard/Premium.
[Touch HD Windows Mobile ROM Development] dawnbringer:
great work thanks!!!!!

man it must have takin alot of your time!!!!!

once again GREAT WORK!!!!!!!!!!:):):):)
[Touch HD Windows Mobile ROM Development] skyli91:
How less FPS is better?
[Touch HD Windows Mobile ROM Development] nanoo-eg:
Subject is very simple.

If someone don't belive the benchmarks here, or think that they don't look the reallity of any rom, can make a test.

Ask a friend whitch has HD and open any tab of manila, or any application. One with first at benchmark rom and the other HD any rom you want.

You will see benchmark say the trouth.

Please can you make benchmark for panosha black pearl roms? I have do my self

and I have take very good marks.

Thank you.
[Touch HD Windows Mobile ROM Development] Eraser85:
yeah, panosha black pearl is a great missing in your benchs :)

Also, could you benchmark dutty's 3.6, too? performance should be increased a bit from 3.4

thaaaaaaaaaanks :)
[Touch HD Windows Mobile ROM Development] kwbr:
Topix is far from a barebone or lite rom. I has many apps included which are not part of a default rom.
[Touch HD Windows Mobile ROM Development] bobsbbq:
Hey my friend , was not referring to yours but in general. I know from my previous test I used to do your ROM's are always fast.
[Touch HD Windows Mobile ROM Development] oliver79:
first of all: RESPECT - great work - thank you for that!!

and.. would you add this ROM

http://xdaforums.com/showthread.php?t=494210&highlight=moritz

would be great :)
[Touch HD Windows Mobile ROM Development] kwbr:
Hi Bob, easy going in Thailand? :)

I will start with a 6.5 project next week. Let's see if we can let this engine smoke :rolleyes:
[Touch HD Windows Mobile ROM Development] bobsbbq:
That would be a good thing for sure. I'm still just using the ati d3d drivers.
[Touch HD Windows Mobile ROM Development] onkolog:
This is my new WM 6.1 ROM results...just for fun)))

I did it like in instruction in the first post...
[Touch HD Windows Mobile ROM Development] akash_nu:
Hi guys,

Can the Topaz vanilla final ROM be added to the list please. I am using it at the moment & must say that it has been performing really good so far. I have not come across any bug whatsoever.

In my opinion, the speed is good as well. Would really like to know how this ROM would compare with the other legendary ROMs like Dutty 3.6 and Damn Small Topix 0.3...
[Touch HD Windows Mobile ROM Development] skyli91:
I second want to know how fast is Dutty's 3.6 rom.
[Touch HD Windows Mobile ROM Development] skyli91:
Wow, it's wery fast! Is it the new winner?
[Touch HD Windows Mobile ROM Development] pzucchel:
HI I found this exercise very interesting. Unfortunately, there is not much information about the ranking strategy and i had maliciously suspected that the algorithm used for the classification was misleading, and/or not robust. With these technical words, i mean that changing slightly the algorithm for ranking, we could have come to completely different results and conclusions. For example, i had the doubt the most of the columns are not representative of daily operations and consequently misleading the ranking algorithm.

In fact, to a first look (in attachment), it is NOT the case. I tried to apply different weights to the ranking, and played with them for a while.

In the essence, apart from few pathological exceptions which are probably due to "bugs", it looks that the fastest roms are the fastest consistently for all the tests, so weighting and/or assuming that some operations are more important than the others is not affecting the conclusions significantly.

As already pointed out, the functionality of the rom is also important, beyond pure performances.

Thanks to all of you guys that put in place such an approach, i think we should try to put more work on it and - compatibly with time - i will be happy in contributing. :)

ADDENDUM: Following this thread, i had changed from DavideUck 6.1 to TOPIX 2.1 and it is confirmed that, in my hands and routine operations, the ROM feels actually faster...
[Touch HD Windows Mobile ROM Development] pzucchel:
proposal

addition: Daeron, are you considering making a wiki page out of this review? it is a pity this thread is not sticky....
[Touch HD Windows Mobile ROM Development] chris5304:
Any chance of an entry for John Patcher's Barebones ROM? Clearly very different in content and functionality from these with Manila etc. and I would be interested to see hom much (if any) performance is gained as a result. Equally Panosha's Clean V2.

Thanks for this thread, the idea is simple but effective!
[Touch HD Windows Mobile ROM Development] dany20mh:
Please Add New ROM :

Miri V15

Dutty V3.6 ( I Think V3.7 Release Is Soon )
[Touch HD Windows Mobile ROM Development] 7Bit:
Nice work, thanks for this information! :)

But I think it would be better to separate WM6.1 from WM6.5 benchmarks, wouldn't it?
[Touch HD Windows Mobile ROM Development] bobsbbq:
Varied Results with TestOpen GL

Hello, I seem to have some differences in my test and yours with the same ROM' etc. Can you help me to figure out what is the difference. Is it my proceedure or is it my device? I tried to follow your instructions in the first post but could never obtain the results you have even for my own ROM and others.

I flashed, then Hard reset, installed TestOpenGL only as this is the one I'm try to work with at the moment.

Then I unchecked turn backlight off in battery mode and unchecked turn off device in battery mode.

Soft Reset

Then run the test.

I have the test #1 is usually between 16-17000, with a occasional right at start of 19000.

#2 seems to be ok at about 5200

#3 is always not more than 18000.

#4 is about 8900-9200

Now here is what I have discovered. If I don't uncheck the box for backlight to dim the results change drastically. Here are my results with backlight auto dimed.

Results

--------------------------------

alpha texture (128x256 rgba)

28299 kPixels/s

torus (3k triangles) one light

6395 kPixels/s

color blending

29666 kPixels/s

sub texture (128x128 15x15)

9439 kPixels/s

OpengGL ES info

--------------------------------

Vendor

QUALCOMM, Inc.

Version

OpenGL ES-CM 1.0

Renderer

Q3Dimension MSM7500W 01.02.03 0 5.1.2

Extensions

GL_OES_byte_coordinates

GL_OES_read_format

GL_OES_fixed_point

GL_OES_single_precision_commands

GL_OES_compressed_paletted_texture

GL_OES_matrix_palette

GL_OES_point_sprite

GL_OES_point_size_array

GL_ARB_vertex_buffer_object

GL_ARB_texture_env_combine

GL_ARB_texture_env_dot3

GL_ARB_texture_env_crossbar

GL_ARB_texture_mirrored_repeat

GL_EXT_stencil_wrap

GL_EXT_blend_minmax

GL_EXT_blend_subtract

GL_EXT_blend_func_separate

GL_EXT_blend_equation_separate

GL_ATI_texture_compression_atitc

GL_ATI_extended_texture_coordinate_data_formats

GL_OES_draw_texture

Now what is funny is even if I set the backlight to the minimum which is basically off, the results are the same lower value. It seems that something is changed when the Autodim feature is enabled that increases the performance.

So now my question I guess is do you uncheck the dim backlight check box?
[Touch HD Windows Mobile ROM Development] pzucchel:
Hi, do you observe differences only in the OPENGL tests or also in the SPB benchmarks?
[Touch HD Windows Mobile ROM Development] bobsbbq:
I have only tried with openGL for now.
[Touch HD Windows Mobile ROM Development] bobsbbq:
It seems the differences in Spb Benchmark are minimal and could be just normal fluctuation. I ran 3 test with BL on and 3 Test with it off. There were variences even within the same 3. So I think it has something to do with the 3d graphics drivers or program. My question to other users of 6.5 and 6.1 can you duplicate this or is there any difference in results with the auto dim of the backlight? Some members are also looking in to this in the graphics performance thread. It has also been mention before with X1 so it might be a driver issue.
[Touch HD Windows Mobile ROM Development] zuadao:
can you pls update the benchmarks to the new dutty´s and kwbr roms and include the black martini ones??

and MODS pls give a sticky to this one :)
[Touch HD Windows Mobile ROM Development] pzucchel:
HI to All, i really suffer in seeing this thread going down and not becoming a permanent reference information for all of us spending time reading this forum. What can we do to keep it up? Daeron, do you expect other people to update the list or you feel you could do it? I feel that, at the beginning, it would be beneficial if you could take the lead.

I don't want to criticize anybody, but i feel it is useless to see a new version everyday, without the vaguest idea about the improvements. As you saw in the test (and i can confirm it in real life) very often Rom eprformances degrade with the next version, and the new benefits often don't compensate for the losses.

Somehow, it is absolutely false that a new ROM is a better ROM, it could be better for somebody and worse for others.

For this reason, having an objective test would allow anybody to take their own call and decide what to try.

(please anybody that believes in this thread post once in a while so to keep it up! :rolleyes:)
[Touch HD Windows Mobile ROM Development] KukurikU:
i'll make it my mission to shamelessly bump this thread at least once a day (if necessary: ;)
[Touch HD Windows Mobile ROM Development] bobsbbq:
I would love nothing more than to to see this happen just look at my old thread here

http://xdaforums.com/showthread.php?t=492868&highlight=ROM+benchmarks

I spent so much time trying to do the same thing but finally gave up.

My biggest complaint about this current thread and I thing I have made it know is in the way it is organized, and it's lack of information about the ROM's tested. And I also question the testing methods as I can't reproduce the same results. For a thread like this to be useful the method has to be spelled out in detail. There must be detailed information about the ROM's. Any users and cooks have to have confidence that the methods and results are real and can be reproduced.
[Touch HD Windows Mobile ROM Development] andes83:
Wow what a great thread it's a shame it's not a sticky and that it's so rarely updated i just saw it browsing through the forum and i think it's essential for users and ROM chefs to know and to compare benchmarks on the ROM's .I would suggest that 6.5 and 6.1 ROM's be divided in two different categories as they are not really on the same level of performance also someone suggested a battery test which is not a bad idea.You can do a test like this for an hour of hard battery useage with wi-fi ,3g (of course you look for the same amount of signal at one place) and bluetooth and 1 hour of no useage at all sleep mode or something like that and post the percentages it's not so hard to put the device in the same amount of stress for an hour.Also chefs have light, premium and medium versions which also have to be taken in consideration and i don't think it's that hard to add these categories.And last but not least (i think it can't be really done but why not try) a bug category of the first bugs that "hit your face" sort of speak. But all in all it's a great thread and the results are very important for users like me that don't like to flash all day long just to find which rom suits them the most.
[Touch HD Windows Mobile ROM Development] daeron.be:
I've added maaany benchmarks make by houms.

You've gotta see this.

And I've updated the google docs link, keep it in mind.
[Touch HD Windows Mobile ROM Development] alper1:
Thank you DAERON! :)
[Touch HD Windows Mobile ROM Development] daeron.be:
BTW.

I'm going to test newest Topix and Laurentius26 rom and update the first post - add FAQ.

Soon.
[Touch HD Windows Mobile ROM Development] corwin_de_amber:
There seem to be some mistake in the result ot "Compress 1 MB file using ZIP (LESS IS BETTER)" test for new onkolog's ROM (Onkolog's 2009.05.19). It is 10 times slower then any other ROMs here. In all the other tests this ROM is one of the bests.
[Touch HD Windows Mobile ROM Development] johnpatcher:
Is there any way to test a ROM for myself? I don't want to load someone with this work, but I want to know how good/bad my ROM actually is :).
[Touch HD Windows Mobile ROM Development] fwt:
Take a look at Post 1

Just below the Word "Method"
[Touch HD Windows Mobile ROM Development] Fallen Spartan:
I noticed an error on your benchmark spreadsheet. I didn't realise Duttys 3.7XT was created and cooked by kwbr???

I'm sure Throy will be glad to see his hard work was done by someone else!!! LOL :p :rolleyes:

You may want to give the rest of your spreadsheet a once over just incase they're more discrepancies. Just a thought

Thanx
[Touch HD Windows Mobile ROM Development] johnpatcher:
Is this up to date? I though that copying files and so on is a new kind of tests, but after looking at it again, it might be a job that spb benchmark will do :).

Thanks anyway.

€dit: Which results of TestOpenGL do you take? It doesn't show any result at all, does it? And taking just some results isn't very objective? Or am I missing something?
[Touch HD Windows Mobile ROM Development] illuzi0n:
hmm shame wanted to see Miri 6.5 v17 benchmark...:(
[Touch HD Windows Mobile ROM Development] akash_nu:
Can the latest TOPAZ VANILLA ROM be included in the list please?!
[Touch HD Windows Mobile ROM Development] daeron.be:
Tests are made by SPB benchmark and TestOpenGL programs. If you don't get any numbers in TestOpenGL than it's time to update your version. :)
[Touch HD Windows Mobile ROM Development] squirreleater:
Hi,

Could I be so bold as to suggest not testing updated versions of the same Rom each time? Maybe put a few of the less popular roms on there?

Johnpatchers Rom, maybe Jasjammings new Martini rom? etc.

Otherwise it is a nice side show to flashing roms and it's interesting to compare your results to how the roms actually feel to use- as a rule your speed tests do seem to match up to how roms feel to use.

Cheers,
[Touch HD Windows Mobile ROM Development] daeron.be:
Thx for pointing them out, both errors are fixed now.
[Touch HD Windows Mobile ROM Development] andes83:
Thanks for new benchmarks some great results there and happy to see that Topix 2.2.1 is performing this high using it right now and i'm more than satisfied with it.
[Touch HD Windows Mobile ROM Development] johnpatcher:
I don't know whether you take results of benchmarks not done by yourself, however I'll attach my results. They aren't that good, but I haven't done any modifications so far, will try to get more in the upcoming releases :).
[Touch HD Windows Mobile ROM Development] dusty_nz:
John Patcher Results

This is what I got. Seem pretty good.

<benchmark-results>

<info>

<platform-name>

HTC Touch HD T8282

</platform-name>

<device-name>

HTC_HD_JohnPatcher

</device-name>

<start-date>

06.10.2009

</start-date>

</info>

<main-tests>

<fs-largefilewrite>

550.500000

</fs-largefilewrite>

<fs-largefileread>

34.500000

</fs-largefileread>

<fs-largefilecopy>

627.100000

</fs-largefilecopy>

<fs-manyfileswrite>

4267.700000

</fs-manyfileswrite>

<fs-manyfilesread>

182.400000

</fs-manyfilesread>

<fs-manyfilescopy>

2816.800000

</fs-manyfilescopy>

<fs-dirlisting>

1222.300000

</fs-dirlisting>

<fs-db>

788.200000

</fs-db>

<grf-ddb>

4.025000

</grf-ddb>

<grf-dib>

30.555000

</grf-dib>

<grf-gapi>

5.920000

</grf-gapi>

<misc-zip>

2592.000000

</misc-zip>

<misc-jpeg>

410.300000

</misc-jpeg>

<misc-arkaball>

8.602000

</misc-arkaball>

<misc-mflops>

3569.200000

</misc-mflops>

<misc-mops>

1001.100000

</misc-mops>

<misc-mwips>

7175.000000

</misc-mwips>

<misc-memcpy>

8.441000

</misc-memcpy>

</main-tests>

</benchmark-results>
[Touch HD Windows Mobile ROM Development] dusty_nz:
John Patcher - OpenGL

alpha texture (128x256 rgba) (MORE IS BETTER)

37581

torus (3k triangles) one light (MORE IS BETTER)

6394

color blending (MORE IS BETTER)

38439

sub texture (128x128 15x15) (MORE IS BETTER)

10752
[Touch HD Windows Mobile ROM Development] pzucchel:
thanks daeron!
[Touch HD Windows Mobile ROM Development] onkolog:
daeron.be

You should test my last ROMs from June 3...because you have only my previous version...It'll be right competition...
[Touch HD Windows Mobile ROM Development] dusty_nz:
adding info

Would be quite good to add basic rom version likes its base OS release.
[Touch HD Windows Mobile ROM Development] The Whistler:
Hi,

just for my understanding: It seems that after the last major revision of your spreadsheet all ROMs have dropped in their "Overall performance" by about 20% - is there a reason for this?

Thanks, The Whistler
[Touch HD Windows Mobile ROM Development] MAMeingast:
Daeron and Houms: Thank you very much for all the work you guys have put into this! Really appreciate it! :D
[Touch HD Windows Mobile ROM Development] KukurikU:
competition??? where? i thought this is a benchmark ....:p anyway if this would be a competition you should stop taking out and live something....with all the light/super light/extra light or infinite light very soon yr ROMs will be 500K and the user will have to add the phone app, notes, ppl and tasks by his own....lol!

onkolog, my dear friend, i am glad you changed yr opinion, let me remind you:

you were talking about dutty's ROM then and but i think you found kwbr's ROMs worth studying.;)
[Touch HD Windows Mobile ROM Development] kemsou:
Error in formula

Hi,

Daeron, houms, Thanks a very lot for your work.

but there is an error in spreadsheet Overall performance formula : N,O,P,Q R are not counted

Regards,

Kemsou
[Touch HD Windows Mobile ROM Development] onkolog:
I'm not your friend...I don't have silly friends with 22 fixes for all new ROMs:D:D:D you should fix your brain before you'll write here next time:D:D:D 22 times:D:D:D:D

And you know it's hide competition between cockers anyway...
[Touch HD Windows Mobile ROM Development] johnpatcher:
And you should fix your language, this isn't apposited anymore, but you have proven to be that dismissive a few times already, so maybe you are just very childish?

Btw: What has he done wrong at all? Providing fixes for common bugs isn't something you have to be ashamed of, is it? In fact he has contributed more to the community than you have, so you should take a step backwards instead of being that offensive!
[Touch HD Windows Mobile ROM Development] onkolog:
johnpatcher

Hello great thief :D I know my English isn't good enough,but I am working :D

How are you)))?Do you still stole files from other cookers or your brain start working? :D

His fixes were for previous ROMs not for new where it was fixed by HTC in new OEM or already fixed by cookers in theirs ROMs...You should think sometimes it's good habits...