yashsingh1/xdaforums huggingface dataset:
[Touch HD Windows Mobile ROM Development] bogdatov:
Benchmark is not everything. At least for me

For a long time I was using Dutty's 3.9 XT ROM. But about a month ago decides to give Laurentius26 v3.55 version a try. It is on 2nd place in ROM benchmarks and way above 3.9 XT.

I installed same software on L26's v3.55.

What I found strange is that v3.55 was not that fast. In fact it "felt" slower and luggy. After a month of trying, I moved back to Dutty 3.9. And I "feel" it is faster, faster response, faster folder openings, etc.

So for me, better benchmarked ROM did not give better experience :(
[Touch HD Windows Mobile ROM Development] MAMeingast:
Nobody said it would! ;)

Benchmarks aren't everything! You always have to try the ROM yourself to decide if it's good or not! BUT: Benchmarks can help choosing which ROMs to test! I've tried many, many ROMs - and my experience is: ROMs towards the bottom of the list tend to be slower than the ones towards the top, but - as to every rule - there are exceptions.

I use Topix 3.1.2 for daily use and it's (atleast in my experience) the smoothest ROM ever - and it's one of the fastest ROMs according to the benchmarks.
[Touch HD Windows Mobile ROM Development] ckong1:
LIA v16.1 performed really well in the benchmark considering it's a 2.5 rom. i'm currently on 14.5 21882 and found that i can keep it for daily use. can anyone advise if i should switch to 16.1? i'm hesitant because of the new style build.
[Touch HD Windows Mobile ROM Development] MAMeingast:
Not sure that Manila 2.5 makes much of a difference because TouchFlo is disabled during benchmarking anyway...
[Touch HD Windows Mobile ROM Development] first_damned:
Second that, im using Topix 3.1.2 and im really happy with it, fast, smooth, stable..everything you need really in a rom..Good old windows mobile 6.1 :)
[Touch HD Windows Mobile ROM Development] MAMeingast:
Update:

Added results for Topix 3.2.8 and Grid Evo 1.7 thanks to Giacominos!

Glad to see I'm not the only one left performing benchmarks! :D

But I'm not too sure about the results for Topix 3.2.8 - they seem a bit bad for me, considering 3.2 is 8 % better!? :confused:
[Touch HD Windows Mobile ROM Development] giacominos:
It's weird I followed all instructions, plus one soft reset between spb and opengl and I've installed stock rom before, I'm going to re-test it and we will see :)
[Touch HD Windows Mobile ROM Development] MAMeingast:
Not saying, that it necessarily is a fault on your side! ;)

Just suprised me!
[Touch HD Windows Mobile ROM Development] giacominos:
It surprised me too , so I'll send new benchmark :D
[Touch HD Windows Mobile ROM Development] accountabc:
It's not that weird actually. The 3.2.8 version contains the ati drivers from the TG01 instead of the neos2007 driverpack used in 3.2, so it makes sense that the performance has changed. Apparently the Neos drivers are better then the TG01 drivers. The Neos driver pack however is beleaved to cause the "lockup" issue discussed here, that's why they were removed
[Touch HD Windows Mobile ROM Development] MAMeingast:
Update:

Added results for Topix 3.2.9 and Topix 3.3, both thanks to Giacominos!
[Touch HD Windows Mobile ROM Development] jrmccann:
Hi,

Has anyone done the final L26 Ultimate Leo 4 Rom? Can't find it on the benchmarks...

THanks

J
[Touch HD Windows Mobile ROM Development] jrmccann:
Thanks mate!
[Touch HD Windows Mobile ROM Development] houms:
if you are intrested in HD2 resultts:

http://xdaforums.com/showpost.php?p=5267999&postcount=92
[Touch HD Windows Mobile ROM Development] MAMeingast:
Thank you for those results!

The HD2's speed is very impressive compared to the HD... The attached diagram compares the fastest HD2 ROM to the fastest HD ROM - and keep the following in mind:

The HD2 has just been released and cooking ROMs for it has just begun.

The HD2 ROM used is based on Windows Mobile 6.5, whereas the HD ROM is 6.1.

The HD2 ROM has Manila 2.5 integrated, whereas the HD ROM uses Manila 2.1. (I'm not sure, if this really makes a difference... but it could!)

Keeping this in mind, the HD2 is a true monster!

EDIT: Interestingly, the HD does outperform the HD2 in two categories (which is why the HD2 ROM does not achieve 100 %): The graphics test GAPI Bitblt and Arkaball 2d (I would never have expected the latter, not sure about the first). No idea what this means... But, as stated before, the ROMs for the HD2 will become a lot faster when newer cooked ROMs are released...
[Touch HD Windows Mobile ROM Development] band27:
Interesting results indeed. As mentioned, the good news keeps on coming with the evolution of cooked roms to feed the HD2, there is truely no telling were it will end up as these measurements are based on the HD2 using the more demanding manila 2.5 and still has no problem beating HD is most categories. Definitely a sign of evolution and things to come.
[Touch HD Windows Mobile ROM Development] yung_vu:
I thought benchmark methodology involved turning off all today screen apps and disabling manila so wouldn't have thought that manila version affected the results. happy to be corrected if this isn't the case
[Touch HD Windows Mobile ROM Development] MAMeingast:
I'm not sure if the Manila version makes a difference... It shouldn't in most tests, but it might concerning the file system tests because a lot more files are present in the ROM with 2.5 compared to 2.1. In any case, it's probably negligible...
[Touch HD Windows Mobile ROM Development] band27:
Yes, they all should but I was under the impression that manila was left running as most of us who use would like to know realistic day to day numbers. In this first post, Manila was not discussed as it was to be turned off or not.

If Manila is left on, I'm sure it would impact some of the departments of testing but not sure to what extent. As users 2.5 Manila certain lets us know of it's presence on the HD's day to day use. Thus, from this, I believe this would reflective in the results.
[Touch HD Windows Mobile ROM Development] MAMeingast:
I'm afraid you are mistaken: Manila is usually turned off for benchmarking! And this is also mentioned in the first post (point 4 refers to the disabling of all today screen plugins - Manila also falls into this category!)... :)
[Touch HD Windows Mobile ROM Development] band27:
Yes, yes, as I saw that point I question why Manila specifically wasn't mentioned.

Coolio, thx for the heads up....;)
[Touch HD Windows Mobile ROM Development] stoolzo:
attachment, where are you?

This is a very useful benchmark and thread, however trying to find the posted attachment can be a real pain especially when XDA is slow which it is sometimes, could I suggest a addition to your method of posting: Post it as a reply as you normally do but put a URL link to this attachment on you main post / first page, I think this is possible. Alternatively put and update on the first page "latest update on page XX" after you have posted.
[Touch HD Windows Mobile ROM Development] alareth1976:
dunno if this benchmarks are that usefull...

why to test roms with sense/manila disabled while benchmarking?

if I'll choose dutty/topix/lia is also for the graphic appearance.

infact, i've found that Topix 3.4 is a way faster than the Laurentius v2 (or also V3 or V4) with sense active...

If I was after a clean rom, surely i'll install a rom without manila/sense.

this is my opinion, btw :)
[Touch HD Windows Mobile ROM Development] MAMeingast:
That is a good idea - BUT: This thread was created by Daeron - and he no longer has the time to participate in the benchmarking of ROMs. As a matter of fact, I'm now the only one left managing this thread and I can't edit the first post, since it's not by me!

Also, I want to warn everybody: I'll be upgrading to a HD2 in a few weeks as well, so noone will be left to take care of the benchmarking table! :(
[Touch HD Windows Mobile ROM Development] band27:
For some of us, there will still be a slow migration over to the HD2 device. In the meantime, we've got you back Mameingast.
[Touch HD Windows Mobile ROM Development] stoolzo:
Appologies, i didnt realise that, you have done a stunning job though. I will be moving to the HD2 in the next couple of months too.
[Touch HD Windows Mobile ROM Development] MAMeingast:
No reason to apologize! :D
[Touch HD Windows Mobile ROM Development] MAMeingast:
Ok - now the time has come! I recieved my HD2 and sold my HD! I can therefore no longer perform benchmarks myself!

BUT: I'm willing to continue managing this thread... So if new results come in, I'll gladly add them to the table and post the updated version in this thread! Since there weren't many new benchmarks in the last few weeks the amount of work this means is quite minimal... ;)
[Touch HD Windows Mobile ROM Development] bobsbbq:
Glad to see you got you HD2. See you over in the HD2 Forum.
[Touch HD Windows Mobile ROM Development] mlemanczyk:
Hi,

Is it possible to edit first post in this topic with attachment to latest benchmark file? It'll simplify finding it. I haven't seen the results yet, so I voted "Sorry, but what are you talking about?" :)

Cheers,

Marek
[Touch HD Windows Mobile ROM Development] The Whistler:
And what I ask myself now is: What are the differences in the ROMs that lead to the different benchmark results?

What is special about Davideuck and MaryOne that they give so remarkable results in the test "Arkaball frames per second"?

Why is the Damn Small Topix so fast in "read 10kb x 100 files" and "copy 10kb x 100 files"?

I do not know much about ROM cooking, but could it be possible to create a fantastic performance ROM by combining the best of all ROMs?

The Whistler
[Touch HD Windows Mobile ROM Development] daeron.be:
Sorry guys, but I've made a serious error and misinterpreted one of the test's results: lower Arkaball's results are better! It has changed things a bit I guess, but not much. Not for top3, at least...

Btw I'm waiting for another Topix release because I really cannot flash this rom properly. RLY...

Thx The Whistler for pointing this out. :)
[Touch HD Windows Mobile ROM Development] mitut:
For me the faster the rom is the easier my phone will get hang :))) .

Just use rom for a week and put in some application u will know which one is the best.

For me now i really satisfy with Onkolog 5.5 rom superlight which i use for more than 2 week without any hang ^^
[Touch HD Windows Mobile ROM Development] The Whistler:
Well, I honestly have no idea what "Arkaball's results" can tell me...:o

The only thing I observed is that there are significant diffences between some of the benchmark results (and many more statistically insignificant differences...:D), and I just wondered if the cooks in some kind of shared effort could create a "the best of" ROM...

But this is slightly off topic...:o

The Whistler
[Touch HD Windows Mobile ROM Development] daeron.be:
Arkaball benchmark tests 2d rendering speed, for example in 2d games.

And yes, creating as good roms as possible is a kind of shared effort anyway, because if some rom scored really good others can use parts from this rom to improve theirs.
[Touch HD Windows Mobile ROM Development] daeron.be:
I've added Topix 2.0 RC1 benchmarks. Enjoy.
[Touch HD Windows Mobile ROM Development] fatdogcat:
thanks, the stats are useful and handy for me
[Touch HD Windows Mobile ROM Development] alper1:
Thanks! As I said, Topix rom is amazingly fast!!
[Touch HD Windows Mobile ROM Development] NNagendran:
daeron,

Great work. Very useful. But would it be possible to add separate WM 6.5 benchmarks? That would really help...

Thanks mate.
[Touch HD Windows Mobile ROM Development] cerato:
I don't think this classification is fair. Really. Some chefs are using newer builds, cores, OS, manilas etc thus gaining a step ahead of "competitors"... And since sharing of resources is long ago with some minor eceptions not a common practice at XDA (sorry chefs, but no matter how much i appreciate yr work, i don't at all like non sharing), benchmarks should be made at EQUAL ROMS . Cuz you can not tell "Hey look, the chef X is better than chef Y". The truth is that the chef X gains more octanes not by himself but by MS or HTC ;) So, bro, pls compare comparable ROMS. Otherwise no use of yr benchmarks at all. Sorry if i sound a bit harsh... no offense. I just wanna be fair enough to everyone.

And DEAR FORUMERS,... PLS SHARE.;)
[Touch HD Windows Mobile ROM Development] KukurikU:
:confused: yr post make no sense at all. it seems that you forgot that the benchmark is for the ROMs and not for the chefs:D

so, what are u talking about? what exactly do u want? it seems to me that u are the one that want to compare the chefs....once u gave them "EQUAL ROMS" there is nothing else to compare! nobody said:

we say the ROM X is better than the ROM Y! nobody knows how much from a good (or bad) ROM is the merit of the build or the merit of the chef. sometimes newer builds are worse than new ones. i was under the impression that we are interested in the fastest ROM around and me personal don't care if it is kwbr's or laurentius's or if it is a new or old build!:p

if there is any competition here, between the chefs (dohh...), i like it. like any other competition is good for the spectators (in sport) or for the customers (in the free market).

like in racing not only the driver decide who the winner is. the motor, the car also counts!!

no, u didn't sound harsh you sound disorientated.

thank you for taking care of what is good for us but on this one i seriously disagree.:p

P.S. i hate big letters!
[Touch HD Windows Mobile ROM Development] PapaDocta:
well said KukurikU... can't add anything to that as i agree with everything you said:D
[Touch HD Windows Mobile ROM Development] JLowe:
I can only say: Onkolog rulez! Lite, fast (ATI-Drivers), battery life, stable, no bugs.
[Touch HD Windows Mobile ROM Development] Matterhorn:
I would be interested to see Onkolog's "latest" Lite rom included, as JLowe said, he has changed the video drivers in the new version which may make a big difference in benchmark results. Another spec in the spreadsheet that may be helpful to some people is the amount of free RAM after a soft reset. With the "older" Onkolog lite, I get 142 meg of free RAM (amazing), with the Damn Small Topix May 20, I get 112 meg RAM with my same apps installed. For the power user, free RAM can be an important factor in deciding which ROM to use.

The Topix is performing very well though and I'm not changing for the moment so I can't test the new Onkolog Lite.

P.S. PLEASE DON'T SAY THINGS LIKE "THIS ROM IS THE BEST" etc. The mods will shut this thread down if you say stuff like that!
[Touch HD Windows Mobile ROM Development] henrik.dk:
Mind adding this ROM to benchmark test? Thanks.

http://xdaforums.com/showthread.php?t=472972
[Touch HD Windows Mobile ROM Development] KukurikU:
and if we'll compare how many programs i'll have to add in each one in order to get a "complete" ROM?? how much is "out" in kwbr's ROM and how much in onkolog's? how many apps i'll have to add after a fresh flash and what the results will be in the end?...depends on the user.

imho, we have a benchmark and we should treated like a benchmark. this one is for speed. period!

i'll never say that the fastest is the best. it seems to me some ppl here are taking this benchmark much too seriously and much to personal. speed is only one parameter and we can measure it. there are things we can not measure precisely like stability, current consumption, candy-eye, etc...lets put them aside. want a good ROM? start flashing until you'll get the one that suits you.

this should be a thread for everybody to come, look, suggest maybe a new ROM to add and not for posting personal comparisons or opinions.

everybody needs a start point in order to choose HIS ROM. you can start with the fastest, you can start with the one that gave you more memory, you can start with the one that has the most beautiful home screen (in your opinion:p).

for the record, i flashed my hd at least 20 times :p
[Touch HD Windows Mobile ROM Development] stoolzo:
good work

Good work my friend, there isnt much out there to compare roms properly and this post does the job nicely.

keep up the good work.
[Touch HD Windows Mobile ROM Development] stoolzo:
stock rom

might be best a good idea to have the latest stock WW on there for a comparison, this will highlight the good work the cookers have done or not :)
[Touch HD Windows Mobile ROM Development] spooky69:
+1 for stock rom comparison.
[Touch HD Windows Mobile ROM Development] Matterhorn:
Fine, it was just a suggestion. Free Ram is quite measurable and important to people that have many huge apps open at the same time, so I thought why not...obviously not:confused:
[Touch HD Windows Mobile ROM Development] MAMeingast:
The results are two pages back... Here.

The results can't be added to the first post because Daeron is no longer managing this thread.

Also, there haven't been any new results in weeks! :(
[Touch HD Windows Mobile ROM Development] acecooper:
hmm

yea... he got a TouchHD2 so tweaking that i guess. I'm waiting for Windows Phone 7 till I get my upgrade.. also when my contract is up for renewal :)
[Touch HD Windows Mobile ROM Development] mlemanczyk:
Thank you :)
[Touch HD Windows Mobile ROM Development] KukurikU:
u might be right on this one. once upon a time :rolleyes: when the great majority was very proud of owning a Diamond with its limited memory and no SD card the free memory was posted together with the ROM's description.

in our days when the HD's memory is substantial bigger:eek: and when u can (as i did) add a 16 giga SD card the memory is less problematic.....
[Touch HD Windows Mobile ROM Development] sonydesouza:
I also think this benchmark is unfair...

I tested a lot of roms and honestly, the best roms for me are Miris roms... fast, bugs free and stable...And, when i see your test, miris rom would be the worse...What a joke !

Moreover, you compared WM 6.5 and WM 6.1...it's not equal...

Anyway...
[Touch HD Windows Mobile ROM Development] s1rl4ncel0t:
IMHO, the biggest joke here is your post, m8 :eek:

If you prefer miri roms, keep on using them. I'm not using the fastest rom in the chart because I value other things such as default programs preinstalled in the rom. I consider these tests as a starting point, not more, not less. It helps me to decide, but it's not the most important decision factor.

Daeron only posted results which came out of the benchmark software. If you dont agree, then contact bench software manufacturer and tell them that their bench software is crap.

Just don't say that daeron's hard work is a joke. He's not responsible for the results. Just for the execution of the tests and posting it's results.

Peace!:)
[Touch HD Windows Mobile ROM Development] crypta:
I would't call them a joke, but often have they benchmarked?

there are significant differences in 3D... ok, there are diffirent drivers available as far as I know... but the normal I/O operations are more hardware related, I think.

using maryones rom for more than 2 weeks and it seems to be pretty stable. and where for my use... some of the fastest.
[Touch HD Windows Mobile ROM Development] bobsbbq:
I will only say that these types of test are never a real answer to overall performance and usability for the user. I myself had the same idea and prepared some pretty impressive spreadsheets with many more test than yours with graphs etc. In the end I decided to cancel the idea do to the fact that in most hardware test the results are so close and the difference to the user is minimal to none. The fact is the only way to give someone a good feel about a ROM is to give your opinion and as we all know that is subjective and against forum rules. The test that are being run are mostly read and write data test. This does not tell very much about usability. Test like battery drain graphical test and overall processor test would definitely help. But then you have other issues which you can't see or answer in your test like, smooth scrolling of the screen, screen responsiveness, how fast screens or programs load etc. Before cooking ROMs I tested so many I can't even tell you how many I have tested and one thing I have seen is on some roms the data test were lightning fast and other things would put the rom to a negative or less attractive choice like battery drain, mem leak, scrolling smoothness etc. So I think you can see where the road leads.

In the end the one big major flaw I see in your work is you must separate WM platforms. There is know way you can compare a 6.1 with a still beta 6.5 in the same sheet.

In the end I wish you the best and thank you for including my ROM and I have a new version out 4.0

If you are interested in the spreadsheets I have let me know, I share.
[Touch HD Windows Mobile ROM Development] AndV:
So, why you don't test Onkolog's newest ROM? I downloaded both Onkolog's (Super Lite [21046 aku 1.6.3 ONLINE][May 19]) and Kwbr's (Damn Small Topix [Base 1.59.709.0, OS 5.2.21046][May.20]) and tested on my device. Here's results As you can see Onkolog's is faster at all! Please update your table with new results.

Here is video confirmation:

http://rapidshare.com/files/237032670/Onkolog_OpenGL.mov

http://rapidshare.com/files/237036267/Onkolog_SPB.mov

http://rapidshare.com/files/237038240/Kwbr_OpenGL.mov

http://rapidshare.com/files/237040527/Kwbr_SPB.mov

P.S. There is one strange thing with Graphics test DDB Bitblt on Onkolog's ROM, twice I have result with 0.27, once with 3,64, so you can insert what is more truthful on your opinion.
[Touch HD Windows Mobile ROM Development] bobsbbq:
You are right, and the truth is almost always the barebone and lite ROM's will naturally be faster as they are not loaded with programs.
[Touch HD Windows Mobile ROM Development] Hardqs:
Personal opinion, and im sure it does not count for much here.

I have had the HD since it came out and have been flashing for ages with different roms.

One thing i could never understand is why so many chefs where leaving all the junk in the roms.

Fair comment by one user said chicken and water doesnt make a good soup, but either does a thick and gunky soup with to many flavours.

Im a windows user on my pc, i dont use the version that comes on disc, i rip it with nlite and take all the rubbish out, rip out all gateways and rubbish that allow backdoor entry monitoring, and it runs super fast, specific drivers are installed rather then having a catalog of a 1000 etc.

To me anything commercial is nomally bloated and under performs. Again look at a PC/ Laptop that you by from a shop, all that power technology and they manage to make it run so slow with mass installed rubbish bloated software.

So personally think KWBR, MIRI and ONKOLOG are on the right track, not just on performance but how OS's should be. clean and fast bug free.

Saying that there should always be a choice to the people. :D
[Touch HD Windows Mobile ROM Development] KukurikU:
one more reason to be thankful to this thread. imho, i think i know why onkolog changed his method of "less is better" and started also adding new drivers and cooking "gurme'" ...;). the answer is: good, old competition.

like i already said in one of my previous posts, (sorry for quoting myself:p):

i hope we'll all be thankful for this thread. and let's start doing this by stopping the *****ing and starting to acknowledge that this benchmark is useful (even in it's limits) and that initiator of this thread deserves our gratitude:D
[Touch HD Windows Mobile ROM Development] manowar1978@gmail.com:
And for battery performance?!
[Touch HD Windows Mobile ROM Development] sonydesouza:
Your post is a joke...

PEACE ! :cool:
[Touch HD Windows Mobile ROM Development] Matterhorn:
I think that battery performance is a difficult test to make for so many ROMs, mainly because it requires to much time and energy to test.
[Touch HD Windows Mobile ROM Development] bobsbbq:
Yes and no, there are some test which measure battery drain in mA which can give you a idea of how much draw is on the battery. The main thing is all devices must be tested in the same way, with the same backlight setting etc. But some batt eating programs or if there is a mem leak you wont no until the program is running.