yashsingh1/xdaforums huggingface dataset:
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
220xx branch results posted

Where do I even begin? I honestly am at a loss for words. For those of you who know me, that's pretty unusual. This was easily the worst branch I have tested. Scores in the master list may be a bit deceiving, but I was stunned while testing them.

I don't even think I have a favorite build. They were universally bad. It was the darnedest thing, too. They got progressively worse as I got higher up the tree. This branch is absolute garbage. If you're a chef and reading this, please don't cook with this branch!

It started out simple enough with 22013. This one wasn't awful, but seemed like it was similar to other branches in that it started off slowly. The video quality was decent, and menus scrolled pretty well. Opening applications lagged a bit, but it was nothing major. The next one was 22016. You won't find this one on the list, because I couldn't get it to move past the splash screen. I cooked it three times with builds from two different sources and got the same results. I doubt this is the fault of the build, but this is the reason it was omitted from the results list for anyone saving or comparing the spreadsheets.

If we skip ahead to the middle, I got kind of upset in doing the testing and took a break. This testing is EXTREMELY time-consuming. It takes about 30 minutes to test out a single build if I stay diligent on it from task29 to the final reset. This is why this branch has taken me so long to post. I was unable to find spare time recently to do the benchmarks. I'm planning on doing the next set, 231xx, tomorrow afternoon, though.

Basically, I have almost nothing good to say about this branch. I highly recommend steering clear of it. I'm going to list the best and worst, but keep in mind that I would pick all of them for the worst if I could. The best one in my opinion is basically the least crappy of the branch.

The bottom line:

22024 - statistically the best

22040 - statistically the worst

22013 - my opinion the best

22047 - my opinion the worst
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Questions and comments

This is sort of off topic, but applies here as well:

If you have a question, don't hesitate to ask. I am normally more than happy to assist a fellow member with any question they might have. However, don't expect me to be nice to you or give you respect if you don't do the same. It's a two-way street. If you want my help, just make sure you have your bases covered. Search before sending me or any other member here a PM. I find it's just plain rude to ask me to do your leg work for you.

I normally get PMs from members asking for various things. I answer about 80% of these PMs. The other 20% I will either ignore, or send back a response of "Search." I don't do this to be blatantly rude to you. Everyone needs to get off of the training wheels at some point. There are very good stickies in each forum that you visit on XDA that can answer most of the basic questions.

Most of my PMs lately have been about this thread. I love to help you guys understand what's going on here, because there is a severe lack of general knowledge on this subject. I hope to motivate others to perform the same or similar tests on other devices. I also hope to spread this knowledge around so that others may benefit from it.

My point is that I'm tired of the silly questions that I'm getting asked on a regular basis now. I'm usually really good about responding to legitimate questions. If I don't reply to you in a day or so, and you searched and still feel that your question was valid, then you can send another reply. Don't spam my inbox every few hours asking the same thing. I'll do what any other member would do and block and report you for spam.

I know I can get harsh with people every now and then, but 9 times out of 10, it's justified. If I'm wrong and flamed you for no reason, or over a misunderstanding, I will humbly apologize, however. I want everyone to use the tools at your disposal on this site to try to find the answer on your own. If you're having trouble, anyone of the members here will guide you in the right direction as long as you've done your part. Otherwise, don't hesitate to contact me. I love answering legitimate questions. :)

EDIT: I'm not calling anyone out for this. If this describes you, then take it and learn from it. I don't hold grudges against anyone here. Oh and make sure your title describes your problem on a new thread or a PM. Titles like "HELP!!!" are completely unacceptable.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] vinceweis:
With all of this data (and thank you very much), I started looking for which tests or variables best correllated with your opinion and "feel" for the device. The overall results do not appear statistically significant for a single test. I thought something might be popping out for you that could give a clue as to which test best represents what a user normally sees. Do you have any hints as to how I might change the weighting to match your experience? I will plow through it on my own but your guidance would be appreciated.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Well, basically, these test only measure a few things. You can't accurately measure the ROM you are using based on these numbers. I only play with each build for a short amount of time, so I make my judgments during the benchmarks while I'm moving quickly between the tests. If something pops out at me, I write it in the notes.

When using a ROM, you should try to get a feel for other aspects of it than just the numbers. Sliding through Sense tabs, opening and closing apps, and downloading mail are all good things try try out to test the actual "feel" of the ROM.

The correlation between the numbers and the "feel" isn't always clear. I normally prefer a good balance of speed and video. I watch TV and movies on my device, along with running other business apps, so this is my preference.

If I notice the menus are zippy and apps open quickly, I can safely assume the build manages its memory well. If the build scores high on the OpenGL test, it is ideal for video quality. Video quality is especially important on the Rhodium because of the dedicated GPU. The results, I find, are more accurate here than on a couple of other devices where media is handled through the CPU instead.

As far as the weights go, those should be adjusted to whatever you like. I prefer more emphasis on the CPU and overall speed, than on the video. If you prefer video quality more, then adjust the weights that way. The spreadsheet is set up to accommodate as many people as possible this way. I just have it set up for my own use.

The results of these tests are almost meaningless to anyone not cooking their own ROMs. Once I finish with these basic test results, I'll get to the good stuff and test out actual ROMs. That's where the numbers will begin to mean something. Hopefully this covered everything you asked. :)

EDIT: I didn't see that you had quoted my bottom line results from this time. Don't take those for anything. I universally hated the 220xx branch. I just felt that I HAD to pick a best and worse, since I was making it a habit. They were all bad this time.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] vinceweis:
Understood. Thanks for the view on performance issues. That helps a lot. I've played with cooking ROMs a little but I don't have the time to keep up with everything, nor could I hope to do any better than the Master Chefs around here. I am just an appreciative user normally in the background. I am watching your data for cooking as a hobby.

Thanks for the excellent response and all the help you offer on the forums to other users.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Cooking becomes quite simple once you get into it enough. I've been playing around with several different ones myself. I'm loving my New Orleans Saints one right now! I normally use Energy ROMs, though because of the stability. Jackos ROMs are generally very stable too. When I flash a ROM to my device, that's what I value most - stability. I use this a personal and business line, so I can't continuously restart, or clear space, or deal with Sense's occasional nonsense.

If you're waiting on better numbers, I doubt you'll find better than 21680, 21681, 21682, and 21683. These four builds are remarkably fast in almost every aspect. If you prefer 6.5 over 6.5.3, then 21916 is the most stable and fastest in that line.

Those are my recommendations, but some people prefer, or have need for, the driver bases in other branches. 236xx is another good branch to try. I've only ever cooked once with 246xx. It's a nice one that some chefs on other devices like. 282xx is other "cult favorite" of other devices.

On the Rhodium ROMs, I normally see only 219xx for 6.5. For 6.5.3 it's some form of 23xxx or one of the 2168x builds. The later of which is catching on slowly, unfortunately.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] DJGonzo:
I was using 23563 and it seems I had the fastest ROM out there for the most part IMHO (I would love for you to benchmark it against others too). I *DID* have some bugs, but even then, it rocked.

I want to come back and finish my ROM in a stable form. Could you compare the 23x builds? Is there a reason why you have not tried 23x? Thanks in advance
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
I'm doing them in order from lowest to highest for the builds I have. Look at post number two for the order. Thanks for the permission.

EDIT: I looked through the builds I have. All I have for 235xx is 23569. If you have any others, point me in their direction and I'll gladly add them to the list.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] maan1976:
21911

Good Job nice results consistent with what I get using Sktools and couple other tools .

but let me tell you some shocking news : do you want a build 3x faster than 21914 and 21916 ?

Look for 21911 . jul 16 NRGZ , the fastest I have bench marked in the whole Windows mobile world , interestingly for both speed and video fluidity .
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
I would gladly test it if I could get my hands on it. All the builds listed in post #2 are the ones I've been able to find so far. If you know where I can find it, point me to it and I'll give it a shot.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] Doo Doo:
Here You Go I Found It --> 21911 . jul 16 NRGZ
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
I didn't mean the ROM. I know where NRG keeps those. lol I meant the SYS build by itself. I would prefer it in XIP.bin format, but I could convert it if I need to.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] Doo Doo:
oh i'm sorry what about these then --> SYS Builds
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
And where in there do you see 21911? Compare that list to post number 2.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] Doo Doo:
well excuse me i was just trying to be helpful i won't butt in & try to help anymore sorry for that
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
All I'm saying is that you asked me about 21911, and then posted a link to all of the builds I currently have. I assumed you had found the build you referenced, but instead I saw the same place where I got most of my builds from. I don't understand what other type of response you would have expected seeing as the builds I have are listed in the second post of the thread.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] Doo Doo:
ok i'll try 1 more time is this what your lookin for --> 21911 SYS Build

also what other builds are you lookin for & i will try & help you find them?
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
That's the one. I'll add it to the results as soon as I get a chance. I'm working on 231xx right now.

EDIT: After downloading the build files from the server, I've found that several files crucial to the cooking process are missing. I will try to franken-mod the build, but I'm not promising much with that.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] monomer888:
Is this the build you're looking for?
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
That one had more of what I needed but was missing some files as well. I was able to get it working by adding what I needed from 21916. It shouldn't affect the scores in any way, though as it was mostly language files. I'll add the results of this build to post 2 in a little while.

In other news, I'm probably going to start discontinuing the use of the graphics, Arkaball, and Memcopy tests in SPB Benchmark. They all come out the same, so I don't feel it would change any results.

EDIT: I tested 21911. Just to be sure, I tested 21916 right after. It is noticeably slower than 21916. It isn't bad, just not the best in the branch. You can feel the difference mostly in the menu scrolling. Opening directories were also a bit slower. Results have been posted.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Results for 231xx and 235xx

231xx

I have to say that I was not expecting much from these results. That being said, I was pleasantly surprised by the results from this branch. Each one was pretty nice all its own. They all looked VERY good in the video department. There were a couple of snags here and there, but overall a solid branch.

I started off with 23120. I was very surprised by the speed of the first build that I had available in this branch. I am usually skeptical with the initial one in each branch. 23120 was a breath of fresh air. It wasn't extraordinary, but it wasn't horribly slow and laggy as the trend seems to be.

Each successive build was very similar until I came upon the failed abortion of a build that is 23132. This build is one of the absolute worst that I've ever seen. After testing it the first time, I was so horrified by the results that I thought I had done something wrong. I actually re-downloaded the build and cooked and flashed it again to make certain that it was not my fault. Even still, it scored the lowest in 6 categories in the second test as opposed to 9 in the first. I decided to use the second set of results just in case. Menus lag for a couple of seconds. The start menu took 4 seconds to display. Applications were equally as slow. I was surprised that it got the scores that it did. I could rant about this one for a while, but I'm sure you get the idea.

Moving along, I noticed that we went back to a stable norm for a while. Directory listings in 23136, 23138, and 23139 were very snappy. I did notice in 23139 that IE has some sort of weird auto-zoom feature in it that takes some getting used to. It tries to proportion the page to the screen automatically, but isn't quite as good at it as Opera.

2314x builds were considerably faster in loading applications than any others. 23142 was very fluid. It felt as if the device was built specifically to run it. It wasn't super fast, but just "felt" like it was taking advantage of the hardware to every extent. It's hard to explain better than that. I would highly recommend this build to any chefs looking at this branch.

Overall, there really wasn't anything eye-popping here. It is an extremely stable and solid branch with very few flaws. I had a good time testing everything here, even the bad one.

The bottom line:

23138 - statistically the best

23132 - statistically the worst

23142 - my opinion the best

23132 - my opinion the worst

235xx

I don't really have anything to say about this one. I only have 23569 in this branch. If you know where I can find more, I would be more than happy to add to the results. I liked this build, though. Nothing flashy, but solid. Check the masterlist to see it stack up to anything.

The bottom line: (because why not, right? :))

23569 - statistically the best

23569 - statistically the worst

23569 - my opinion the best

23569 - my opinion the worst
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] monomer888:
Very interesting results, thanks for taking the time to do the benchmarking. :)

Just wanted to point out that there seems to be a 'gap' in the masterlist xls at cells A23 to P24.

Update on closer inspection - just realised you've put a movable white box over builds 21911 and 21914. I assume it must be intentional.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
I don't see any gap. Screenshot?
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] seeM_ZA:
I assume he's talking about this movable block :):
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Well, that makes two of you... I honestly don't see any of the white spaces at all. I've re-uploaded them to be sure. Hopefully, that fixes it.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] monomer888:
Yep that's the one. :)

Yep that fixed it. :D
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] maan1976:
any info benchmarking for 21684
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Just uploaded it. I like 21683 better.

No need to use large fonts. I can read just fine.

I also added 23145 to the list. I really like that branch.

I'll try to get to flashing 236xx tonight. I only had enough time to do those two builds right now.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] Stephane Dufault:
Happy, then sad!

Man, that exactly what I was looking for, thanks for all the work :D

bI was looking for some benchmark to help me choose a build, But too bad the numbers are not more conclusive, there no build that is better across, even within a group :confused:

Some read is better but write is worst, some alpha texture is good but sub texture is bad, some video is good but Flash is bad...and to make it worst your feedback are not supported by the results (do not get me wrong, they are appreciated).

Some recomendations:

- The results that have less then 5% Variance should be ignored in the Overall.

- Some results that are way off the rest, should be ignored (put in a seperate list) as they wrongly influance they Variance (i.e. 21665,21674,21677,22013,22027).

- Split the 6.5 and 6.5.x results in 2 xls, since I am not shure they can be directly compared...

- is there a bencmark that would test the OS&Manila&main App's, like a script that would go through menus, tabs and screens then App like Excel, Opera...to measure how much time it took?

Thanks,
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] Stephane Dufault:
Bench for Manila build?

Seperate question, is there any bench out there for the different built of Manila 2.5 (2012-19)?

Thanks,
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Are you a chef? If not, I believe you might be misinterpreting the results here. I'm not testing the ROMs for "feel" as some people do. I'm using the hard numbers as a base, mostly for chefs, to see where they will need to improve in their own ROMs. These results are not meant for the average user to pick which one he likes and go with that. It is a jumping off point to decide which one may suit your kitchen the best. In the future, I will begin testing custom ROMs, but I have to get through the basics first.

There are no apps cooked into these builds, including Sense/TF3D. It is only the SYS and OEM packages. All EXT packages have been left out on purpose.

I would recommend that you disregard anything that varies 1% or less. 5% variance is a big difference. You are more than welcome to sort the list how you choose, but I'm not going to remove anything from it.

You can sort the file by 6.5 and 6.5.x if you feel like it. I made a masterlist for a reason. It is going to include every build I test. You should keep in mind that this is a project that I began on my own to help myself choose a SYS build suitable for my personal kitchen. Constructive criticism is always welcome, but I wouldn't expect me change much on this. I have it set up how I like it.

As for your other question, no, I don't know of any test program that will test manila versions. When I do begin testing custom ROMs, I'll put out a guide of how I do so.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
21684, 23145, 23146

I don't have much to add here. 21684 was nice, but I found it slower than 21683 in testing. No real issues with it, though.

23145 and 23146 were both very nice to test. 23145 was my favorite between the two. I really like this branch of WM.

I'm working on 236xx right now. They should probably be done tonight. It's started off nice with this branch. There have been no big problems. It's fairly stable so far.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] wmg316:
Do you like 23145 over 21683? I know NRGZ28 switched from the 231xx series on 9/26 moving to the 216xx series. His last 231xx series was 23138...and he never cooked the 23145.

To me, 21683 and 21684 have been the best ROM's I've run (chef made)...just wondering if the 231xx builds s/be looked at again?
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
I was all about 216xx for a while. I'm not sure, though. 23145 was really fast. I didn't think 21684 was as quick as 21683. I'll run them side-by-side when I get a chance. I'm almost done with 236xx right now. I keep putting it off for real life. :( I'll get to it soon, I hope. I just need to take 4 hours out of my day to actually finish them.

In other news, I have come up with a more accurate way to measure the custom ROMs. Those are still a ways off, but I've come up with a method for them that I feel will work best for speed (time it takes to benchmark) and reliability. I may separate the blank builds and move them to the General area of the forum and only keep the custom ROM results here. We'll see. :)
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] skygatevn:
Pls help

I'm use osKitchenZero to dump ship rom RUU_Leo_S_HTC_WWE_3.14.405.2_Radio_15.42.50.11U_2.15.50.14_LEO_S_Ship and next i'm use osKitchenZero to import Builds 28244_MSXIPKernel_SYS_QVGA_WQVGA_HVGA_VGA_WVGA_0409_FROM_DFT. And finish to flash to my HD2. Everything works ok but when i'm have incoming call it not show Caller Name, Picture (I'm already save my contact by PIMBackup and restore contact after flashed ROM) it's only show a white square. Are someone help me fix it. thanks so much
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] anhphe076:
Try OEM u cook! not *cab file.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] osu9400:
What is WMD?
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Weapons of mass destruction? lol What are you talking about?
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] dky500:
Could it be Windows Mobile Development??
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] osu9400:
From the original post:

"280xx, 282xx = WMD."
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Ah, next time be more specific like that. Yes, as dky500 mentioned, it stands for Windows Mobile Development.