yashsingh1/xdaforums huggingface dataset:
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Results

I based these blank uncompressed ROMs on the latest shipped T-Mobile 6.5 ROM (1.91.531.4), disabled all ext packages, and swapped builds. I've listed all of the builds that I currently have. If you have one for me to test that's not on the list, send it my way as long as it's WVGA.

SYS builds tested:

216xx: 21659, 21661, 21663, 21665, 21671, 21674, 21677, 21679, 21680, 21681, 21682, 21683, 21684, 21685, 21686, 21687

219xx: 21904, 21905, 21907, 21908, 21909, 21911, 21914, 21916

220xx: 22013, 22018, 22019, 22021, 22022, 22024, 22027, 22031, 22036, 22037, 22038, 22040, 22041, 22042, 22044, 22046, 22047

231xx: 23120, 23121, 23129, 23130, 23132, 23133, 23134, 23135, 23136, 23138, 23139, 23140, 23142, 23144, 23145, 23146, 23148, 23149, 23150, 23151, 23152

235xx: 23569

236xx: 23651, 23654, 23656, 23658, 23659, 23662, 23664, 23667, 23670, 23676, 23678, 23680, 23683, 23686, 23688, 23689, 23690, 23691, 23694, 23697, 23698, 23699

246xx: 24609, 24610, 24611, 24614, 24618, 24619, 24620, 24626, 24627, 24628, 24630, 24631, 24635

250xx: 25018, 25024, 25026, 25027, 25028

282xx: 28233, 28236, 28237, 28238, 28240, 28242, 28243, 28244

290xx: 29002, 29003, 29005, 29007, 29008, 29009

Custom ROMs tested so far:

NRGZ28 Energy: 21916, 21684, 28244, 23148, 23699 - Update coming soon!

Ondraster LBFAR: 21899, 23569

dotcompt DeepShining: Coming soon!

Problem builds:

These are builds I was either unable to import into my kitchen, or cook for various reasons:

22016, 21664, 23123, 23125, 23126, 23127, 23566, 23694
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Method and relevant information

If you are planning to send me results or anything, be sure to include the version numbers of the programs you used. SPBBenchmark and Test OpenGL are both freeware. Coreplayer Mobile costs around $30.

Benchmark programs used:

1. SPBBenchmark 1.6

2. Test OpenGL 18.01.2010

3. CorePlayer Mobile 1.2.5

Test Method for blank builds:

1. task29 then flash the ROM without the SIM card.

2. The storage card is left in to install the benchmark cabs.

3. Go into airplane mode.

4. Disable switching screen off and turning the device off.

5. Install benchmarking apps

6. Soft Reset

7. Run SPBbenchmark and disable the following:

Built-in applications (whole category)

Arkaball frames per second

Copy 1 MB using memcpy

8. Run TestOpenGL for about 3 cycles of benchmarks (each cycle is 4 benchmarks) and then turn it off

9. Run three Avatar trailers, note down the results (%). Use default Coreplayer settings and run benchmark immediately after files are loaded.

Trailer found HERE. Note that trailers ARE copyrighted material and are NOT subject to fair use. The link provided is to show what I am using. I own the BluRay disk and ripped it from there. The link is also 720p. My original clip was 1080p. Each exported trailer was set to 800x480 resolution.

One trailer was converted to 25fps, 768 kbps - XHQ

One trailer was converted to 23.976fps, 512 kbps - HQ

One trailer was converted to 20fps, 384 kbps -MED

I wasn't able to determine the original qualities from daeron.be's thread, so I made my own. If you know more about it, please let me know for more accurate results.

10. Run Video Flash benchmark with PIE using a downloaded version of the animation found HERE. If you plan on doing this yourself, download it to your device and run it with IE. Just put it in a root dictionary (\) and then input \benchmark.swf address in IE. On some ROMs you might have to wait a while before the flash starts (there won't be any signs of loading). Run the benchmark (if you want your result to be compatible with mine you MUST run it with flash file in full screen).

11. Collect results

Test Method for Custom ROMs:

Same as above with three exceptions.

1. Run SPB Benchmark with ONLY "Built-in applications (whole category)" test disabled. Leave everything else on.

2. Connect the device to the PC after running the SPB Benchmark test. You may have to go to Start > Settings > Connections > USBtoPC and untick the "Enable faster data synchronization" box to get it to work.

3. Copy the "Spb Benchmark Results.xml" from \My Documents to the PC and grab the results from there with the decimals. I use every decimal in the file. I then trim the trailing zeros in the Excel results file.

I figure since some people will probably use this as a guide of which ROMs are "best" or "fastest," that the results should be as accurate as possible, which is why I'm including the decimals in the custom ROM results, but not in the blank builds.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Requests

If someone wants to change the BG colors of the spreadsheet to something they like better, be my guest. If I like the colors, I'll keep it.

Please take the poll at the top of the page. If enough people use this thread with other devices, then I will consider moving it to the Chef's Central forum.

I am slowly adding custom ROMs to the results. I need a few people to notify me when the ROMs you're using are updated. This includes Energy, Simplicity, DeepShining, Core Cell, and other popular ROMs.

Custom ROM testing

Ask your chef if he/she would mind having their ROM(s) tested and compared in my results. See below for rules:

All custom ROMs would be separate of the clean builds, but compared all together.

Understand that once permission is given, it cannot be taken away. (I don't want anyone trying to revoke permission after results are posted to take advantage of my system.)

Have the chef PM me with the green light if they want it included.

All results will either be performed by me, or verified by me.

I will not measure battery performance. See FAQ.

These results are not conclusive to picking a "good" ROM. It's a jumping off point. Test yourself and see which one works for you. This tests certain factors of the ROMs. This doesn't test for stability or feel of the ROM. Do that yourself.

Chefs are allowed to "pick and choose" which ROMs they want tested. If they cook multiple flavors or builds, they can decline to have certain ones excluded from results.

If you have a problem with my methods, you are welcome to do them yourself. I am doing my best to ensure that this is as objective as possible.

Contact

If you feel like contacting me or need to request info or whatever, you may PM, e-mail, or message me on Twitter.

Donate

Coffee keeps me cranking out results. DONATE to keep the caffeine flowing. If I see that you've donated, I will probably take a bit more care in listening to your problem/suggestion/comment. Make sure that you include your XDA user name so I can give you credit.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] uncleswoop:
beautiful... cant wait to see results
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] Jackos:
Subscribed.

Imho you should test blank WM builds. Or at last be sure that the EXT/XIP packages are same. However the idea of testing different builds fits me.

Shame that usually the fastest builds leak memory like hell.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
I used to believe that until we started seeing the 21680 and 21681 builds come out. They really are quite faster than most of the others. When NRG was using the 23xxx builds, that was the case where the faster they ran the more memory was "spirited away," so to speak. I'm finding a much more stable experience with the 21680 and 21681 builds than most that I've seen.

In any case, the results will speak for themselves. The first three should be posted in a few hours or so.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
For anyone who missed it, I've posted the first test file. I only did a single test for each one. In the future I will do an average of three results per test. I only did a single one due to time constraints mostly. That being said, this first file is just a test both for me and for anyone viewing to get a feel for what it's going to look like.

Any questions; let me know.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] monomer888:
This isn't a reg key, but I think there are mortscripts that disable Sense. Maybe this might help?

http://xdaforums.com/showthread.php?p=8684203

I think JVH3 talked about scripts that disable sense too, so perhaps he might be able to help you out there too?

Great work, this is really interesting! :)
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] seeM_ZA:
Using mortscript you can do this:

RegWriteDword("HKLM","\Software\Microsoft\Today\Items\" & ManilaName,"Enabled","0")
RedrawToday
Sleep(5000)

ManilaName is usually HTC Sense, I think. The sleep can probably be shortened. :) Nice thread!
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] sh4d0w86:
why not just build a rom free of all ext packages?

just sys and oem packages.

that way it is a better comparsion of sys builds.

no?

sh4d0w86.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] Jackos:
Disable Sense:

[HKEY_LOCAL_MACHINE\Software\Microsoft\Today\Items\HTC Sense]
"Enabled"=dword:0

Flight mode on boot:

[HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Power\State]
"Phone"=dword:00000000
[HKEY_LOCAL_MACHINE\Software\Microsoft\RIL]
"LastEquipmentState"=dword:00000001
[HKEY_LOCAL_MACHINE\System\State\Phone]
"Status"=dword:00400030
"Radio Ready State"=dword:00000000

Random offtopic: I believe that all roms should have the radio turned off by default.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] seeM_ZA:
I'm curious, why?
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] Jackos:
Because not anyone wants to run the Connection setup after rom upgrade? Not anyone wants to get all those sms messages right after flash. It's just more convenient then taking the sim card out and putting it back into the device when you're ready (not healthy for the card too).

Not to mention the CDMA phones without simcard, having the radio enabled by default in ROM must be a pain in the ass if you flash several times or hardreset to test different software!
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] toucan:
Excellent work on this, cajunflavoredbob!

I look forward to reviewing the results that you come up with on your comparisons.

I am already interested in some of the results that you found in the first tests and will watch that pretty closely.

One thing that I noticed - when I first opened the spreadsheet the colors pretty much come on like a freight train - can those be toned down a tad? These old eyes finally adjusted to them but a lighter shade of colors may be more aesthetic... :D

Thanks for this contribution!
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
If you read the first two posts, I addressed this twice. That's what I plan to do. I need to get a feel for this myself and find ways of doing it quickly before I move on to the kitchen.

Wonderful! Thanks. This will save some time! I agree with you on the radio being off by default. With the number of times I flash my device during the week, it would really help not to hear the alerts going off twenty times when I start up. I use NRG's ROMs by day, backup with Sprite Backup, and then toy with my personal ROM at night normally. Now, of course, I'm going to be flashing even more than that with this new project. :D

I can't promise you that I'll get to it, but I'll put it on my to-do list. I basically stole the formatted speadsheet from the other thread. I was about 2/3 of the way through with making my own, when it dawned on me that there was probably one already made. Their formulas covered some things I hadn't yet thought of, so I'm going to be using this for a while.

In other news, I'll be working on this again today, so check for updates tonight. I'll post any updates in the second post.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] piaqt:
One down (see attached).
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Thanks, but I still need the backgrounds. It helps me find the fields more easily. I'm going to experiment with different colors soon. This isn't my top priority, though. I think, lighter, not brighter colors would do better. We'll see what happens. I'm working on cooking the new ROM right now. Well, not right now, but I'll be starting soon. Hopefully I can get this completed within the week. I can only flash/work at night on this as I need a functioning device during the day. :D
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] toucan:
Haha... I wholeheartedly agree - it is absolutely necessary to have the colors - but perhaps not so bright! :)
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] sh4d0w86:
i thought that was the plan.

saw that you didnt do that.

so. figured. i misunderstood.

glad to hear that is the plan.

looking forward to results.

thanks.

sh4d0w86.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
For anyone following this, check the fifth post every now and then. I'll post things to help out with for anyone who wants to contribute. Right now my biggest issue is getting XDA_UC to cook into the clean ROMs properly. I played with it for several hours last night with no positive results. It doesn't HAVE to be XDA_UC specifically, as I know there are other UC applications out there, I've just had most of my experience with that one. I need to be able to auto install cabs, import reg files, and copy a file to root.

Other than that, things are moving slowly. Here's a list of the builds I'm working with today. I hope to have a new result file posted tomorrow morning. I wanted to have it done tonight, but installing and setting up things on my own is eating up a fair amount of time.

216xx -

59

61

63

65

71

74

77

79

80

81

82

21682 is the newest build in that series. Hopes are high that it will outperform the others as they seem to be getting better with each successive build. We shall see... ;)
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] osu9400:
Great info, but this brings up other questions:

1. Where do the leaked builds come from? Note: I am looking for a better answer than simply "Microsoft." :)

2. It seems MS is putting quite a bit of work into 6.5.x these days. Seems odd condiersing 7.x is getting all the press. Are they adding new features to 6.5.x?
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
1. Click the names in the third paragraph of the first post. I removed the actual quotes to clean it up some. I'll add something along these lines later on.

2. I'm not a big fan of WP7. I don't like the look of it, nor am I fond of the software itself. There are still lots of devices using WM6 in some fashion. MS isn't the only ones developing the builds.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
216xx benchmarks posted

The first set of benchmarks using a "clean" ROM was just finished and posted. I used all the builds I had available in the 216xx series. Here is my break down of the testing.

21665 was statistically the best build, but I liked 21682 the best. I found a much faster transition in and out of menus, and programs. Touches seemed more responsive, and scrolling was a pleasure. It just seemed to take advantage of the system resources more than the others did. 21681 was a close second in my opinion, even though it did terrible on the tests.

The 2167x builds seemed to be made for smooth video over anything else. They each looked amazing when doing the 3D tests and video playback testing. Videos played silky smooth on them, but they were awfully slow doing anything else. 21677 was by far the worst of the bunch. It lagged doing the simplest things, however, it did have great video quality.

2166x and 21659 were pretty average in my opinion. I didn't really notice anything spectacular about them. I did note that 21663 felt really fast, but that was only until I loaded the next ROM. 21665 was fairly swift and was visually stunning in all of the video tests.

The bottom line:

21665 - statistically the best

21677 - statistically the worst

21682 - my opinion the best

21677 - my opinion the worst
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] wmg316:
Hmm...

Wondering if my graphics variance is due to a difference between Tilt 2 and Touch Pro 2???

I tried stock (well...Energy) drivers, and Neo's Driver pack (my results...just a clean install of Energy ROM was lower results)...and I was significantly lower in some of the graphic areas.

For the tests run, I ran each 5 times and took the average.

'alpha texture' and 'color blending' are WAY OFF from your results...

Any ideas?

Tests run on 'Energy.RHODIUM.21682.Sense2.5.Cookie.2.0.Oct.20' with OCT at 768.0

EDIT:

I do agree...21682 is the fastest cooked Energy ROM yet!!!
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] wmg316:
I graphed...if your intersted...
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] wmg316:
I love stats! But I'm not sure why you weighted the way you did?

These weights are good for a DVD player...not a cell phone! ;)

Why would the CPU test not be weighted higher, followed by the memory tests, followed by the graphics tests?
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
The difference in your results is likely because you're comparing apples to oranges. The other builds in the file you posted are blank ROMs. There are no EXT packages. I only included the SYS packages from the build and the OEM packages from the latest T-Mobile stock ROM. You tested with a heavier ROM. If you want to compare an Energy ROM, use one of the reference builds and compare it to my initial test file with the three results.

I thought about adding graphs, but I wasn't sure if anyone would care that much as I have the file set up to sort the builds out by whatever test you want. If there's demand for it (and I have time), I'll think about adding graphs.

I noticed the same thing at first. When did you download the file? I had changed the weights and uploaded the new file yesterday evening to account for that. You are more than welcome to change the weights however you feel once you download the files, though. I've got it set up right now for my needs. Just to be sure, though, I've uploaded the spreadsheets again.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] wmg316:
Ahhhhh...apples and oranges it was. I downloaded your first file today which were your results with the NRG ROM's and compared to mine...much more in line. Sorry; my oversight!
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
New build results posted: 219xx

I FINALLY got around to testing out the next batch of builds: the 219xx series. There are seven builds that I found here. These are all 6.5 builds with the start menu at the top. It was kind of neat playing with them as I normally prefer 6.5.3 over 6.5. Most of these builds were pretty fast. They felt faster than the 216xx ones I reviewed last time.

It seems that these builds follow the trend that you can either have amazing video quality OR amazing speed. It doesn't appear that you can have both. The build that comes closest, though is 21916. This one gets my pick for the best overall and my favorite. The only flaw I found in it, was that PIE seemed to be slow to open.

The results disagree with me, however, as 21914 was statistically the best overall build. I feel that 21914 was outstanding in video quality, but wasn't quite a swift when scrolling through menus or opening apps. It wasn't slow, but compared to 21916, it was a no-brainer. I look forward to seeing what comes next in this line, as they felt as if they got progressively better with each higher build.

Statistically, 21905 was the worst overall. I would have to agree. In a set of builds full of average performance, this was the slowest. Menus lagged, apps took too long to open, and video was terrible. This is the worst I've seen so far. I feel sorry for anyone using this build, honestly.

Overall, there wasn't anything else to really jump out of this set. Most of the builds were mediocre and either had great video or great speed. 21916 was the only one I've seen to have a very good mix of both. It was quite impressive, actually. I am actually curious to see what else comes out of this build line.

The bottom line:

21914 - statistically the best

21905 - statistically the worst

21916 - my opinion the best

21905 - my opinion the worst

Other News:

I've added 2 new spreadsheets to the second post. One is the results from this set only, while the other is a "Master List" with all of the builds listed on it. I am planning to move this onto Google to host it soon. This should make viewing it a bit easier and eliminate the need for me to upload this several times. You guys would also be able to view updates in real-time, since TV is all over this "live" thing lately. (Cleveland Show anyone?)

If you would like to change the background colors, be my guest. I don't have a problem with them, but some people do. If you change the colors, post it, and we'll see what everyone likes.

I will be working on 220xx tomorrow, so expect those results soon. There are twice as many of them, though, so I expect it will take quite some time.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
Benchmark custom ROMs

If you want me to do benchmark results for custom ROMs, I will do so soon. There are a couple of stipulations on doing so, however. If you check out the original thread for this project, you'll come across a huge flame war. This was partly due to the fact that they were testing and comparing results of some chef's ROMs who did not want their results posted. While I personally don't agree with them trying to "censor" the results, I do understand the reasons. It is for this reason, that I am requiring you to obtain your chef's permission before I do a benchmark for it.

Don't just make a post saying, "Hey test the latest Jackos ROM for me. Don't worry, he said it was ok." If you do this, I will probably either mock you or ignore you. Get your chef to either send me a PM with the go-ahead, or link me to the post where he/she said it was alright. I want this written out so that there are no mistakes. Make sure the chef understands that there are no "take-backs" to doing this. Once they say ok, they can't go back on it afterward. This prevents people from using me to see the results and then wanting it removed if it isn't at the top.

With that being said,

DO NOT

post comparative results from multiple custom ROMs on this thread without the aforementioned permissions. I will list which ROMs are ok to use as I get confirmations. That about covers that. I will post more on this at a later date if there is some interest in it.

EDIT: I received permission to use ROMs from OndraSter, Jackos, dotcompt, seeM_ZA, and NRGZ28 without restriction.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] osu9400:
Good info, but what is COM1, COM2, COM3, etc...

I saw that HTC updated the Tilt 2 this month with build 21887.5.0.86.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
It's just a name for the branch. I don't have any of the 218xx builds yet. I'm working on it, though.
[Touch Pro2, Tilt 2 Windows Mobile ROM Development] -Jonny-:
Thanks, there's some really interesting results here.