[Touch Pro2, Tilt 2 Windows Mobile ROM Development] cajunflavoredbob:
FAQ

When posting to this thread, please use proper English only, and do not use profanity.

You should not type like a high school girl, either. Leave the "plz," "sum1," and "u" back on the playground.

Ask for help like an adult and be treated like one.

1. What are these benchmarks for?

Basically this is just a measure of performance on different SYS builds to see which one is the fastest. These tests do not indicate which build is really “the best.” It is a measurement of speed on the process of copying files, reading directories, and playing video. To truly “test” a ROM, you need to spend a few days with it.

2. Which radio version are you using?

To the best of my knowledge, this has no affect on the results. However, my current radio version can be found in my sig. I do not change it for these tests.

3. Does a task29 before each flashing affect the results?

There are problems with "ghosting" in ROMs where residual data is left over from a previous ROM. To avoid this, I perform a task29 first. If you want to know more about task29, then visit that thread.

4. Are those benchmarks accurate?

I test the ROMs with SPBBenchmark once as it averages the results of 13 tests each. I use three rounds of OpenGL and use those results as it takes the highest result from there, which is what we want. An average in that category is useless as we want to know what the best result is for this, not the average. I run each Avatar trailer three times and take that average. Flash is pretty straightforward and kind of just for fun, so I only do one test. All of this means that the results may vary by a very small margin. If you don't believe my findings, you are always welcome to test yourself. Anything within a 1% margin of error is acceptable.

5. Why do you disable "Built-in applications (whole category)," "Arkaball frames per second," and "Copy 1 MB using memcpy" for SPB Benchmark on the blank builds?

They don't make a noticeable difference in the results. I could grab the *.xml results file from \My Documents, but it's not that big of a deal in the blank builds as they are just a stepping stone for chefs to see what they like and don't like. If you want to see the results, test them. I use them for custom ROMs because it makes sense to do so. People will interpret these results loosely, and it is better to be more accurate with them.

6. Why aren't you using Linpack?

Great question! I would if this were Android as there are current programs for it. The only one available for use on the WinCE platform is almost three years old at this point. Also, SPBBenchmark measures MFlops, so I don't really need it, do I?

7. Why don't you test battery life?

Battery life testing takes too long, plain and simple. When you flash a new ROM, getting the battery set properly takes about two or three cycles (drained to 10% and then charged to 100%) before it "settles." Even if I did nothing but play movies with the backlight and sound at max while downloading files via wifi and using Bluetooth file transfer, it would still take about an hour or so to drain the battery enough to charge it back again. You would need to drain and then charge AT LEAST once, preferably twice, before running a battery meter to test. This means I would be stuck on one ROM for hours before moving to the next one. My current set up allows for about 30-45 minutes per ROM to test. This is much more reasonable than 4 hours each, don't you think? If you decide to do test it yourself and want to share, then be my guest.

8. I didn't see you use xxxxx version of a build. Why is that?

I either don't have access to that build, or it isn't a WVGA build. If it is a WVGA build and I don't have it listed, send me a link, and I'll test it out.

9. Can I test the ROMs too?

Sure, why not? I won't use your results, though. There was a massive flame war in the aforementioned original thread that was partly fueled by other users posting results. I don't plan to start any nonsense here.

10. Would you mind testing a custom ROM for me?

YOU MUST GET THE CHEF'S PERMISSION FOR THEIR ROM'S RESULTS TO BE POSTED. I need them to PM me with an ok. I basically want it in writing. NOTE: Once a chef says it's ok to test, I will not remove them from the tests; make sure they understand this.

11. Why am I getting different results?

There are several little nuances that can influence the results in minor ways. If you're getting significantly different results than I am, then I'll look into it again. Keep in mind that using a different device than I am may affect the results as well. However, I will not use any other users' results without verifying them myself.

12. Some of this material looks familiar. Did you steal this?

Yes, basically I did. I figured I'd restart something similar for reasons I've mentioned as well as making it more Rhodium specific. Besides, the original poster, daeron.be, hasn't even logged on in a long time. His thread hasn't been updated since September of 2009 as well. I'm going to use most of their formatting and previous material to build on. If someone has a legitimate problem with this, then PM me.

13. I don't believe that you got "X" results with "X" ROM! I think it's WAY better than all the rest!

I don't care what you think. That isn't a question anyhow. I'm doing this for myself and sharing with you all. Either take it for what it's worth or do it yourself.

14. You didn't respond to my post. What gives?!

I have a life. I'm on XDA way more than I should be as it is. I will generally respond to questions and the like. If you just post to say "Good job" or "Thanks, man," don't expect me to respond. I might, just don't expect it. Just consider this answer here as a general "Thanks" and "You're welcome."

15. You're from New Orleans? What's it like?

People are wonderful; food is amazing; Mardi Gras rocks. Let's keep the thread on topic, please.