[Touch HD Windows Mobile ROM Development] daeron.be:
Hello guys, I've decided to make a benchmark of latest cooked roms, because nobody else did anything like that for our beloved Blackstone.

It happened again - our spreadsheet grew too big for another online spreadsheet hoster. Therefore, we have to swich different way to distribute our work - via attachments in this topic. To get newest results, just look for a latest post from me, Houms or MAMeingast. Sorry for the inconvenience, but I'm sick and tired of all problems with those services.

To measure performance of each rom I use two free programs:

SPB Benchmark (v1.6.0, build 62)

TestOpenGL by heliosdev (v. 09 May 2009)

Coreplayer (always say what version you used - e.x. v1.3.2a) * optional, it's not a free app!

Matrix trailers:

http://shintak.info/ppc/download/betaplayer/sample/RL_XQ_640x480_1500_128.avi

http://shintak.info/ppc/download/betaplayer/sample/RL_HQ_640x352_1024_128.avi

http://shintak.info/ppc/download/betaplayer/sample/RL_MQB_320x240_512_128.avi

To score "overall rom performance" I'm checking how well is a rom doing in a single benchmark (rom's result/best result or best result/rom's result, it depends on whatever better score is a higher number or a lower) and taking an average of all one rom's results. More % does the rom score, better its overall performance is. If any rom scored 100%, it'd mean that it won all benchmarks.

I've added 'weights' function, It's working and they affect overall score. Give me your feedback about how big they should be for each test. If you export this spreadsheet you can edit them to fit your needs!

What are we doing if rom fails some test? That's easy. We make this test's cell white, and put there an average rom's result.

Test Method:

1. Flash the rom and do a hard reset.

2. Go into airplane mode

3. Disable switching screen off and turning the blackstone off

4. Disable all today screen apps and plugins etc * optional

5. install benchmarking apps

6. reboot (soft-reset)

8. run SPBbenchmark : disable external software tests like word and IE or file explorer (whole category)

9. run TestOpenGL for about 3 cycles of benchmarks (each cycle is 4 benchmarks) and then turn it off

10. run three matrix trailers, note down the results (%). Use default Coreplayer settings and run benchmark immediately after files are loaded.

11. collect results

FAQ:

1. Shouldn't you mention radio version you're using every benchmark? Does radio version affect the results?

I hope it doesn't, because I turn all connections off - I go into the airplane mode

2. Does hard reset after each flashing affect the results?

I have no idea, but somebody said it could happen, so I'm doing it every time I run a benchmark.

3. What are these benchmarks for?

These benchmarks are just for me to help me choose the "best" rom in terms of performance. And I'm a good guy and I let you see these results and even correct me if I make a mistake :). And Houms asked me if he can join. Sure! :) And now when I cannot test rom for myself, you all are welcome to send your own results, as long as you don't cheat and follow the rules how to do it (see the sacred METHOD above) :)

4. What version of rom (e.x. Miri's) are you testing? Big, small, Premium, Deluxe, Standard, Non-Standard etc?

Well, I try to specify what version I test, but sometimes I forget. And yeah, I always use Standard Miri's roms. But does it make any difference, if they only differ in things such as TF3d or not-TF3d, if I disable it?

5. OM*G! My rom is the best and in your tests it's not! Remove these results immediately, because they confuse people, and don't you dare to test my rom ever again!

Yeah. Sue me. :)

6. Can I send you my results?

Sure, I'd be glad to add them but you have to follow the Method described and send them via e-mail < daeron [at] [put-my-forum-nick-there] >. ;)

7. Are those benchmarks accurate?

Well, I hope they are ;). Some roms were tested twice, and differences were always smaller than 1%, but you can say that they might vary by 1% (I'm talking about overall performance difference!). If you aren't sure, you can always check. But do this on a clean rom with the method above.

Big THX goes to houms and MAMeingast for hard work on this project.