[8125, K-JAM, P4300, MDA Vario Software Upgrading] jorge_culv:
Many of you guys are good at programming and can whip up cabs in your sleep--I'm not so good at C++, etc but here's my contribution, hope it helps someone.

Analysis of Wizard ROMs

Objective: Try to determine if there is performance improvements in newer ROMs. Also to see if the 12->8 MB paging size has a measureable effect in performance.

Summary of Results:

There doesn't appear to be much of a measurable difference in performance among these ROMs, with the exception of the last two ROMs tested. For the rest of the pack, the hardware responds about the same with a little noise. Summitter's 2.17 ROM appears a little above average, but this could be just a testing glitch, and its not significantly above the others. This appears to indicate that the upgrade to decrease the paging pool has no effect (to this benchmark).

I'm concerned about the consistency of results with the last two. They were performed under slightly different test conditions (at home, vs at work for the others), and there's no reason ShogunMark's at least (if not both) shouldn't be closer to the others in performance. I plan to rerun these tomorrow and see how they compare.

Method: I flashed a decent cross-section of the ROMs out there--don't feel insulted if yours isn't included. In fact, I'd be happy to add yours to the mix--just ask! For each ROM listed, I flashed then immediately recorded stats from Settings>System> About, Device Info, and Memory. Then added PC Pro Labs Pocket PC Benchmark and embedded vb runtime (required for Benchmark), removed USB cable, soft-reset and ran benchmark software once to obtain last few columns of results.

Assumptions: The biggest assumption is that this is decent benchmarking software. While it was written for WM2003, its probably not a big stretch to assess WM2005. I konw Spb has a benchmark program, but it ran for about 1.5 hours on the first ROM. Too long for me, guess I'm just impatient! Other big assumption is that higher values are better, although this appears inversely releated for the file read/write benchmarks compared to the kb/sec measurements. This may also be true for the others, butsince they're all about the same it really doesn't matter.

Additional Observations: At the very least, this was an interesting exercise to record some comparable data regarding the different ROMs. ROMs have evolved over time, and sometimes there are questions about which ROM contains what, etc. This might lead to another idea where we keep track of the ROMs in some type of registry to track consistent information about each.

Analysis Improvements: This could be improved by using more current benchmark software. Spb Benchmark is a decent candidate, but takes a while to run. Also, the analysis could be improved with many benchmark runs per ROM instead of just one run. This would average out the "noise" betweens runs and might give a better indication of slight performance trends.

Finally, the method used above will be repeated under similar conditions for the last two ROMs to see if the provided results are valid.

Terms:

BM - Benchmark

GDI - graphics display test

CPU - central processing unit

kb/s - kilobytes per second (used for read/write tests).