[Nexus 7 General] bftb0:
As far as the micro bench tests are concerned, I'd say: look closely at your data.

You have "cpu(secs)" values of 0.75 seconds and 0.001 seconds. :eek:

Seriously? Benchmark scores that are 750 times better or worse than others due to the ROM? That is incredibly difficult to swallow. More likely, the benchmark exhibits a pathology that appears somewhat randomly depending on uncorrelated activity.

As far as Antutu goes, I'll put a little bit of data where my mouth is. I went and did exactly what I recommend: measuring Antutu over 14 iterations using exactly the same ROM.

I was pretty careful & did the following:

- Started with a CM10.0 Nightly ROM (20130504), no gapps.

- Then, I removed all the following system APKs from the ROM:

ChromeBookmarksSyncAdapter.apk

ConfigUpdater.apk

FaceLock.apk

GenieWidget.apk

GmsCore.apk

GoogleBackupTransport.apk

GoogleCalendarSyncAdapter.apk

GoogleContactsSyncAdapter.apk

GoogleFeedback.apk

GoogleLoginService.apk

GooglePartnerSetup.apk

GoogleServicesFramework.apk

GoogleTTS.apk

LatinImeDictionaryPack.apk

MediaUploader.apk

NetworkLocation.apk

OneTimeInitializer.apk

Phonesky.apk

Provision.apk

SetupWizard.apk

Talk.apk

Talkback.apk

VoiceSearchStub.apk

This produces a ROM which is pretty quiet in terms of having extraneous things running. In addition, I ran all tests with WiFi off to attempt to keep the tablet as quiet as possible. I'm not sure if you did that or not with your tests.

I then sideloaded four apps:

Antutu (v 3.3)

Trickster Mod (v 1.20.623)

Advanced Task Manager (v 5.6)

ES File Manager (v 1.6.2.5) -- not used

GScript Lite (v 1.1.2) -- not used

Finally, I ran the Antutu benchmark using the same ROM 14 times. For each trial, I:

- rebooted the tablet (this replicates a boot of a new ROM install)

- checked settings to insure that WiFi was off

- Launched Trickster Mod and changed governor to performance, as in your tests

- Launched ATM to see what was running, but otherwise did nothing. (Note - this app is NOT used as a task killer).

- Ran the full Antutu benchmark exactly once for that boot cycle. If the benchmark hung, I rebooted the tablet rather than re-starting Antutu.

Here's the results (or, click on the "spread80.jpg" attachment) as a JPG file.

I recorded the min/max values for each Antutu benchmark out of your spreadsheet (yellow highlight) into mine, and placed them side-by-side with the min/max values I experienced from simply repeating the benchmark. The "Overlap in Results %" score is computed relative to your reported min/max range as

min(Ymax,Mmax) - max(Ymin,Mmin) / ( Ymax - Ymin )

where Y{max,min} = Your max/min score, and likewise M{max,min} = My max/min score.

As you can see, some of the individual Antutu benchmarks have a natural variation on a single, stripped-down, quiet ROM that are as large as what you measured across all ROMs (e.g. RAM and CPU integer tests), and others (CPU float, 2D Graphics, 3D graphics) are pretty large - 30%.

Because I made my measurements with a stripped-down ROM, it is likely that my min/max ranges are far smaller than what will be observed for a random ROM. That means that ROM-to-ROM comparisons will generally be far worse (natural variation in benchmark statistics for each ROM), and thus even more horrible than what I've shown here.

A Rank score for these large-overlap individual benchmarks is not much different than a pure random number.

What you are trying to figure out is a worthy enough effort, but the method you are using is fraught with an overwhelming amount of statistical problems due to the nature of what you are trying to measure and how you are measuring it.

I don't want to discourage you (I realize how much time it takes to do the testing and put the results together), but by the same token, bad information is sometimes worse than no information at all.

Think of it this way. Imagine that there was a special, custom-built "ROM" that only had a single application running on it - absolutely nothing else was running on the tablet. Let's further suppose that that application was Antutu. It might run fairly repeatably, but not exactly repeatably - even the kernel itself will run some of it's own threads on a schedule that is unsynchronized with the running of Antutu.

Now, add a few more "apps" or background services, whatever. Each of these things will mostly sit idle (not runnable - waiting on an timer, interrupt, I/O completion), but occasionally will become runnable. The kernel scheduler will load those thread contexts every so often so long as they are runnable, and so they will "interfere", or "steal away" resources from the benchmark to the extent that they are runnable.

And that is kind of what you are trying to measure, something like asking the question: "How much do all these pre-installed applications & services that vary from ROM to ROM compete with a foreground task/app that I am interested in?"

The problem with a small number of runs is that all of the crap which might spontaneously become runnable, (and thus cause a lower benchmark score) during the time your benchmark of interest is running will:

- produce loads of widely varying duration

- occur on schedules from milliseconds to days which are random relative to the start time of your benchmark

So, the problem with running a short-duration benchmark is that these "interfering loads" will be present sometimes, and at other times not - even if you never change the ROM. The only way to average the effect of all of the unsynchronized loads is to run a benchmark that has a duration of hours or even days - that way, when it is run, it is certain to capture all extraneous activity every time it is run. (Which is to say, repeatebly)

But, I'm pretty sure that you actually would like to use your tablet for fun stuff rather than commit it to weeks of testing.

Sure you can take multiple measurements and compare averages. But note that the scientifically accepted method for comparing mean values requires computation of a "confidence interval" that depends on both the observed variation (std. deviation) for each group of measurements, plus the number of samples. Typically - when the natural variation of a measurement (std. deviation) is large relative to the difference in means between two groups of measurements, you need a tremendous number of samples to be able to tell if a difference in average values is statistically significant.

Is there a way to measure what you are after? Perhaps. Here I'll suggest one way:

Enable the "ondemand" governor.

Enable the Developer option "Stay awake"

Plug the tablet onto the charger (so it stays awake)

Record the contents of /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state

Wait for 8 hours (go to bed maybe)

Record the contents of /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state

the difference between the frequency bin counters between start and end will *partly* reflect the cumulative total of "idle" activity of the ROM. (Partly because it doesn't count the work done by multiple cores on line).

Note in this case a benchmark app is not used at all - what you measure is the amount of idle activity caused by pre-installed applications and frameworks.

good luck