yashsingh1/xdaforums huggingface dataset:
[Hero, G2 Touch Android Development] Arkymedes:
Added EZHero 2.9 and I might say that it blew my mind away. Awesome performance in 3D.

It raised the bar, that's for sure.
[Hero, G2 Touch Android Development] kad3t:
Just out of interest, how many of each tests do you run on each set up? When I've done my testing I've noticed that doing 3 of each was a bare minimum and 5+ was recommended for averaging the results afterwards as they could differ dramatically between tests on same ROM. ^__^
[Hero, G2 Touch Android Development] Arkymedes:
Hello.

I discard the first result for caching purposes then do 3 more and average then.
[Hero, G2 Touch Android Development] kad3t:
Awesome. Thanks!
[Hero, G2 Touch Android Development] adwinp:
3 samples will never make a statistically representative group.

You should take AT LEAST 10 samples (even 5 would be too little); discard any results x>5%, THEN make an arithmetic mean.
[Hero, G2 Touch Android Development] kad3t:
True that, but given the fact that most people run their tests once and post the results straight away, having somebody run them multiple times (even 3+) produces a result far more accurate than a single run.

I've runned mine at least 3 times (some of them like linpack and AndroidPI around 25+ each) but knowing that Arkymedes does his more than once is comforting because there obviously is a difference on each run but we also can't have them running infinite number of times to get the most accurate reading. Few is enough the get a rounded estimate and if he keeps the same number of tests for each ROM then the mistake threshold should be the same for each ROM and results whilst not being perfect should at least be on the same accurancy level for all ROMs.
[Hero, G2 Touch Android Development] Arkymedes:
Hi. I agree with you but that's not my intent as I don't have the time nor patience for that. Also, this comparisons were made for my amusement only and I decided to share it here. It's pretty much "take it, or leave it".

Exactly my point of view about it. And also, they are relevant since I use the same method and same hardware for all the ROMs.
[Hero, G2 Touch Android Development] -2y2k-:
how can HeroDroid have one of the highest AN3DBENCH and the lowest GPU score??

can anything be made out of that?
[Hero, G2 Touch Android Development] Arkymedes:
Hello.

An3DBench measures 3D. Softweg's Benchmark, 2D.

Cya!
[Hero, G2 Touch Android Development] -2y2k-:
so its better at 3D than 2D??
[Hero, G2 Touch Android Development] shokkapic:
You have VanillaEclair 4.0, Cronos Droid 1.7.1, Androbin_CR3_5.0.7, SenseHero 2.1 Build 5 and Hik@Ri 2.2.0 99 still to test. :D
[Hero, G2 Touch Android Development] Arkymedes:
That's what the test says.

Yes I know.. the world cup is holding me down! !
[Hero, G2 Touch Android Development] shokkapic:
(Daqui a uma semana logo vemos quem ganha. :P)

eng: In one week we'll see who wins the match Portugal vs. Brasil.

If 2D is worse than 3D at HeroDroid, obviously there is a problem somewhere. We should tell this to its developer. :)
[Hero, G2 Touch Android Development] -2y2k-:
june 25th that day will be a GREAT game! lets see these next games before that finale in the group of deathh

edit:

could we get some updates on HeroDroid 0.7 and Chrono's 1.7.1 (or is that not done here anymore??)
[Hero, G2 Touch Android Development] shokkapic:
Also EzHero 3.0, HeroDroid 0.7 and Villain 10.3. :P

A good idea was to test the Official 2.1 too, so a better comparison could be made between which ROMs are being good or bad developed in terms of 2D and 3D rendering. :)

I know I'm already giving you a lot of work, but if I had the time, I would be very pleased to help you or even make a new thread just about CPU benchmarking. :)
[Hero, G2 Touch Android Development] Arkymedes:
I try to do as much as I can because it's very inconvenient also since it's my main phone.

About the CPU bench, you could start yes. The thing is to have a good method and use the same hardware all the time.

PS: Nao se esqueca que o ultimo jogo foi 6x0 pra gente :P
[Hero, G2 Touch Android Development] shokkapic:
Androbin_CR4_5.0.7 released. :D

PS: Na altura o C. Ronaldo não jogava tanto, nem tínhamos uma equipa tão sólida. Podemos não ganhar muitos jogos, mas também não os perdemos :P
[Hero, G2 Touch Android Development] -2y2k-:
thanks for the updates on newer ROMs! :)
[Hero, G2 Touch Android Development] Arkymedes:
Updating in a few minutes with some new ROM's

PS: Ahhhhhhhhhhhhhhhh! Para para! Esse jogo foi em novembro de 2008! Vai me dizer que o C.Ronaldo nao jogava tanto ANO PASSADO??? Em fevereiro de 2007 Portugal fez 2x0 no Brasil com C.Ronaldo e ele destruiu a selecao brasileira!

Mas de qualquer maneira, tenho certeza que vamos perder dessa vez. Odeio esse time do Brasil retranqueiro e defensivo :(
[Hero, G2 Touch Android Development] -2y2k-:
just a request, if you could PLEASE separate sense and non-sense?

doesnt have to be another graph but maybe just keeping sense at the top in alphabetical order and then starting again with non-sense right below it.
[Hero, G2 Touch Android Development] shokkapic:
PS: Em 2007 ainda tínhamos alguns jogadores bons da selecção antiga (a selecção que o Scolari usou no Europeu de 2004 e no Mundial de 2006). Depois do Europeu de 2008, foram-se embora os restantes. Agora, se compararamos 2008 com 2006, mais de 90% da equipa mudou... Agora em 2010, ainda pior. Só temos o C. Ronaldo, o Simão, Raúl Meireles, Deco que sabem jogar. Os outros 19, não têm tão grande destaque. E o seleccionador também não é grande coisa. Tem medo de arriscar o jogo. Vamos lá ver como corre o jogo contra vocês :P

One thing that I've noticed is that it doesn't make any sense to have 3 different clock speeds in the same graph.

E.G, more than 90% of the Hero's can't run @ 768Mhz, and ~50% can't @ 710Mhz.

Unfortunattely, I'm included in those groups. The maximum clock I can use is 691Mhz, just as the most part of the people.

I think the tests should be made with the clock locked at the maximum ("Performance Mode", not "On Demand"), and maybe with a value lower than 691, maybe 652 (if not possible, we could ask the developer to release an edited kernel with that added value).

The objective is to compare the GPU stressing among the same CPU clock speed, for all the different ROMs.

If you compare the same NVidia graphic card with a Pentium II and with an i7 CPU, it isn't properly fair, is it?

Let my 1st round exams finish (two this week) and I will try to help you with this, cause it isn't easy, to be able to analyse all the different ROMs and the different ROMs updates.

Also, and per example, if EzHero 2.9 has the same specs than EzHero 2.7, discard the 2.7 release.

I'm tired of writing such a long text... 8-)

But good work Arkymedes. ;)
[Hero, G2 Touch Android Development] Arkymedes:
I understand what you say but in my case, since I don't have problems with my Hero going to 768MHz, I'm running the test at the speed the developer intended the ROM to be used. And this graphs are for illustrative purpose only.

Also is complicated to underclock all of them to the same level for the exact reason above. Some developers might not like the idea, since some ROMs were not designed to work under certain speeds.

Anyway, I'm using as you thought, the maximum clocked allowed by the ROM's kernel under "Performance" settings using SetCPU.

And I don't agree with your analogy about the Pentium/Nvidia comparisons because in the Hero's case, the CPU=GPU.

But I will try to redo some of the tests to put all of them with 691 maybe. It's just too time consuming to do the tests over and over again just to change a little setting that will hardly change anything significant in the final results as you could see with some of the tests.

Also, I like to keep different versions of the same ROM just for comparison purposes, as you could see with the eZHero ones.

Thanks for the feedback! :)
[Hero, G2 Touch Android Development] -2y2k-:
now looking at it the results, how would you analyze this? where do the ROMs stand at the end from a Sense and Non-sense point?

none of these tests tell you about the responsiveness so what do you get from these results? or is it just a pure numbers game?

dont get me wrong i appreciate your work GREATLY, just curious...
[Hero, G2 Touch Android Development] Arkymedes:
Hello, I did some clean up here. Re-did the bench for the ROM's that were not clocked to 691. So, now you can compare then from a clock perspective at least.

I didn't sort them by "Sense or No-Sense" because for me it's not relevant, since there are ROM's that are not entirely based on Sense neither Vanilla. For example the Daguu Espresso.

And this tests TELL you about responsiveness. Specially the Softweg's one that is the most relevant for this case. Well, my opinion at least. The main launcher is heavily dependant in how the phone can handle 2D drawings and processing no?

Anyway, I will take a better look at the tests and better format them to make more sense :P
[Hero, G2 Touch Android Development] shokkapic:
7-0. :D

Good luck for the next match. ^^

Also, great job re-running the tests @ 691Mhz. ;)
[Hero, G2 Touch Android Development] Arkymedes:
Yeah man, that was an excellent match! It will be very interesting to see Brazil x Portugal, even though I think they will make a game of "comadres" :P

And thanks :) I added Androbin CR4 and Villain 10.3
[Hero, G2 Touch Android Development] -2y2k-:
superhero 2.0 PLEASE!
[Hero, G2 Touch Android Development] Feeyo:
Please use Cronos 1.5.1 for your performance tests.. 1.7.1 is a bombarded crap release :)
[Hero, G2 Touch Android Development] -2y2k-:
waiting for your 1.8 release!!! :)

and obviously a froyo release if your doing one! haha.
[Hero, G2 Touch Android Development] hansmrtn:
How about a 3rd batch soon? EzHero 3 was pushed out to fast, it was a fail :D
[Hero, G2 Touch Android Development] BRijpstra:
This was such a nice thread! Is the whole benchmarking project abandonned by now or have I missed a thread?
[Hero, G2 Touch Android Development] ArunasV:
Could you please put Kimera 2.1 results here? Thanks
[Hero, G2 Touch Android Development] Arkymedes:
Nothing abandoned here :) Time is something that I rarely have, so I try my best to be as max I can here.
[Hero, G2 Touch Android Development] BRijpstra:
Time, such a scarse item indeed :-).

But goody, I'll keep an eye on the thread then!

/edit:/

Hey, you added Villainrom 12 & EzHero 5, nice! Indeed two of the ROMs I was curious of, especially compared to Cronos 2.01 [my current installation, signature not yet updated].

Benchmark technically it looks like Cronos is a fair choice for now (assuming the results will be more or less equal to 1.71), but after EzHero will have ironed out the errors it's gonna be difficult to choose between, well, all of them :-)

So cheers again for your effort man
[Hero, G2 Touch Android Development] Arkymedes:
Added:

- Kimera 2.1

- Cyanogemod 5.0.8

- Villain 12.0

- EZHero V5
[Hero, G2 Touch Android Development] shokkapic:
I left you a PM, Arkymedes. :)
[Hero, G2 Touch Android Development] Arkymedes:
Added Official Hero 2.1 @ 528 MHz
[Hero, G2 Touch Android Development] zorxd:
Is there other benchmarks that do not focus on 3D and/or floating point (such as linpack)?

None of these benchmarks are representative of my usage. What about a benchmark of navigating in menus, starting apps, using google maps, etc?
[Hero, G2 Touch Android Development] shokkapic:
Hi zorxd.

Tomorrow, I'll be benchmarking the CPU through all the 2.1 ROMs.

If you could be more specific, I could try to implement that as part of my work.
[Hero, G2 Touch Android Development] Arkymedes:
Well, in my opinion... 2D benchmarks are pretty much "navigating in menus", CPU is pretty much "starting apps" etc...

Anyway, at least I don't know any thread that do benchmarks for this specific cases. You are free to start one, I wouldn't mind :)
[Hero, G2 Touch Android Development] shokkapic:
I agree with Arkymedes, though I don't mind to compare the overall fluidness between the ROMs.

2D stands for switching the menus and widgets, writing SMS, watching pictures, etc.

CPU stands for starting and running heavy applications.

3D stands for applications, and maybe some widgets.

Since as Arkymedes is already working with 2D/3D benchmarking, I will try to complete his job by benchmarking the CPUs.
[Hero, G2 Touch Android Development] -2y2k-:
1.1r2 please!
[Hero, G2 Touch Android Development] Arkymedes:
Hello all,

I just wanted to say bye since I no longer have an Hero and I'm now waiting for the Samsung Galaxy S to arrive.

Just wanted to thank all for the very pleasant stay in these forums and I just hope the SGS will one day have the amount of developers and support from the XDA members as the Hero owners have.

Well, that's all!

See you all on the SGS threads :)

PS: Also, for the moderators, feel free to change the ownership of this thread and maybe someone else can continue with the benchmarks :)
[Hero, G2 Touch Android Development] btdag:
How do you create the graphs?

Can you send me or upload the most recent data?

I don't have a job at the moment. I might be able to take over this project.

I might integrate the GPU benchmarks and CPU benchmarks into one topic if I do.
[Hero, G2 Touch Android Development] shokkapic:
Ohh... It's sad to see you go. :x

If I had the time right now, I would continue your work and mine.

But this 2 weeks will be difficult to gather the time.
[Hero, G2 Touch Android Development] btdag:
Added Chocolate Eclair 2.0

more to come over the next few days
[Hero, G2 Touch Android Development] dkelley:
holy moly.... after seeing it's performance, I'm thinking CE2 will be the next rom I flash. although I've grown to dislike sense, that's impressive performance, only topped (hopefully) by the froyo rom I'm running right now (I'd REALLY like to know how well it performs in these tests... will have to test myself I guess :-)

cheers,

Don
[Hero, G2 Touch Android Development] dkelley:
did some tests on my own froyo hero phone:

Froyd 1.0 with testkernel running at 691.

live wallpaper running (flying androids!!). no apps known to be running so this is probably a fairly typical result.

note: this is just on my phone with a million apps installed, but still ...

an3d: 2114 (BOO)

neocore sound on: 27.2 fps (BOOO)

neocore sound off: 31.4 fps (BOOOO)

benchmark (gpu only for this thread): 112 (really bad score! how disappointing!)

Crappy results all around. AND YET, I can play raging thunder 2 on this thing better than on any of the 2.1 roms. why?
[Hero, G2 Touch Android Development] Hacre:
Those results aren't that out of line with what you should expect from those benchmarks on a HTC Hero. JIT has very little to do with 3d rendering.

Games themselves see some improvements when dealing with repetitive code/code loops due to JIT but that's about it. There's been an update to Froyd btw. Be glad the scores are that high to be honest, considering how new the port of Froyo to the Hero is.
[Hero, G2 Touch Android Development] dkelley:
fair enough. there are things that are notably slower in froyo on the hero so far, simple things like the animations when you hold down the power button and choose power off or restart, the animations are uber-jerky compared to VE4.0.

I'm sure there are aspects of the rom that are going to be sped up over time. but games and apps run so much faster on this rom compared to jit enabled 2.1 roms that it's really quite shocking to me that the scores were so low.

Shows that what slows many things isn't the drawing code it's the gameplay code itself or, as you noted, things that loop identically. It's been a Loooong time since I learned what jit compilation speeds up (like when JIT was invented, I'm a 41 year old computer programmer LoL). It's slowly coming back to me bit by bit.

I'll do some other tests on it for the general performance speed thread to see how it stacks up.
[Hero, G2 Touch Android Development] btdag:
VanillaEclair 5.0 added

As a teaser for what's to come the graphs have been updated ;)

On a separate note - despite benchmark results I have found VanillaEclair to be much more responsive than ChocolateEclair - even when running LauncherPro on Chocolate. This has nothing to do with graphical performance though...
[Hero, G2 Touch Android Development] Hacre:
To make the tests more real world, you should randomly touch/swipe/longpress on the screen at least during Neocore to simulate actual game play.
[Hero, G2 Touch Android Development] btdag:
This would be unrepeatable.

The whole point of benchmarks is that they are comparable and repeatable. Without those factors benchmarks are pointless. If you want "real world" performance - try them out yourself.

Edit: FroydVillain initial results added - surprisingly uninspiring so far
[Hero, G2 Touch Android Development] Hacre:
How is holding your finger on the screen not repeatable?
[Hero, G2 Touch Android Development] Feeyo:
Why did u use 1.7.1 from Cronos against all those new ones? 1.7.1 is ooold and the worse release ever from Cronos :)
[Hero, G2 Touch Android Development] Hacre:
You had newer releases than that out before we had VR12 out didn't you?
[Hero, G2 Touch Android Development] btdag:
You're missing the point - these benchmarks are done under strict test conditions. These conditions are highly unlikely to be used on a phone at any point in time, therefore attempting to gain any form of "real" conditions will be invalidated due to the test conditions themselves. Therefore, what is the point? A pure test environment will allow for more consistent and comparable results. Contaminating it in any way will only serve to reduce reliability and fail to bring any extra validity.

I'm getting around to Cronos - these tests take a lot of time. I have only had access to the spreadsheets for just over a day :).
[Hero, G2 Touch Android Development] Hacre:
So basically, the benchmarks are worthless when it comes to actually using the phone. Good to know!

Before I get some sarcastic retort about only raising this issue because Villain isn't "winning" that has nothing to do with it. However anyone using your benchmark results to choose a ROM that's good for gaming (it is after all, a 3d performance test by and large) are going to be sorely disappointed if they use your "winning" AOSP ROMs due to the touchscreen driver being borked when handling input.
[Hero, G2 Touch Android Development] btdag:
They aren't worthless - they show core differences between the rom's. They shouldn't differ significantly to general use, except in normal conditions the results are lower and less consistent across all roms.

I personally actually quite like VillainROM and have continued to favor their releases due to their community support and themes etc - having said that their rom's don't appear to benchmark graphically very well (not necessarily meaning they perform worse in day to day apps). I am merely taking over the benchmarking role for those who like to pick their rom's based upon benchmark results.

I would love to know which ROM you think is currently winning? Each rom has it's strengths and weaknesses according to the benchmarks so far. I am still undecided about which are currently "winning".

Also please bear in mind that smooth gameplay is considered anything over 25 fps - which every custom rom is higher than on neocore. However, the results are an average frame rate so on every rom there is the possibility that the frame rate may drop below 25 on certain games.
[Hero, G2 Touch Android Development] dkelley:
I agree, valid point. however how does he work around that? holding a finger tests for each rom as well as one more color on those test results? That could be interesting... if not 100% perfectly consistent due to finger moving etc.
[Hero, G2 Touch Android Development] pulser_g2:
Just hold it in same place. That should be fine. It's basically to show that on most AOSP ROMs, their performance in that respect will be pretty low, making many screen-related games unplayable.

If you've got a small piece of metal that works as a capacitive 'finger' (I've got a bit here that does the trick for my testing), then you could just sit that on the screen.

But for any gaming purposes, the screen test is important IMHO, as some games become unplayable due to the low FPS (I'm talking about ~4 FPS in my experiences of some ROMs)
[Hero, G2 Touch Android Development] rakig:
There is also a discrepancy between what a benchmark tells you (core speed, single app, specific tests) vs what you'll experience using the rom (menu animations, touch lag, multitasking etc...)

These are showing the core performance, CPU optimizations, and in most cases they will apply only to specific situations.

All benchmarks are like that. Real world is different for each person, so benchmarks are really a 'best case, nothing else screwing things up/slowing stuff down' benchmark.

If its possible to specify an exact point/time to touch the screen, and somehow replicate that exactly across multiple roms then it might be useful, but I don't see any easy ways to standardize that so each rom can be considered reliable to cross-compare.
[Hero, G2 Touch Android Development] dkelley:
from what I understand, it's having the screen being touched that does it, not the location on the screen. there is a touch down trigger event ongoing and slowing things down.

just having a finger on screen for the duration will give a pretty good idea I would think. certainly as consistent as the test results will be without a finger on screen (which you will find vary a bit if you run the tests a few times).
[Hero, G2 Touch Android Development] btdag:
I understand what is being said. I am still not considering including it in the benchmarks for the following reasons:

1. Most have already been done and adding a new one now will be a massive hassle (reinstalling lots of roms which have been previously tested)

2. It is so far unknown whether this trigger event is consistent across rom's or inconsistent and how big a difference it makes

IF there is some research to show that there is a significant difference between rom's with this event then I will re-consider adding the extra benchmark.
[Hero, G2 Touch Android Development] dkelley:
1. of course LoL, that would be an insane project :-)

2. true, although with froyd claiming it is fixed in their rom I suppose it would be an interesting comparison between it and a different froyo rom on hero and with a non-froyo rom on hero. Maybe that's something someone else can put together at least as a thread post here for conversation purposes?
[Hero, G2 Touch Android Development] dkelley:
froyd screen touching benchmark test (unofficial):

Summary udpated: initial reaction was bad but in effect actually pretty good results.

So just for fun, I ran neocore, sound off, and went manic on my screen as though playing toonwarz.

without touching screen: 31fps

touching screen as though playing game (multitouch, repeated taps on one finger while dragging/holding other): 20fps

just holding one finger on screen whole time without moving: 25fps

Notes: of course I haven't tested other roms (I don't think this needs to be bolded since it doesn't change the results above... I've remove my negative judgement calls though since they weren't valid without comparison proof). This could be a huge improvement by comparison with other roms... and tested . without touching I got 30.8 and 31fps again, so pretty consistent non-touching results, so changes are definitely from screen touching.

anyone with a different rom care to try the same informal tests out of curiosity? at least for sake of discussion, even though they won't be very scientific results?
[Hero, G2 Touch Android Development] pulser_g2:
1) Fair enough. If I was you, I wouldn't do that either. It's too much hassle

2) Basically, any AOSP (ie. non Sense) ROM will have this issue, and will get about 50% performance drop when finger is on the screen, since HTC did something to their kernel, which means without sense, ROM performance sucks due to the touchscreen driver.

As I type, out of the non-sense ROMs, FroydVillain 1.1.0 and above is the only ROM to have this fixed.