How Many Android OEMs Cheat Benchmark Scores? Pretty Much All of Them 189
An anonymous reader writes "After Samsung got caught out cheating on benchmarks (Note 3, Galaxy S4) AnandTech has done a detailed analysis of the state of benchmark cheating amongst Android OEMs. With the exception of Motorola, literally every single OEM they've looked at ships (or has shipped) at least one device that does benchmark-specific CPU optimizations. AnandTech also thinks it will get worse before it gets better. 'The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors.' The article notes that Apple doesn't do any of the frequency gaming stuff."
And Apple (Score:5, Informative)
With the exception of Motorola...
And Apple. Apple and Motorola/Google are the only two companies that don't boost their devices for benchmark tests. If you're going to give credit to one, please do be fair and give credit to the other.
I respect both of them for that level of integrity and I hope they stick to their guns and remain honest.
I may be an Apple fanboy (and I am) but I'm really looking forward to seeing what Motorola starts releasing in about a year once Google's able to, as they said, flush things out of the system and start releasing truly Google-designed products.
CPU Benchmark Shenanigans ROI?? (Score:2, Informative)
'The hilarious part of all of this is we’re still talking about small gains in performance.'
The even more hilarious part is that OEMs are going to the trouble to do this when CPU benchmark scores are a very small factor in the decision of most consumers to buy these phones. I doubt that the ROI is higher than say, oh, improving the user experience of the GUI or call QoS.
Re:Not cheating (Score:2, Informative)
The performance quoted simply is not available to apps that are not on a whitelist of benchmark applications. It literally does not represent any part of the phone's non-benchmarking performance.
Re:And Apple (Score:4, Informative)
Re:Or, alternatively (Score:5, Informative)
Benchmarks are about as useful as manufacturer spec sheets. Take both with a a few metric tonnes of salt.
Re:Probably wont get better (Score:2, Informative)
The discrepancy is deliberate. For a long, long time drive capacity was quoted in the same units that the computer used for storage: binary SI prefixes, not decimal ones. The change to "1 megabyte = 1 million bytes" didn't set in until the 2000s.
Kudos to Apple for making their specs and their OS use consistent units, but it's still a marketing bullshit decision.
Wrong, they are boosting clock speed above normal (Score:3, Informative)
I was under the impression that what they were doing was ensuring that the clock speed was running at full, not slowed down for power saving, etc.
No. They are running at a clock speed that no real application will see under any circumstance, either the GPU or CPU cock increased.
It is within what the parts are rated for but not what the device was built to run at normally.
I assume some games and other applications also force the processor to full
There is no way to build a game on Android that can run at the speed the benchmarks are getting run at on each of the devices "cheating".
Re:Wrong, they are boosting clock speed above norm (Score:2, Informative)
No, he was right. The phones CAN and DO reach that clock speed. Read the AnandTech article. The graph of CPU speed shows it quite clearly.
Re:Or, alternatively (Score:5, Informative)
Actually, manufacturers do report their own efficiency numbers, and the EPA spot-checks them.
http://business.time.com/2012/12/10/more-reason-to-be-skeptical-about-new-car-mpg-claims/ [time.com]
Comment removed (Score:4, Informative)