Samsung Fudging Benchmarks Again On Galaxy Note 3 258
tlhIngan writes "A few months ago, Samsung was caught gaming benchmarks on the Galaxy S4 (International version). They would lock the GPU at a higher-than-normal frequency when certain applications were run, including many popular Android benchmarking programs. These had the expected result of boosting the performance numbers. This time, the Galaxy Note 3 was caught doing the same thing, boosting CPU scores by 20% over the otherwise identical LG G2 (which uses the same SoC at the same clock). Samsung defends these claims by saying the other apps make use of such functionality, but Ars reversed-engineered the relevant code and discovered it applied only to benchmark applications. Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost."
Re:If this was Apple... (Score:5, Informative)
Citation required, because all I can find is: http://www.theregister.co.uk/2003/06/25/apple_denies_fiddling_g5_xeon/ [theregister.co.uk] ... which seems to be refuting the claim...
Simon.
Re:If this was Apple... (Score:5, Informative)
Re:Does not computer (Score:4, Informative)
There is a file containing a list of all the common benchmarking apps, and everything in the list is a benchmarking app - nothing else. When one of those packages is run, the phone locks the frequency of all cores to fMax and also seems to fiddle with the GPU.
The result is a battery-nightmare, but a boost of 20% to *only* benchmark apps. This is despicable - plain and simple.
See http://arstechnica.com/gadgets/2013/10/galaxy-note-3s-benchmarking-adjustments-inflate-scores-by-up-to-20/ [arstechnica.com]
Simon.
Re:If this was Apple... (Score:5, Informative)
Not so much Samsung in the Nexus dept.
Nexus One was by HTC
Nexus 4 was by LG
Nexus 7 is by Asus
Nexus 10 is by Asus
Anyway, it doesn't matter to me who makes the phone, I look at the features, OS and apps. Samsung has done a good job of marketing the Galaxy series. Some people buy because of good marketing. I still hate it when the manufacturer or the telecoms giant mess with the interface and applications... it's usually not an improvement.
Re:Does not computer (Score:5, Informative)
Here is what the article actually says:
The ironic thing is that even with the benchmark booster disabled, the Note 3 still comes out faster than the G2 in this test. If the intent behind the boosting was simply to ensure that the Note 3 came out ahead in the benchmark race, it doesn't appear to have been necessary in the first place.
Apparently the "damning" part was completely fabricated by the submitter.
Re:Huh? (Score:2, Informative)
I was perplexed too. After reading the article, I figured it out:
It's damning in the sense that it's meant to skew comparisons with other Android devices, not to make it look better than their own previous offerings.
Re:If this was Apple... (Score:0, Informative)
Nvidia, AMD, and Intel have all been caught doing it. I'm seriously done caring. I'll look to REAL WORLD benchmarks instead of synthetic ones.
http://www.geek.com/games/futuremark-confirms-nvidia-is-cheating-in-benchmark-553361/
http://tech.slashdot.org/story/09/10/12/2341240/intel-caught-cheating-in-3dmark-benchmark
http://community.futuremark.com/forum/showthread.php?141723-AMD-cheating-by-turning-Tessellation-off
If you think that adopting a 64 bit processor is "copying", then you are completely deluded, and there's little point in discussing it with you. Suffice to say that droids have reached their 32-bit memory cap, and there's only one logical move to make.
As for best hardware, I don't think Apple really competes in this regard:
http://www.gizmag.com/galaxy-s4-vs-iphone-5s/29030/
But neither do other droids.
http://www.ign.com/articles/2013/08/02/motorola-samsung-htc-and-sony-comparing-android-flagships
Re:If this was Apple... (Score:4, Informative)
Re:Failed to note (Score:4, Informative)
Here is your citation. (Score:5, Informative)
Ok, I remember reading the Apple benchmarks myself (in utter disbelief - even for Apple it seemed too much), and this article you linked to does not agree with my memory. So let's go directly to the source. Read that benchmark paper yourself on archive.org : http://web.archive.org/web/20030727103031/http://veritest.com/clients/reports/apple/apple_performance.pdf [archive.org]
I gave it a quick look to refresh my memory and here are some highlights:
- They DISABLE hyper-threading on the SPEC rate test, which is the multi-processor test. Then, they ENABLE hyper-threading on the SPEC base, which is the single-processor test!!! They defend this by saying something like "hyper-threading is slower some times". Well, they sure know that, since they only enable it when it will slow down the Pentium! I would have given them the benefit of doubt if they had disabled (or enabled) it for both tests, but selectively enabling/disabling it means you know what you are doing.
- They use -O3 -fast -ffast when compiling for Apple, which uses fast math non-IEEE optimizations. Of course they had the Intel CPU run accurate/IEEE spec code - there is no equivalent -ffast-math used.
- They go on making some other "crazy" optimizations on the G5 like "modify CPU registers to enable memory Read By-pass", or installing a special malloc library that optimizes for speed by sacrificing memory just for the single-threaded benchmark. This is not how you benchmark for comparison purposes, especially if your optimizations for the competing platform are "turning off update" and "turning off hard drive sleep" (they obviously put that stuff just to pretend they "optimized" there as well).
And I am sure there are other things as well, this was from a quick read. And of course let's not mention that they compare the G5 with an Intel P4 CPU, when, at the time, AMD's Athlons/Opterons (64bit versions were just out as well) were destroying Intel (in performance, not sales - but that is another story).
In general, that paper is so ridiculous that I can't believe Apple had kept promoting it after they had been outed. But then again, given Apple's target audience, the explanation is simple. What was even more ridiculous is that when Apple started selling the Intel-based Mac they had kept for a while the section of their website that showed how much faster the G5 Mac was compared to Intel and then on the Intel Mac pages they had comparisons which showed how the Intel Mac is faster than the G5 Mac. No shame!