Forgot your password?
typodupeerror
Android Cellphones Handhelds Software

How Many Android OEMs Cheat Benchmark Scores? Pretty Much All of Them 189

Posted by Soulskill
from the at-least-they're-consistent dept.
An anonymous reader writes "After Samsung got caught out cheating on benchmarks (Note 3, Galaxy S4) AnandTech has done a detailed analysis of the state of benchmark cheating amongst Android OEMs. With the exception of Motorola, literally every single OEM they've looked at ships (or has shipped) at least one device that does benchmark-specific CPU optimizations. AnandTech also thinks it will get worse before it gets better. 'The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors.' The article notes that Apple doesn't do any of the frequency gaming stuff."
This discussion has been archived. No new comments can be posted.

How Many Android OEMs Cheat Benchmark Scores? Pretty Much All of Them

Comments Filter:
  • Easy solution (Score:5, Interesting)

    by TheRealMindChild (743925) on Friday October 04, 2013 @09:38AM (#45034925) Homepage Journal
    The benchmark software should randomize the process name on launch
    • by h4rr4r (612664)

      So then you just look for behavior. Oh look a bunch of unnatural activity, change the governor to performance instead of on_demand.

      • by JanneM (7445)

        So, make the benchmark software resemble the composite behaviour of common classes of apps. OGL benchmarks effectively act as the "typical" 3D game and so on.

        At which point cheating becomes pointless, as any tweaking in favour of performance on those benchmarks immediately hit you as worse batterly life and high temps for all users running any similar kinds of apps, including your reviewers.

        • by Sockatume (732728)

          You want something that's consistent and reliable from device to device and run to run; that consistency dooms any attempt to cloak the behaviour of the app to failure.

      • Oh look a bunch of unnatural activity, change the governor to performance instead of on_demand.

        That shouldn't make that much of a difference... if the governor is in the low_power setting, limiting the CPU to minimum clock speed, then there'll be a performance hit on benchmarks, but with on_demand, by the time the system has realized it needs to change the governor to performance, the clock speed will have already been increased to maximum due to CPU load. Meanwhile, you're wasting clock cycles monitoring what stuff is doing....

        • by h4rr4r (612664)

          I was thinking to cheat battery test for example or have it lift the max CPU clock above what is normally used.

          • Fair enough, And not an unreasonable measure.

            Truthfully, and I know I'm in the minority in a discussion about benchmarks, cell phones have been "good enough" for quite a while, and I bought mine based on feature list and price. I bought the cheapest phone that had all of the features I wanted: FM radio, GPS, bluetooth, wifi tether, and that's it. Didn't even care whether it had 4G connectivity (though it did). Even a bottom-end $100 smart phone with an 800MHz single core processor is in the performance rang

            • by h4rr4r (612664)

              I totally agree. I want a better display, more color accurate and such more than faster CPU.

      • "So then you just look for behavior. Oh look a bunch of unnatural activity, change the governor to performance instead of on_demand."

        That's an excellent idea. Now all you have to do it look for a way to do it with almost no overhead.

      • by mwvdlee (775178)

        The goal is to make detection so complicated that the simple act of detecting a benchmark reduces performance to levels impossible to compensate by cheating.

      • by sjames (1099)

        Battery life measurements should take place with the benchmark app running.

    • Or, alternatively (Score:5, Insightful)

      by Truth_Quark (219407) on Friday October 04, 2013 @09:44AM (#45035001) Journal
      The phone manufacturers should not be dicks.
      • Re: (Score:3, Interesting)

        by Anonymous Coward

        On the contrary, the testers should just be bigger dicks. "We detected benchmark-specific optimizations in products #1, #2 and #3, so they all got zero points."

        • You think there's fanbois here, imagine how torn up they'd get if they did that.

        • by gnasher719 (869701) on Friday October 04, 2013 @11:14AM (#45036061)

          On the contrary, the testers should just be bigger dicks. "We detected benchmark-specific optimizations in products #1, #2 and #3, so they all got zero points."

          That seems quite arbitrary. What about "To test the battery, we tested how many minutes the battery lasts while running benchmark X". The cheaters will get shorter battery life.

        • by smash (1351)
          Pretty much.
      • by Nerdfest (867930)

        I'm a little confused about this. I was under the impression that what they were doing was ensuring that the clock speed was running at full, not slowed down for power saving, etc. From my point of view, that would just give a consistent reading of how fast the phone could run and wouldn't be considered 'cheating' unless they were boosting the clock speed up over normal running speeds. I assume some games and other applications also force the processor to full, but perhaps this is not the case. The whole t

        • I was under the impression that what they were doing was ensuring that the clock speed was running at full, not slowed down for power saving, etc.

          No. They are running at a clock speed that no real application will see under any circumstance, either the GPU or CPU cock increased.

          It is within what the parts are rated for but not what the device was built to run at normally.

          I assume some games and other applications also force the processor to full

          There is no way to build a game on Android that can run at the

          • by drerwk (695572)

            There is no way to build a game on Android that can run at the speed the benchmarks are getting run at on each of the devices "cheating".

            What happens if you name your game to match the benchmark that the phone is looking for?

          • by Nerdfest (867930)

            Thanks, as I said, I was under the impression they were just ensuring it was not reduced. If there's no way an app could get the same speed then it is most definitely cheating. I'm surprised they don't overclock. If you're going to cheat you might as well do it right.

          • Re: (Score:2, Informative)

            by Anonymous Coward

            No, he was right. The phones CAN and DO reach that clock speed. Read the AnandTech article. The graph of CPU speed shows it quite clearly.

          • by ifiwereasculptor (1870574) on Friday October 04, 2013 @12:03PM (#45036613)

            either the GPU or CPU cock increased.

            Whoa, can cell phones do that now? I hold these things to my ear, for Christ's sake!

            • by Dogtanian (588974)

              either the GPU or CPU cock increased.

              Whoa, can cell phones do that now? I hold these things to my ear, for Christ's sake!

              Yeah, they'll f**k your brains out... literally.

        • It makes comparing two phones harder because you won't know which of them is "cheating."
      • Re:Or, alternatively (Score:5, Informative)

        by L4t3r4lu5 (1216702) on Friday October 04, 2013 @10:33AM (#45035535)
        Pfff. Car manufacturers tape up the air intakes and door seams on their cars to do fuel economy runs, just to eek out the every last 0.1mpg. Running your car like that for any reasonable period of time would wreck the engine pretty quick.

        Benchmarks are about as useful as manufacturer spec sheets. Take both with a a few metric tonnes of salt.
    • you're assuming the benchmark authors aren't working directly with the hardware manufacturers.

    • Re:Easy solution (Score:5, Insightful)

      by hairyfeet (841228) <bassbeast1968@gma i l . com> on Friday October 04, 2013 @10:19AM (#45035335) Journal

      Even easier...don't use benchmarks which are easily rigged, choose real world applications and cook up some tests using them!

      The problem is ALL the benchmarks have been rigged by somebody for ages...X86? Any benchmark compiled using ICC is rigged against AMD and boosts Intel by 30% (look up "Intel Cripple Compiler" if you want to learn more, end result is benchmarks are completely useless on X86 unless compiled with GCC) and as we see here ARM? Rigged by everybody. This kind of crap has been going on since "Quack.exe" and is why you shouldn't trust ANY benchmarks. I'll always remember one of the reviews before the guy was told about ICC when he was testing netbooks "These new netbooks with AMD seem to be MUCH snappier...yet it loses to the low end Atom by nearly 30%, and I just don't know why"...the reason why is the benchmarks are all rigged!

      So if you want to bench an ARM chip? Root the phone, install Bash, and run a script that runs real applications in real world scenarios. there are more than enough apps out there that cooking up their own informal benches should NOT be hard and until they do? Just remember to replace the word benchmark with bullshit every time you see one of these articles.

      • by Sockatume (732728)

        The idea that people should have to come up with less-reliable, improvised tests because the hardware manufacturers are going to crawl over each other to cheat any consistent, scientific one is kind of depressing. You kind of know that all these phone companies are first-rate bullshit artists, with Samsung running at the fore, but the idea even that their engineers are going out of their way to fake their product to the top of the specs pile is just sad.

        • by hairyfeet (841228)

          Yes but until these companies stop cheating, just as Intel to this very day cheats with ICC then what choice do you have? If every scale is being rigged by the manufacturer then the only choice is to not use their scale, correct?

          And in both cases since you have access to a CLI writing a script that calls real world applications shouldn't be very hard. as long as all the corps are rigging there really isn't a choice.

        • OIn the contrary, they should come up with more meaningful tests. On the x86 side, there are quite a few staples of testing that are also real-world scenarios, like RAR decompression, encryption and video and mp3 encoding. For GPUs, an assorted bundle of real games are great for measuring performance.

          • by hairyfeet (841228)

            Exactly and all of those can be trivially automated with scripts so they can be just as accurate as any benchmark WITHOUT allowing the companies to rig the test!

            At the end of the day if everybody has rigged the test then the test is worthless, that's it, no more argument on the subject, the test is worthless and meaningless if everybody cheats and as we have seen both on X86 and on ARM cheating is rampant, so what to do? There is only one thing you CAN do and that is to throw out the test and use real app

      • Instead of automated benchmarks of hardware, why not real world human benchmarking where a group of people is given a set of tasks to do on a given cell phone platform and see who can do them faster?

        Automated technical benchmarks make sense when the workloads more or less approximate the benchmark -- video gaming, 3D modeling, disk throughput, etc.

        But unless I'm living totally in the dark, most people aren't buying cell phones oriented towards single-task performance (eg, gaming). They get used for many ta

    • by erroneus (253617)

      Better solution. No benchmark test should be accepted unless the software was written AFTER the release of the hardware being tested. And then, yes, randomize various things including the compiled code.

    • by mwvdlee (775178)

      Alternatively, can we get an app that changes the name of any game to one of the benchmarks' names?

    • by tlhIngan (30335)

      The benchmark software should randomize the process name on launch

      Can you even do that on Android? Generally to rename the package on Android you need to take the APK apart and rename the DEX files there.

      Right now, it looks like the most impressive tests are ones done with the browser purely because to cheat three means generally it works for everyone.

      (Though even then I can see why they'd cheat - an iPhone 5s dual core processor can keep up with quad core SoCs running nearly twice as fast (1.3GHz A7 vs. 2.

  • And Apple (Score:5, Informative)

    by Anonymous Coward on Friday October 04, 2013 @09:50AM (#45035085)

    With the exception of Motorola...

    And Apple. Apple and Motorola/Google are the only two companies that don't boost their devices for benchmark tests. If you're going to give credit to one, please do be fair and give credit to the other.

    I respect both of them for that level of integrity and I hope they stick to their guns and remain honest.

    I may be an Apple fanboy (and I am) but I'm really looking forward to seeing what Motorola starts releasing in about a year once Google's able to, as they said, flush things out of the system and start releasing truly Google-designed products.

    • Re: (Score:3, Interesting)

      by Anonymous Coward
      Posting anonymously since I'm a motoroogle employee... you'll be disappointed. I certainly am. At this point, I expect google to shut us down or spin us off.
      • Re:And Apple (Score:4, Interesting)

        by JanneM (7445) on Friday October 04, 2013 @10:20AM (#45035367) Homepage

        I've heard a number of people I trust comment on how the Moto X just feels really good in the hand and how the screen and size is just right. So the phone is certainly not bad at all, and most of us don't actually buy the high-end phones any more than we buy the high-end cars or bikes. A really good mid-range phone is exactly what I want; I'm already making calf-eyes toward the coming Sony Z1 Mini.

        What's hurting Motorola for me, personally, is that it's simply not on sale in most of the world, and seems unlikely to ever be. It's not just about being able to get it where I live, but having a phone designed from the start to be usable in all major regions of the world as I travel.

        • Re:And Apple (Score:4, Interesting)

          by Trillian_1138 (221423) <slashdot AT fridaythang DOT com> on Friday October 04, 2013 @01:52PM (#45037669)
          Just picked up a Moto X to replace my aging Galaxy Nexus. I'm on Verizon, which makes me skeptical I'll see another Nexus phone anytime soon. I like the near-stock Moto X setup, and the little tweaks (active notifications, touchless control, etc) are pretty fun. It's a pretty great phone, and I hope it helps boost Motorola in the Android world. I do think they priced it too high (although I still bought it, so take that as you will..) but I highly recommend the phone to anyone who plays the occasional, not to graphic-intensive games, wants their phone to be quick and responsive, likes stock Android, etc. It's an awesome phone.
      • by gaiageek (1070870)

        Posting anonymously since I'm a motoroogle employee... you'll be disappointed. I certainly am. At this point, I expect google to shut us down or spin us off.

        Care to elaborate on why we'll be disappointed? I ask because this thread is focused on (perceived) performance of smartphones, and I know Motorola caught some criticism for not giving the Moto X better silicon to compete with other flagships, but personally, I think their tactic is pretty smart: focusing on functionality for the average user instead of performance capabilities which most people don't care about. I'd actually consider buying a Moto X if it weren't for the non-removable battery (deal breake

    • With the exception of Motorola...

      And Apple. [...] If you're going to give credit to one, please do be fair and give credit to the other.

      They did, and it's in the summary. You didn't even have to read TFA. Additionally, the headline narrows this down to Android OEMs, so that's why Apple was excluded from the discussion until the very end.

      The article notes that Apple doesn't do any of the frequency gaming stuff.

    • Re: (Score:3, Insightful)

      by h4rr4r (612664)

      I fear Motorola is dead.

      The Moto X was supposed to be a device with high amounts of post Google input. It is a high priced mediocre mid range device. They still cater to carriers unlike Apple. They need to compete with the 5S, One X and S4. Instead they are competing with last years product.

      Moto needs to copy Apple in some things. Make 1 main device, sell the old one or a cheaper version as well. Max 2-3 devices. Refresh them once a year. This lets third parties make all kinds of doodads for the device. Put

      • Re:And Apple (Score:5, Interesting)

        by Dixie_Flatline (5077) <vincent@jan@goh.gmail@com> on Friday October 04, 2013 @10:07AM (#45035247) Homepage

        It's not a mid-range device. It's only mid-range if you look at the spec sheet and nothing else. Its (non-gamed) benchmarks are actually pretty good for all this talk of 'mid-range'. They did the same thing Apple did and tried to balance out performance with battery life. They didn't put the biggest screen in it, and they have optimised silicon to listen for commands without keeping the CPU on all the time.

        Specs aren't the war that anyone should be trying to win in the mobile space. That kind of thinking is why there are phones that only last half the day.

        • by h4rr4r (612664)

          It is a mid range device if you hold one. Go get one and tell me it is not a midrange device.

          I am not suggesting chasing specs, I am suggesting building a premium device.

          • It can't be worse than the Samsung's I've handled. And the HTC One is premium and beautiful, but that doesn't seem to be working out.

            I will accept that you personally are calling it a mid-range device on the build quality alone, but so much of the commentary has focussed on how it has 'mid-range specs' and is overpriced for the screen/CPU/GPU that's in there.

            There are trade-offs to be had. iPhones are top-tier devices, but they've got smaller screens than the top-tier Android phones. The S4 is a plasticky t

            • by h4rr4r (612664)

              The One is a really nice device. If they would kill SenseUI already it would be the best Android device on the market.

              Not only build quality but some specs matter, think less about things like resolution and more about how accurate colors are or the white balance on the device is. Pentile vs RGB IPS is a good example of this sort of thing.

              The S4 might have a lot of plastic, but it makes it rather durable. They should have had the S4 Active as the only model. Instead they made that AT&T exclusive. It is

    • Good point, the article should have congratulated all the companies that were not caught cheating on android performance tests:

      Microsoft,
      Walmart,
      Berkshire-Hathaway,
      Lockheed Martin,
      that "self employed" beggar across the street,
      Burger King,
      Ford Motor Company, ...

      That could take awhile actually, lets just congratulate the company you're a fan-boy of then shall we?

  • by Anonymous Coward

    'The hilarious part of all of this is we’re still talking about small gains in performance.'

    The even more hilarious part is that OEMs are going to the trouble to do this when CPU benchmark scores are a very small factor in the decision of most consumers to buy these phones. I doubt that the ROI is higher than say, oh, improving the user experience of the GUI or call QoS.

  • Hilarious... (Score:4, Insightful)

    by Savage-Rabbit (308260) on Friday October 04, 2013 @10:05AM (#45035229)

    The hilarious part of all of this is we’re still talking about small gains in performance.

    The hilarious part of all this is that most people really don't give a rat's as about performance when selecting a phone or even a tablet. The criteria are things like: how does it handle? How intuitive is the UI? Can I watch my favorite online video feeds on this thing? Are any buttons in annoying palaces? What's the price? Does this thing have software to view and edit MS Office files I get sent by mail? The only performance tests these smartphone and tablet things usually get is playing around with a display example in the shop and seeing if the UI is nice and snappy. Nobody excepts tech nerds gives a rats ass that a Samsun Galaxy 4 get a few more FPS in Modern Combat than an iPhone 5.

    • by Sockatume (732728)

      Nobody cares about the benchmarks quantitatively, but when you're putting down something like $700 on a computing device getting something that's "the fastest" is a big driver of decisions. I'm sure that guaranteeing a bunch of "Samsung Galaxy blah takes performance crown from Apple iPhone whatevs" stories in the tech press on the basis of a 0.5% difference in the Zootybench score is exactly why Samsung does this crap.

      • Naw, I would argue that having the fastest phone was only a deciding factor when Apple started boasting about their new 64bit CPU in the iPhone 5s. Prior to this month the average consumer would not even know or care if their phone had 1, 2 or 4 cores or how many goggleflops it was able to perform.

        Apple decided the only way they can differentiate their iPhone 5s amidst all the comments that they are no longer innovative is to create a competitive market based on useless CPU performance numbers, just like w

        • Re:Hilarious... (Score:4, Insightful)

          by Savage-Rabbit (308260) on Friday October 04, 2013 @11:32AM (#45036271)

          Apple decided the only way they can differentiate their iPhone 5s amidst all the comments that they are no longer innovative is to create a competitive market based on useless CPU performance numbers, just like what Apple did with Retina displays. Before Retina, nobody cared about pixel density. Before A7, nobody cared about CPU performance or its bittyness. Before the iPhone 5s camera, nobody cared about the size of the CCD pixel on their phone camera.

          Dude, It's not that long ago that articles like this slashvertisment [slashdot.org] were plastered all over the web accompanied by comments filled with enthusiastic boasts by hoards of Android fans detailing how iPhone performance sucks ass. Even if blisteringly fast benchmark performance is pretty low down on the list of most people out to buy a smartphone I still can't fault Apple for trying to put a sock in the collective mouth of the Android community. There is a certain personal satisfaction to be had from making the choir of hard core Android fans shut up about benchmarks until Samsung comes up with a still faster device (hopefully free of benchmark cheating this time) even if the customers will probably hardly notice this stupid pissing contest.

    • I might not care as much about performance as I care about battery life, and a rating of that is performance/Watt.

      • I might not care as much about performance as I care about battery life, and a rating of that is performance/Watt.

        Ditto, that's probably the only part of a benchmark test that I really care about.

    • by tompaulco (629533)

      The hilarious part of all this is that most people really don't give a rat's as about performance when selecting a phone or even a tablet. The criteria are things like: how does it handle? How intuitive is the UI? Can I watch my favorite online video feeds on this thing? Are any buttons in annoying palaces? What's the price? Does this thing have software to view and edit MS Office files I get sent by mail? The only performance tests these smartphone and tablet things usually get is playing around with a display example in the shop and seeing if the UI is nice and snappy. Nobody excepts tech nerds gives a rats ass that a Samsun Galaxy 4 get a few more FPS in Modern Combat than an iPhone 5.

      Don't forget "Does it start with an 'i'". For some people that is the strongest factor. And it could be a plus or a minus factor.

    • Here are some tests I've done on my devices compiling Povray [povray.org] 3.6 (single threaded) and rendering the benchmark scene. I was rather surprised that my new Nexus 4 (Qualcomm S4) is slower per core than my SGS II (Samsung Exynos 4). Yet the Nexus 4 doesn't feel any slower and is my main phone now due to the larger and higher resolution screen.


      Athlon II x4 (2.8 GHz): 179.82 pps ; 64.22 pps/GHz
      Exynos 5 (1.7 GHz): 77.36 pps ; 45.51 pps/GHz (-mfloat-abi=hard -mcpu=cortex-a9 -mthumb -mthumb-inter

  • by aitikin (909209) on Friday October 04, 2013 @10:19AM (#45035345)
    Is that the table is titled "I Can't Believe I Have to Make This Table". Made me smile.
  • by tompaulco (629533) on Friday October 04, 2013 @10:25AM (#45035425) Homepage Journal
    If you measure it, it gets better. Nobody ever stops to think why. They mostly just think they are good managers. But in fact, if you create a measuring tool to measure qualities of a device, the manufacturers will work to make that measurement better. If you make a measurement to determine how your employees are performing, they will perform better according to that measurement. That's just the way it works.
    You can't measure everything, so you're best bet is to try to keep the measurement methods secret and change them frequently. Unless, of course, your measurements are intended to improve a particular area, then by all means, measure on.
  • This has been going on since benchmarks were a thing. So a long time.

    GPU in particular had a bad rep for this. However they actually got the benchmark software altered!

    Simply checking for a process name and OC, isn't exactly all that sneaky. Maybe a bit unethical, but still. Whoever is running the benchmark could easily check for that. Of course it depends on how closed down the OS is for inspection also.

  • Native Code Execution, it runs as fast as it is going to run.

Life is difficult because it is non-linear.

Working...