Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Android Cellphones Handhelds Stats

Samsung Fudging Benchmarks Again On Galaxy Note 3 258

Posted by timothy from the selective-activation-sounds-nicer dept.
tlhIngan writes "A few months ago, Samsung was caught gaming benchmarks on the Galaxy S4 (International version). They would lock the GPU at a higher-than-normal frequency when certain applications were run, including many popular Android benchmarking programs. These had the expected result of boosting the performance numbers. This time, the Galaxy Note 3 was caught doing the same thing, boosting CPU scores by 20% over the otherwise identical LG G2 (which uses the same SoC at the same clock). Samsung defends these claims by saying the other apps make use of such functionality, but Ars reversed-engineered the relevant code and discovered it applied only to benchmark applications. Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost."
This discussion has been archived. No new comments can be posted.

Samsung Fudging Benchmarks Again On Galaxy Note 3 More

Samsung Fudging Benchmarks Again On Galaxy Note 3

Comments Filter:
  • If Apple did this, people would be up in arms!

    • Re:If this was Apple... (Score:5, Insightful)

      by Bill_the_Engineer ( 772575 ) on Tuesday October 01, 2013 @12:34PM (#45004677)
      Are you implying that us android users wouldn't be "up in arms"? It affects us more. While apple sells two models of the iPhone, we have a multitude of android phones to choose from. Samsung messing with the benchmark has the potential to cause a customer to chose samsung over HTC, Google Nexus, or Motorola phone.

      • Re: (Score:3, Insightful)

        by Anonymous Coward

        Why would you even consider synthetic benchmarks when choosing a phone? Planning on trying some bitcoin mining on the go?

      • Last I check the Nexus was a rebranded, unlocked, Samsung phone.

        Futhermore, most of the common people, when they now say an "Android Phone", what they mean is a Samsung phone. That's the product they now associate with being the "Android Phone/Tablet". You'd be surprised how many don't get that LG, HTC, and others have the same "android" OS.

        In fact I think it will be interesting to see what happens in a few years and if a couple players in the Android phone market drop out, after all I don't think the oth

        • Re:If this was Apple... (Score:5, Informative)

          by mspohr ( 589790 ) on Tuesday October 01, 2013 @01:01PM (#45005031)

          Not so much Samsung in the Nexus dept.
          Nexus One was by HTC
          Nexus 4 was by LG
          Nexus 7 is by Asus
          Nexus 10 is by Asus
          Anyway, it doesn't matter to me who makes the phone, I look at the features, OS and apps. Samsung has done a good job of marketing the Galaxy series. Some people buy because of good marketing. I still hate it when the manufacturer or the telecoms giant mess with the interface and applications... it's usually not an improvement.

        • Re: (Score:2)

          by h4rr4r ( 612664 )

          The Nexus 4 and up coming Nexus 5 are LG devices.

          The Galaxy Nexus was the last Samsung Nexus device and it is already over a year out of date.

          Samsung could try that, but then they lose access to the google play market and that ends their game right there.

        • Last I check the Nexus was a rebranded, unlocked, Samsung phone.

          Check again, the current Nexus 4 is manufactured by LG Electronics. The first Nexus was made by HTC. The Nexus S and Galaxy Nexus were made by Samsung.

          Futhermore, most of the common people, when they now say an "Android Phone", what they mean is a Samsung phone. That's the product they now associate with being the "Android Phone/Tablet". You'd be surprised how many don't get that LG, HTC, and others have the same "android" OS.

          Citation? Nexus

      • No Implication (Score:5, Insightful)

        by SuperKendall ( 25149 ) on Tuesday October 01, 2013 @01:15PM (#45005227)

        Are you implying that us android users wouldn't be "up in arms"?

        No implication is needed, we can see quite plainly there is very little outcry over this, just as there wasn't before. Android users simply accept this is the way things are, in a way they do not with any Apple problem whatsoever.

        • Re: (Score:2)

          by eagee ( 1308589 )
          I disagree, I don't buy Samsung products for this very reason, and I haven't bought an Apple product since our iPad 1 got upgraded to a version that barely works (with no option to downgrade - bastards). I don't have the time to protest crap companies, I just don't buy their crap.

        • Apple is held to a higher standard among consumers and industry for their behavior than Android makers are. This is just fact. Good news is that for the most part, they meet that standard.

      • Anand is reporting that many Android handset makers are cheating, not just Samsung, so HTC and/or Motorola may be cheating as well. They do mention that Nexus line does not cheat.

    • Re: (Score:2)

      by mveloso ( 325617 )

      You'd think they'd stop doing this, since they keep getting caught. However, crime pays. They've made substantially more money off of their phones than the puny fine they had to pay in the Samsung vs Apple trial. They doing it again to Dyson vacuums.

      The market is rewarding their bad behavior, and they're going to just keep going.

      Samsung's integrity is closely tied to the size of their profits. Once the lack of integrity starts impacting their bottom line, they'll find some. Until then, it's business as usua

    • Re: (Score:2, Funny)

      by Anonymous Coward

      If Apple did this, people would be up in arms!

      Both Apple and Android use ARM CPUs.

      Now, if it had been happening on the Surface tablets (non-RT, of course) people would be up in Atoms.

    • Re:If this was Apple... (Score:5, Funny)

      by DickBreath ( 207180 ) on Tuesday October 01, 2013 @12:48PM (#45004875) Homepage
      > If Apple did this, people would be up in arms!

      If Apple did this, they would be suing Samsung for copying.

    • Re: (Score:2)

      by jpvlsmv ( 583001 )

      If Apple did this, people would be up in arms!

      Both Apple and Samsung use ARM CPUs.

      Now, if this had been a Surface (non-RT) tablet, people would be up in Atoms!

    • If Apple did this, people would be up in arms!

      There's a long history of apple making exaggerated claims of what their devices are going to do for you. People have been up in arms for a long time.

    • Re: (Score:2)

      by larwe ( 858929 )
      But it's a totally different situation with Apple - because there is nobody else who makes iOS devices. So you'll never get a situation where a fake benchmark will cause someone to choose "HTC iOS phone X" instead of "Apple iOS phone X". And comparisons between different OSes are so apples to oranges (no pun intended) that I would heartily distrust them anyway.

    • If apple *got caught* people would be up in arms.

    • You know why that is? If I've invested in the Android platform, purchased a bunch of apps that I don't want to lose, and just generally don't feel like switching to a new platform, I can buy from any number of manufacturers who aren't Samsung when Samsung pulls shit like this. If I've invested in iOS, purchased a bunch of apps that I don't want to lose, and just generally don't feel like switching to a new platform, I can buy from... Apple... when Apple pulls shit like this.

      By insisting on being the only

  • Should have done a battery benchmark (Score:5, Interesting)

    by jandrese ( 485 ) <kensama@vt.edu> on Tuesday October 01, 2013 @12:39PM (#45004747) Homepage Journal
    They should have opened up a benchmarking app on both phones (the G2 and the Note3) and then did a battery life test on both phones with them "idle".

    • Re: (Score:3, Insightful)

      by jovius ( 974690 )

      The benchmarking itself seems to be flawed. Samsung wants to benchmark the devices at their full capacity, to see what they are capable of (the higher setting is reached in normal use of some apps anyway). The testers would probably like to do real world comparison tests (and not rely just on numbers). I don't see Samsung doing anything wrong here, even though the benchmarking apps are specifically chosen.

      • Re:Should have done a battery benchmark (Score:4, Insightful)

        by jandrese ( 485 ) <kensama@vt.edu> on Tuesday October 01, 2013 @02:10PM (#45005875) Homepage Journal
        They're changing the operating profile of their phone to inflate the benchmark results. In real life you would never be able to achieve numbers like that on a phone with a halfway decent battery life, even on applications that behave similarly to the benchmark. No normal use would achieve numbers found in the benchmark, they can't because they involve running the phone super hot and burning through the battery in no time, even when idle.

        If the hack were to go to full power when plugged into the wall, then I could maybe see a case for this being legitimate, since it would mean you could theoretically achieve the same results by simply playing your game or whatnot while plugged in, but because they're only switching on full power mode for a handful of specific benchmark applications there is just no excuse.

        • Re: (Score:2)

          by jovius ( 974690 )

          I'm not sure if they actually inflate anything, because they simply set the device to the highest setting (and it's not higher than where the device sometimes operate even normally).

          If you read that Ars Technica article it's evident that the max setting is reached in typical camera use for example. It's not limited to benchmarking apps. For benchmarking apps it's always on. This would be consistent with the idea of testing the device at max capacity. How about if the OS itself is hacked to give 533 MHz cons

      • Re: (Score:2)

        by icebike ( 68054 )

        The benchmarking itself seems to be flawed. Samsung wants to benchmark the devices at their full capacity, to see what they are capable of (the higher setting is reached in normal use of some apps anyway). The testers would probably like to do real world comparison tests (and not rely just on numbers). I don't see Samsung doing anything wrong here, even though the benchmarking apps are specifically chosen.

        You have a point, that a device should be tested to its highest capacity.

        However, if EVERY other application is PREVENTED from achieving that highest capacity the tests are already biased.

        And in truth, that is not always a BAD thing. After all, locking the other cores to full power for a test, but preventing that for day to day usage is simply a battery saving feature. Having multiple cores is useful so that other work can be performed at the same time as your phone checks mail or some such.

        But on the oth

    • Re: (Score:2)

      by Xyde ( 415798 )

      4 cores at 2.3Ghz? It might catch on fire. This is using the big.LITTLE chip right? I don't think they're designed with a thermal envelope to stay fully clocked and on for that long.

  • Comment removed (Score:5, Interesting)

    by account_deleted ( 4530225 ) on Tuesday October 01, 2013 @12:41PM (#45004773)
    Comment removed based on user account deletion

    • Re: (Score:2)

      by aitikin ( 909209 )
      I'm with you. The only way that makes sense to me is if the code is written such that it works by reading what exactly the app is trying to do and then responding due to that, but even then, it would stand to reason that it'd do that for any intensive app that does gaming or something like that, thus backing up Samsung's claim...

    • Re: (Score:2)

      by jandrese ( 485 )
      The Note was a couple of percent faster on renamed benchmark apps, and a whopping 20% faster on normally named benchmark apps. The point is that they were already faster so cheating wasn't even necessary.

      • Re: (Score:2)

        by icebike ( 68054 )

        The Note was a couple of percent faster on renamed benchmark apps, and a whopping 20% faster on normally named benchmark apps. The point is that they were already faster so cheating wasn't even necessary.

        Not really. A couple of percent faster is not humanly detectable. 20% is.

        Any programmer worth his pay can tune code to achieve 2%.

    • Re: (Score:2)

      by Cinder6 ( 894572 )

      I wouldn't call it damning, either, but it doesn't prove that the optimizations aren't app specific. The phone might still bench faster with non-renamed tools than with renamed ones (indeed, that's what the summary seems to claim). I would argue, however, that if the Note 3 beats the G2 "naturally", then there wasn't really any need to cheat this way.

      Regardless, I'm with others: it's misleading at best, false advertising at worst.

    • Re:Does not computer (Score:4, Informative)

      by Space cowboy ( 13680 ) on Tuesday October 01, 2013 @12:59PM (#45005005) Journal

      There is a file containing a list of all the common benchmarking apps, and everything in the list is a benchmarking app - nothing else. When one of those packages is run, the phone locks the frequency of all cores to fMax and also seems to fiddle with the GPU.

      The result is a battery-nightmare, but a boost of 20% to *only* benchmark apps. This is despicable - plain and simple.

      See http://arstechnica.com/gadgets/2013/10/galaxy-note-3s-benchmarking-adjustments-inflate-scores-by-up-to-20/ [arstechnica.com]

      Simon.

      • Re: (Score:3)

        by icebike ( 68054 )

        Why did you feed us a link to the exact same Article as the Summary does?

        Did you think that would trick us into reading the full article?
        This is Slashdot. We are wise to those ploys.

    • Re:Does not computer (Score:5, Informative)

      by Bill Dimm ( 463823 ) on Tuesday October 01, 2013 @01:01PM (#45005033) Homepage

      Here is what the article actually says:

      The ironic thing is that even with the benchmark booster disabled, the Note 3 still comes out faster than the G2 in this test. If the intent behind the boosting was simply to ensure that the Note 3 came out ahead in the benchmark race, it doesn't appear to have been necessary in the first place.

      Apparently the "damning" part was completely fabricated by the submitter.

      • Looks like everybody is playing games to boost scores or readers. I wonder if that works for me , out damn spot.

        Here is what the damning article actually says:

        Apparently the "damning" part was completely fabricated by the submitter.

        This kind of stuff is done by everybody. I was forced to go to COMDEX once and marketing wanted something to draw people at the show and so somebody :) created a " matching tweak" because they looked in the competitors code and saw a cheat. I have seen some really good bench mark cheats and what I saw most at COMDEX was engineers upset because somebody else's cheat was so flagrant and over the

      • Looks like the submitter confused irony with "damning". Not exactly great journalistic writing in the summaries provided by submitters lately. Meh, that's what we get for paying them absolutely nothing I guess.

      • Re: (Score:2)

        by Minwee ( 522556 )
        Ironically, many people aren't clear on what "ironic" means.

    • Re: (Score:3, Insightful)

      by prisoner-of-enigma ( 535770 )

      Not sure how this is "damning". I'd have thought it would prove the principle that the optimizations aren't app specific.

      What am I missing?

      It's not app-specific, it's app *name* specific. It's analogous to the Quake/Quack benchmark scandal [hardocp.com] years (OMG, more than a decade...time flies) ago. Samsung wrote this boosting protocol to enable itself when running benchmarks and *only* when running benchmarks. There is no legitimate way to invoke it, so no user will ever see the benefit of it when running any app *other than* the benchmark itself.

      For the inevitable car analogy: you take a Samsung car for a test drive, and when you floor it you feel 200h

    • Re: (Score:2)

      by icebike ( 68054 )

      I'd have thought it would prove the principle that the optimizations aren't app specific.

      The optimizations ARE app specific, in that they are specific to all apps except benchmarks.

      Granted, you can twist your mind around to see the reasoning behind it, but in doing so you must come to the conclusion that using all 4 cores is so expensive in terms of battery power that running 4 cores is for the most part forbidden. And if forbidden, why have 4 cores?

      It locking fore cores to high power mode yielded a 2% advantage in performance this wouldn't be a big deal.
      But it yields a 20% performance boost in an app that is already stressing the processor.

      Doesn't that mean that the penalty for allowing apps to spin up 4 cores to handle peak load is SO BATTERY INTENSIVE, that Samsung won't allow it?

      Isn't that an argument for two cores?

  • Yawn (Score:4, Insightful)

    by sunking2 ( 521698 ) on Tuesday October 01, 2013 @12:41PM (#45004775)
    And 95% of consumers could care less as long as the screen looks nice and the battery lasts more than 2 hours.

  • Samsung != Android (Score:3)

    by Freshly Exhumed ( 105597 ) on Tuesday October 01, 2013 @12:42PM (#45004783) Homepage

    Take it out on Samsung for doing evil, or at the very least getting caught at it.

    • And who exactly is blaming Android? Samsung is called out in the title and almost every sentence in the fantastic summary. And the word Android is only used once to refer to the benchamarking apps.

  • No shock ... (Score:5, Insightful)

    by gstoddart ( 321705 ) on Tuesday October 01, 2013 @12:43PM (#45004805) Homepage

    There's lies, damned lies, statistics, and vendor performance numbers.

    I'm a little disappointed that there isn't actually any penalties for fudging your benchmarks -- it's blatantly lying to consumers about your product.

    And to me, that seems like it's bordering on fraud.

    • I'm a little disappointed that there isn't actually any penalties for fudging your benchmarks

      But there are. For example here in Finland you could release the hounds of KKV [www.kkv.fi] (Competition and Consumer Authority).

  • Huh? (Score:4, Insightful)

    by dgatwood ( 11270 ) on Tuesday October 01, 2013 @12:44PM (#45004815) Homepage Journal

    Even more damning was that the Note 3 was still faster than the G2 when run using 'stealth' (basically renamed) versions of the benchmarking apps which did not get the boost.

    Wait, what? How is that more damning? It sounds like that means the benchmark is faster even without cheating, which means that they've changed the kernel scheduler/idle timers/clock stepping in a way that, at least for the sorts of tests performed in the benchmark, improves performance—presumably because their case design and/or battery capacity is better, allowing them to get away with less processor throttling. That sounds like it is almost inarguably a good thing. And that's coming from somebody who has dealt with several of Samsung's products and hated almost all of them. What's with the hate?

    Unless, of course, they're being too aggressive about keeping the clock speed high, in which case you might argue that their battery life isn't what it should be... but that's pretty subjective.

    • Re: (Score:2, Informative)

      by Anonymous Coward

      I was perplexed too. After reading the article, I figured it out:

      It's damning in the sense that it's meant to skew comparisons with other Android devices, not to make it look better than their own previous offerings.

  • It's how Bill Gates built his fortune.
  • Battery life still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58409.png [anandtech.com]

    Browser speed still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58440.png [anandtech.com]

    Graphics performance still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58425.png [anandtech.com]

    For a device this much larger and heavier, you would expect they could overclock the chipset and get more performance than this.
    • How's that sub-par? It's the fastest phone available that doesn't lock you into Apple's walled garden. The iPhone isn't really useful for comparison, because then you're stuck with all the limitations that go with Apple products.

    • Re:All of this and the benchmarks are still subpar (Score:4, Insightful)

      by Ecuador ( 740021 ) on Tuesday October 01, 2013 @05:49PM (#45008963) Homepage

      Battery life still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58409.png [anandtech.com]

      You are comparing a phone with a 4 inch screen, with a "phone" that has a 5.7 inch screen. You can't compare battery life when the screen is what uses up most of the power. If you want a huge screen you have to compromise on battery life (and many other things - seriously, the note is ridiculously big to use as an every-day phone).

      Browser speed still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58440.png [anandtech.com]

      I don't suppose Samsung can do much about that. It is quite possible that with the same CPU, an Android would still be slower than an iOS device. Sure, Google has made a fast Java VM, but it still is a Java VM, right? For example, I had a Nokia N9 running Meego/Maemo. It could run circles around Android phones with the same CPU.

      Graphics performance still behind the iPhone: http://images.anandtech.com/graphs/graph7376/58425.png [anandtech.com]

      Ehm, this result (to which you cleverly linked directly - hiding the context) is ran in native resolution. The Note has almost 3x the iphone's resolution, so it would be pretty strange to come on top in fps. But in all the other GPU benchmarks which are ran at 1080p it does come on top of the iphone.

      But in any case I personally prefer a phone that has a good battery life, it can fit in my hand and lets me do whatever I want with it. So that rules out the note and the iphone ;)

  • Benchmarks are problematic by their very nature in that they are typically predictable and a manufacturer can simply say they have tuned their product for a given application. Let me give a good example of this from a product that isn't made for consumer use just to make my point:

    http://www.anandtech.com/show/7165/ocz-announces-zdxl-pcie-sql-accelerator-ssd-solution [anandtech.com]

    This is a PCIE SSD product designed to boot server performance explicitly for Microsoft MS SQL Server. This product has been explicitly designed

    • The difference is whether they admit up-front that it's been optimized for a specific application, or act like it handles *everything* better when that is not in fact the case.

  • Really? That's like using Slashdot polls for something important. Non-story.

  • ...are the dents left behind after drop-testing the thing until it breaks,

    Anything else is just a made up number.

Slashdot Top Deals

Some people manage by the book, even though they don't know who wrote the book or even what book.

Close