Chipset Maker MediaTek Accused of Cheating in Benchmarks

An anonymous reader shares a report: We've seen several smartphone brands cheat benchmarks over the years, ostensibly in a bid to earn some ill-deserved praise among enthusiasts. But a new report suggests chipset manufacturer MediaTek could be gaming these scores. AnandTech uncovered evidence of benchmark cheating by MediaTek when it received a Helio P95-powered Oppo Reno 3 Pro (European version) and a standard Dimensity 1000L-powered Oppo Reno 3 (Chinese model). The outlet's suspicions were raised when the Reno 3 Pro beat the Reno 3 in the PCMark benchmark utility. This was strange, because the Helio P95's Cortex-A75 CPU cores are two generations older than the Dimensity 1000L's Cortex-A77 CPU cores. Furthermore, the P95 only had two of these cores versus the newer chip's four heavyweight cores.

A stealth version of the PCMark benchmark utility -- which manufacturers can't identify -- was installed on the Reno 3 Pro. This revealed a 30% drop in benchmark score compared to the previous questionable score, with Anandtech saying some tests in the benchmark dropped by 75%. The outlet also tested the Chinese version of the Reno 3 Pro, which swaps the Helio P95 for a Snapdragon 765G, and noted the phone ran the benchmark without resorting to cheating. AnandTech then dug into the offending device's firmware files and found references tying benchmark apps to a so-called "sports mode." It's believed that this mode ramps up things like the memory controller and scheduler in order to facilitate faster performance.

Say it ain’t so!

[jellomizer]

Chip makers distorting their benchmarks that is unheard of!

Re:Say it ain’t so!

[Cid Highwind]

And then getting caught by a "stealth" version of the benchmark, like ATI and quack3.exe [slashdot.org]

Re: Say it ain’t so!

[friesofdoom]

Ever single large GPU vendor has a team of people working on optimising the drivers for a specific game. This includes performance improvements, and "after-release" fixing of game graphics bugs, directly in the deriver with out modifying game code. They usually do this by checking the executable name, so changing the executable name will almost certainly cause problems in modern games. This is not cheating. It's the GPU vendor fixing broken game code for the developer.

Re:

[xlsior]

It is cheating if it increases the speed by disabling features like anti-aliasing and such by ignoring the settings the users explicitly enabled in the game/app. The increased speed comes at a cost of reducing the 'pretty', so the benchmark is no longer comparing apples to apples.

Re: Say it ain’t so!

[friesofdoom]

I agree, but that's not what AMD do.

Re:

[AmiMoJo]

As that ancient story mentions there is a real question here over if this is cheating or just legitimate optimization. As Mediatek says in their statement they optimize for certain apps that they know are very demanding, the same way we had per-game profiles for GPUs back in the day.

Where do you draw the line? If a game developer detects the type of GPU installed or the number of CPU cores and optimizes for them is that cheating? What about if Nvidia builds a check for a certain game into their driver that

They have a point though

[the_B0fh]

Isn't the point of benchmarks to show what is the maximum performance without falsely optimizing or bypassing the algorithms used? Or at least maximum possible performance on that set of hardware and software.

Re:

[Xenx]

The problem comes if/when they have a performance mode that can only be used behind the scenes by the benchmark tools. If the user cannot use those modes for the apps/games they want, the benchmark results are even less useful than they already are. If the phone maker allows the user to activate those modes, then there is no problem here.

Re:

[the_B0fh]

That would mean it's a manufacturer issue then. I mean, if you buy a car that's capable of doing 200HP and the car manufacturer locks it down to 100HP, is it the fault of the engine or the car manufacturer?

Re:

[Xenx]

I don't have enough information to agree or not. The article indicates that the code that enables the "sports mode" exists on phones from multiple manufacturers. While not conclusive, it does indicate that the code was likely included by MediaTek. If this is something that MediaTek added and doesn't allow to be used, outside of noted benchmark apps, it's on MediaTek for the deception.

Re:

[the_B0fh]

Agreed. If only MediaTek could use it, then yeah. If it was up to the phone manufacturer as their response implies, then it's on the phone manufacturers.

Re:

[arglebargle_xiv]

If you buy a car that's capable of this level of pollution and you gratuitously drive it in a manner completely unlike that in the emissions testing lab and so it emits more pollution, is that the fault of the car manufacturer?

Re:

[the_B0fh]

Dunno. Ask Volkswagen

Re:

[AmiMoJo]

In this case they would more likely have a list of games that trigger the sport mode. Unfortunately TFA doesn't include a list of other apps that trigger it so we can't see if they are just cheating on one benchmark or if this is a general optimization for apps they know how to tune.

Re:

[sabbede]

Well, TFA does have MediaTek's response - "MediaTek’s chipsets are designed to optimize power and performance to provide the best user experience possible while maximizing battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance."

Which sounds to me like it uses behavior to determine the performance state, not a list of apps. Which, let's face it, makes way more sense than using

Re:

[sabbede]

According to MediaTek's response to the test, it does detect when apps need the boost and automatically switches into "sports mode". I don't know if the user can activate it manually, but so long as the need is reliably detected I don't see a problem. I can't manually put my desktop's GPU into its max power state manually (so far as I know), but it definitely kicks in when I run a game. And since I wouldn't want it running flat-out all the time, I see that as a good thing.

The VW of chipmakers

[nospam007]

Nicely done.

Re:

[kot-begemot-uk]

VW studied from AMD and Nvidia. Mediatek has repeated to the letter what they were doing ~ 18 years ago.

It is just whippersnappers which do not know that.

Now, get yer scooters off my lawn.

Re:

[nospam007]

"VW studied from AMD and Nvidia. Mediatek has repeated to the letter what they were doing ~ 18 years ago.

It is just whippersnappers which do not know that. "

I'm 64 more snapping, less whipping.

Except the exact opposite

[sabbede]

VW put their engines into an artificially low-performance mode during testing that was never triggered in normal use. In this case, it is the benchmark preventing the hardware from going into high-performance mode when the need for it would have otherwise been detected.

benchmarks...get over it

[p51d007]

99.9% of users look at "numbers" for some reason when purchasing products. How about just trying the phone out in a store. If it is smooth and fast, just go with it. Most have a return policy anyway. Unless you are uber geek, who cares! Phones are way faster than what the majority use them for anyway.

Everybody cheats in benchmarks

[gweihir]

Usually by running them in completely unrealistic situations or in situations which specifically disadvantages the competition. Intel went so far, for example, to turn off a lot of optimizations in its C compiler when an AMD processor was detected, without any need to do so.

Who is the fool?

[backslashdot]

Why are we accepting of chipset makers benchmarking their own stuff? Who's the fool in that?

Why bother?

[GrahamJ]

Appleâ(TM)s chips are so far ahead everything else looks like crap anyway.

Re:

[Xenx]

In terms of the chips, that just isn't true. Depending on the specific test run, the A13 and the Snapdragon 865 trade places. The benefit of Apple devices is that the software optimizations and design decisions often lead to better real world performance and battery life.

I'm not a fan of some of the trade offs that Apple has made in the software to accomplish this. I prefer the Android approach where you've have some control over this. I get leaving more apps running in the background affects things, but

Not surprised.

[Gravis Zero]

They are also GPL violators, so unethical behavior like this is unsurprising.

optimizations

[sad_]

drivers for gpu's contain all kinds of optimizations depending on the application/game that is running, so the driver may operate different between two games just to make them perform the best. one size does not fit all, is probably the idea behind this.

nobody complains this is done for games, even though game fps is also used a lot as a benchmark. how is optimizing for a dedicated benchmark tool different? it's just bringing out the best possible performance under the conditions.
applications could optimize

Is that really cheating?

[sabbede]

The performance boost in "sports mode" is real, right? And the "stealth" testing tool evades the detection routines that would trigger the high-performance mode? So, where's the cheating?

If this "sports" mode is for some reason only available to benchmarks maybe there'd be an argument that they're cheating, but if games and other demanding processes also get boosted then it is the benchmark that cheats by preventing itself from being accelerated, resulting in artificially low numbers. Like if a benchm