Lies, damned lies, and benchmarks: Why mobile metrics still matter | Wifi Walker, J B Chaparal Properties

Lies, darned lies, and benchmarks: Why mobile metrics still matter

Awesome Chart

There has been a behind-the-scenes contention during Extremetech newly on possibly or not mobile inclination should be benchmarked and what value (if any) these tests provide. The evidence is generally that factors like a look-and-feel of a device play a most bigger purpose in how we decider a smartphone than a desktop or laptop, with design opening measurements being reduction critical as a result.

While it’s loyal that opening metrics need to share a spotlight with some-more biased measurements when reviewing smartphones and tablets, I’d disagree that a new bang in handheld sales and a arise of Android and iOS is explanation that we need improved benchmark collection as against to chucking them altogether.

A benchmark is tangible as “any customary or anxiety by that others can be totalled or judged.” The tenure encompasses some-more than despotic measures of performance, a fact that’s quite critical in mobile products. According to a systematic method, a dimensions is deliberate current if it’s both accurate (meaning it measures a true/actual value of a thing) and accurate (the formula can be exclusively reproduced). In a box of benchmarks, a third criteria is necessary. In sequence to truly validate as valid, a exam contingency also be relevant, definition that a formula returned indeed have some temperament on a altogether opening of a device in question.

Accuracy vs. Precision
Accuracy vs. precision. Relevance would come into play if a archer was banishment a duck instead of an arrow.

This final indicate has necessitated a growth of opposite metrics for measuring mobile performance. In a normal PC market, benchmarks solemnly developed alongside a computers whose opening they measured. 2D desktop and 3D gaming/workstation metrics were combined as these capabilities emerged and were widely adopted.

IBM Simon
The initial smartphone, circa 1992.

Smartphones and tablets developed most some-more rapidly. The tenure “smartphone” has been in use given 1992, though a complicated epoch kicked off in possibly 2007 (if you’re an iPhone fan) or 2008 (HTC’s Dream, a initial Android smartphone). In a 3 years since, a multimedia capabilities, wireless/cellular speed, and opening of these inclination has grown exponentially.

As a result, benchmark formula have indeed turn more important even as exam developers and reviewers have struggled to keep up. The Droid Razr, expelled in early Nov 2011, is significantly faster than a Droid 3, that launched in July. Part of this disproportion is a fact that a Razr’s CPU is 20% faster than a Droid 3’s, though browser-based benchmarks like Rightware’s Browsermark and SunSpider uncover a Razr outperforming a Droid 3 by 50-100%.

Droid3-vs-Razr
Original information and formula accessible during Anandtech

The reason for a disproportion is that a Droid 3 runs Gingerbread 2.3.4, while a Razr runs 2.3.5. Google done a series of network opening improvements in 2.3.5, and a Razr’s opening reflects it. Mobile carriers are notoriously terrible about updating their devices, and Motorola’s Android 4.0 proclamation usually mentions a Razr, Xoom, and Bionic. The Droid 3 isn’t even mentioned on a company’s central ascent page.

Right now, a two-year agreement cost on a Droid 3 during Verizon is $199, a Razr is $299. If network opening isn’t critical to you, a Droid 3 competence good seem like a improved deal. For anyone who uses a Internet extensively, a Droid Razr is positively a improved option. Since carriers usually offer bonus pricing when users determine to extensive contracts, a opening disproportion between picking a Droid 3 and Droid Razr will still be in outcome dual years from now. This creates a $100 disproportion a lot easier to swallow.

The introduction of quad-core phones and some-more absolute mobile GPUs means we’re going to see some-more instances where program updates have a poignant impact on device performance. For users who devise to use phones for 2-3 years, opening metrics are going to matter.

  • 1 of 3



Leave a Reply

You must be logged in to post a comment.