Rethinking benchmarks for a new consumer

Posted by Jon Peddie, President, Jon Peddie Research

Analyst Opinion - We have all seen the excellent work various web site and analysts like us have done over the years in comparing the latest graphics AIBs, and, more recently, the new graphics-enabled motherboards. Some have been more thorough than others, but all contain performance measurements from 3Dmark/Vantage and/or FPS in game play. And for the high-end game enthusiast where performance is everything, that’s enough. But is it enough for the rest of us?

Now that the world is starting to come out of the financial shock of 2008 we (analyst types) can see that there will be a new consumer for the next two years or so. The tightening of credit, the exposure to financial disaster and our global awareness of over-extended people has changed our collective consumerism behavior and attitude. There will be less frivolous, less impulsive and less irrational buying. And when purchases are made, they will be more value-based. Also, we have become ‘green’ aware (finally). The new consumer will be sensitive to those parameters as well. It’s not likely that every consumer will make green his or her first criteria for purchase, but a few will.

Budget, and benchmarks per buck will be the most common first evaluation, and I think that will even reach into the lunatic fringe (count me in that group) and the enthusiast. Yes, we will want the highest benchmark score, but the delta benchmark can’t exceed the delta cost.

In a recent evaluation by Tom’s Hardware (April 8, 2009) on an Intel and an Nvidia motherboard: “G45 And GeForce 9400: Integrated Chipsets For Core 2 : Intel Or Nvidia?,” Tom’s Hardware Guide (THG) did a great and fair job of evaluating the relative performance of the two motherboard’s integrated graphics (IGPs) chips, and showed the Nvidia mobo to be a clear winner with an overall 3DMark Vantage score (CPU and GPU) of 4358 using a 2.83 GHz Core 2 Quad Q9950 vs. 1690 for the Intel mobo with the same CPU.

The two mobos are pretty much equal with the biggest difference being Intel having 10 shaders running at 800 MHz, and Nvidia’s IGP having 16 shaders running at 1.2 GHz. In non-performance items, Intel has more USB and PCI support than the Nvidia board.

An article by Anandtrech (October 15th, 2008), “The IGP Chronicles Part 3: Nvidia's GeForce 9300,” lists the idle power consumption of the two IGPs as 68.4 W for the Intel mobo and 69.6 watts for the Nvidia mobo, and under load (Anandtech chose Company of Heroes as the test) the two mobos, with cool & quiet enabled, drew 96.1 watts and 88.2 watts respectably.

And with a little web search, I was able to find an Intel Desktop Board DG45ID for $110, and a DFI LAN GF9400-T2RS Nvidia board for $130.

Listing those three factors (benchmark, price, and power) you get dramatically different results on a single parameter basis. In the case of this comparison of two mbos, we see that a simple 2D comparison of benchmark scores, shows Nvidia’s mobo is 2.58 times faster than Intel’s mobo, but when you go to an n-dimensional analysis the Nvidia Mobo is only 2.38 times better than the Intel mobo.

It must be pointed out that this analysis wasn’t done to prove one company’s product is superior to another (although I imagine Nvidia will be delighted with this finding), but rather to arrive at a better metric for evaluating products for the mainstream consumers.

And it doesn’t mean one should omit the base analysis in forming a buying decision. You can, for instance imagine a product that has horrible benchmark score, but is ridiculously cheap and doesn’t use any power. That product would win, but would be terribly disappointing as well.

A point in time

Benchmarks are temporal. As soon as they are posted, they are already wrong, especially a multi-variant benchmark like PDW.  Drivers get improved and change the raw performance benchmark, and prices drop over time. Wattages stays pretty consistent under load, but manufactures have been known to detune their parts a bit to get a better power usage measurement.

We have a new generation of consumers who have to get the most value and performance possible on a limited budget. The new consumer needs more than a single parameter measurement of a product. Using performance, price, and power in combination gives a better view of what to expect for our investments.
A better Mark

As a consumer, I want the most performance (3DMark) per dollar (BpD), with the lowest wattage. I’m calling that the PDW mark (performance-dollar-watts).

Summary

We have a new generation of consumers who have to get the most value and performance possible on a limited budget. The new consumer needs more than a single parameter measurement of a product. Using performance, price, and power in combination gives a better view of what to expect for our investments.

Dr. Jon Peddie is the founder of multimedia market research firm Jon Peddie Research. The opinions expressed in this commentary are solely those of the writer.