As expected no one was going to be quiet for long on the Futuremark audit report released earlier today. After much discussion and emailing, we were able to get some official reaction from both Nvidia and ATI in regard to the Futuremark audit report.
Nvidia released the following statement to us –
“Since NVIDIA is not part in the FutureMark beta program (a program which costs of hundreds of thousands of dollars to participate in) we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer. We don’t know what they did but it looks like they have intentionally tried to create a scenario that makes our products look bad. This is obvious since our relative performance on games like Unreal Tournament 2003 and Doom3 shows that The GeForce FX 5900 is by far the fastest graphics on the market today.”
From this statement it is clear to us that are calling into question the entire validity of synthetic benchmarking. While we would agree that in most cases real world applications are in many ways a better mode of comparison between products, the drivers are optimized for performance with specific applications. Some believe that Futuremark has an ax to grind with Nvidia, but at least in their statement, this does not appear to be the case.
It is clear that we would agree that certainly Unreal Tournament 2003 is a good example, we would still have to say that it is too early to be totally sure about Doom3 as this not yet a released product. Certainly, you can draw some conclusions as to the performance potential of products on the Doom3 engine, it will only be after Doom3 is closer to release that the full performance story can be told. Again, we are forced to ask the question, Do these driver optimizations effect gameplay or image quality? It would seem only if they do, will these optimizations be called into question.
ATI also came under fire in the Futuremark audit report as well, and they responded by issuing the following statement to us –
“The 1.9% performance gain comes from optimization of the two DX9 shaders (water and sky) in Game Test 4 . We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically and functionally identical to Futuremark’s and there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, and so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST.”
Well, it seems that ATI is making the correct move to address the issue, by removing the code from the drivers that gives the appearance of an advantage. It would seem on the surface that ATI’s situation is a bit different from that of Nvidia’s. While they can label it as an optimization, it is clear over the next several weeks both companies will have to start being more specific about the difference between an optimization and what is now being commonly referred to as a “cheat”.
One additional thing that has become clear. Depending on which sources you listen to, Nvidia while still participating in the Futuremark Beta program had access to the last code drop before the final code was released for 3DMark03. Several sources have seemed to indicate that the differences between the last code that Nvidia has access to and the final build where very minimal.
What does this mean in the big scope of things? Many questions remain, but as long as reviewers as well as OEMs continue to use 3DMark03 as one of the standard tests that is used to judge the performance of hardware, these issues are going to continued to get heated. Right now, Lars and Tom are hard at work evaluating the situation and hope to provide a more in depth report shortly. While we all have our early opinions of the situation, it is clear that you, the readers want us to properly research and look at all sides of the issue before giving you our in depth opinion, on what can now only be called a mess. This is what we are going to do.
In the mean time, we will be monitoring our THG Community forums for your comments about the 3DMark03 situation, to get your perspective, input, and feelings about the situation. Drop into the graphics section of our forums and feel free to leave us some feedback, we are listening.