The Murky World of 3D Benchmarking

Dom · Jun 11, 2003, 01:46 AM

A few years ago, a benchmark developer working for Ziff-Davis' benchmark operations group lamented, "In a few years, we won't be able to reliably or repeatably benchmark anything due to the dynamic optimizations of the OS, drivers and apps." It would appear that this prediction is being borne out today. And ironically, it's not entirely a bad thing for consumers.

Giving developers a programmable 3D pipeline that allows them to create customized code paths for different GPUs means that gamers living at different ends of the 3D food-chain will hopefully all be able to have at least a good (if not great) gaming experience. The problem is how to evaluate 3D performance in an "apples to apples" manner, and give consumers a good idea of what kind of performance they can reasonably expect from a given piece of hardware.

As we move further into the world of the programmable 3D pipeline and vertex/pixel shaders, this problem will persist and make level performance comparisons increasingly difficult. Is there an easy answer to solve this problem? Not really. But there are things that can be done to give consumers the information they need to make an informed buying decision.

Read More...

Al_Vampyre · Jun 11, 2003, 05:32 AM

I think there are two things that are needed to improve the situation...

1) Benchmarks based on actual games

2) Randomization of the run of the benchmark, something akin to UT flyby benchmark but with a completely random camera path so that no-one can predict which frames etc will be shown and which won't. The quality settings for these should be fixed so that IQ cannot be changed.

I also feel that the results should be given on a relative rather than absolute basis...a lot of people place too much emphasis on actual figures. We should choose a nominal figure for FPS lets say 100fps and then find out the maximum resolutions and quality settings that the card will run at 100fps.

This would mean that the optimizations would all be geared towards improving IQ (and therefore your gaming experience) rather than pure speed.

Just my 2cents worth (or it would be if the UK joined the Euro!)

Jun 11, 2003, 01:46 AM	#1
Dom DriverHeaven Extreme Member Join Date: Jun 2002 Posts: 12,942 Rep Power: 0	The Murky World of 3D Benchmarking A few years ago, a benchmark developer working for Ziff-Davis' benchmark operations group lamented, "In a few years, we won't be able to reliably or repeatably benchmark anything due to the dynamic optimizations of the OS, drivers and apps." It would appear that this prediction is being borne out today. And ironically, it's not entirely a bad thing for consumers. Giving developers a programmable 3D pipeline that allows them to create customized code paths for different GPUs means that gamers living at different ends of the 3D food-chain will hopefully all be able to have at least a good (if not great) gaming experience. The problem is how to evaluate 3D performance in an "apples to apples" manner, and give consumers a good idea of what kind of performance they can reasonably expect from a given piece of hardware. As we move further into the world of the programmable 3D pipeline and vertex/pixel shaders, this problem will persist and make level performance comparisons increasingly difficult. Is there an easy answer to solve this problem? Not really. But there are things that can be done to give consumers the information they need to make an informed buying decision. Read More...

Jun 11, 2003, 05:32 AM	#2
Al_Vampyre DriverHeaven Senior Member Join Date: Jan 2003 Location: Scotland Posts: 1,512 Rep Power: 55 System Specs	I think there are two things that are needed to improve the situation... 1) Benchmarks based on actual games 2) Randomization of the run of the benchmark, something akin to UT flyby benchmark but with a completely random camera path so that no-one can predict which frames etc will be shown and which won't. The quality settings for these should be fixed so that IQ cannot be changed. I also feel that the results should be given on a relative rather than absolute basis...a lot of people place too much emphasis on actual figures. We should choose a nominal figure for FPS lets say 100fps and then find out the maximum resolutions and quality settings that the card will run at 100fps. This would mean that the optimizations would all be geared towards improving IQ (and therefore your gaming experience) rather than pure speed. Just my 2cents worth (or it would be if the UK joined the Euro!) __________________ "Knowledge is knowing that a tomato is a fruit; wisdom is knowing not to put it in a fruit salad" - Brian O'Driscoll - Ireland Rugby Team 2009 Grand Slam winning Captain.