Testing Methodology
In PCMark04 many, if not all, of the standard uses for a home PC are covered in one way or another when the full suite is run. In order to give a fair impression of the performance of the PC being tested futuremark have chosen to use a hybrid of the 2 main testing methodologys. These 2 methodologys are Application and synthetic. Application benchmarking involves taking a full application such as Word or Excel and running a set of tests using this program. Synthetic benchmarking uses a simulated code which emulates a particular task and in the case of benchmarking this task/its results are recorded. Futuremarks approach involves taking a freely available program such as DivX and implements section from the program, such as encoding and benchmarks using the “real life” program code. This results in an excellent true to life breakdown of the users PC performance. It also means that the benchmark remains a manageable size.

What’s Tested?

System Test:
File Compression / De-compression
For this test Futuremark encorporate Zlib which is a free tool for compressing data. This test is stresses the CPU using integer computation. The test runs a compression/decompression of 5 files over a period of 20 seconds. The results are measured in megabytes per second.

File Encryption/Decryption
For the encryption/decryption test Futuremark chose to use the Blowfish Algorithm, a freely available encryption algorithm. As with the compression test this is also a CPU intensive Integer test. The test is also run for 20 seconds using a group of 4 files with the result being measured in Megabytes per second.

Image Processing
Again an integer test using the CPU a set of 3 jpeg images are decoded continually over 20 seconds. The result is measures in millions of pixels per second.

File Decompression test is run concurrently with the Image Processing test, additionally the File Compression test is run concurrently with the File Encryption test.

Grammar Check
Grammar check uses Link Grammar Parsing Library which is available from and developed by The School of Computer Science at Carnegie Mellon University. It performs a grammar check in much the same way as Word or Openoffice would. Again this is an integer computation and so is testing the CPU and a single text file is tested continually for 20 seconds. The results are processed in kilobytes per second.

Virus Scan
F-Secure antivirus is used for this test. The software consists if 2 dll’s and the virus library file which scan 21 files for a total of 20 seconds. The results are measured in megabytes scanned per second and this test stresses CPU and ram.

Grammar Check and Virus Scan are run concurrently

Web Page Rendering
For the web page rendering Internet Explorer 6 is used. The speed of rendering 3 web pages is measures continually over a 20 second timeframe. The test is again integer and stresses CPU and memory. Results are in Pages processed per second.

Audio Converison
The audio conversion test uses the Ogg Vorbis librarys which are opensource (and similar in theory to MP3) to convert a wav file into ogg vorbis format. The wav file is 1.8mb in size and the time taken to convert it is reported in kilobytes per second. Rather than an integer based test this is a floating point algorithym which tests the CPU.

Video Compression
There are 2 video compression tests used in PCMark04. The first is a conversion of a 1.8mb Mpeg file into Windows Media 9 format. The second is a conversion of a 7.4mb file from DV format to DivX format. The conversions again use floating point algotithyms and the results are recorded in frames processed per second.

Graphics Memory
The graphics memory tests are written by Futuremark to stress the CPU, memory, AGP graphics bus and graphics memory subsystem and use code which is not available to the public. The test is based on DirectX® APIs , runs for 10 seconds and the results are measured in frames processed per second.
PCMark04 also uses the a directX 9 scene using the havock physics engine to stress the CPU and graphics capabilitys of the machine being tested. The graphics card is tested using integer operations, the CPU with floating point operations. In basic terms the scene consists of 600 blocks falling through a structure. The collisions between boxes and the walls of the structure are all calculated in real time. Added to this there are 4 lights sources illuminating the scene and all surfaces are multitextured. Frame based rendering is used which dictates that the same number of frames are generated on all PC’s, regardless of the hardware used. The output from this test is measured in frames rendered per second.

CPU Test:
The CPU test is run using a selection of the system test sections. For obvious reasons the tests which stress the graphics or memory components are not tested. This leaves a total of 9 tests. 2 sets 2 tests from the above 9 are run simultaneously on different processor threads. Following the completion of these tests the remaining 5 tests are then run in sequence.

Memory Test:
The memory test reports the performance of not only the man RAM within the PC but L1 and L2 cache. In order to test these components thoroughly Futuremark wrote C++ and assembly code to test various scenarios. The tests performed are reading data blocks from memory, writing data blocks to memory, performing copy operations on data blocks, random access to data items and latency. As with all good benchmarks the memory/cache is cleared before each test to provide fair and accurate results. The results are measured in megabytes per second.

Note: Latency is not used in the final score.

Graphics Test
The graphics test is split into 2 sections, 2D and 3D. For the 2D section there are 3 tests:
Transparent Windows: - in this test 10 windows are drawn onscreen and then faded in and out. The results are then recorded in average number of windows drawn per second.
Graphics Memory: - Tested as per the System Test suite.
Video Playback: - As with the video encoding test 2 files are used one a Windows Media 9 file at 1024x768, the other a DivX file at 640x480. The files are played back at the fastest speed possible.

NOTE: Video playback is not used in the final score as the highest playback speed it is dependant on the users hardware/settings.

The 3D tests are:
Fill Rate:- the speed at which the test systems graphics card can draw textures onto
3D objects is the fill rate. The Fill rate test is split into two sections:
Single-Texturing: A single texture is applied to 64 3D objects in turn.
Multi-Texturing: Multiple textures are applied onto 3D objects within a scene. More than one texture can be added in any one pass until 64 textures have been applied.
These results of the above tests are in million texels per second (MTexels/s).

Polygon Throughput Tests: Polygon throughput is a measure of the graphics cards ability to draw simple 3D objects. A scene with no texturing however a large number of polygons is drawn. 2 lighting methods are then applied and each tested.
Single Light: A single, one directional light source is used.
Multiple Lights: 8 lightsources are used which point at different angles/areas.
The result is in millions of triangles per second (MTriangles/s).

Hard Drive Tests
In the majority of the tests you have read about so far Hard Drive performance has very little or no affect on the results (most of the tests are loaded into memory before being executed). The hard drive test allows the user to specifically test the performance of their drive. The tests use Rankdisk which is again a freely available program which is developed by Intel. Here is Futuremarks explanation of how RankDisk works.

For these tests we use RankDisk, an application developed and copyrighted by Intel®. RankDisk is used to record a trace of disk activity during usage of typical applications. These traces can then be replayed to measure to performance of disk operations for that usage.
RankDisk records disk access events using the device drivers and bypasses the file system and the operating system’s cache. This makes the measurement independent of the file system overhead or the current state of the operating system. In replaying traces, RankDisk always creates and operates on a new “dummy” file. This file is created in the same (or closest possible) physical location of the target hard disk. This allows the replaying of traces to be safe (does not destroy any existing files) and comparable across different systems. Due to the natural fragmentation of hard disks over time, they should be defragmented before running these tests.
The traces used for each test were created from real usage.

The following four input traces are used:
Windows XP Startup: This is the Windows® XP start trace, which contains disk activities occurring at operating system start-up. This trace contains no user activity.
Application Loading: This is a trace containing disk activities from loading various applications. It includes opening and closing of the following applications:
_ Microsoft® Word
_ Adobe® Acrobat® Reader 5
_ Windows® Media Player
_ 3DMark 2001SE
_ Leadtek® Winfast® DVD
_ Mozilla Internet Browser

File Copying: This trace simply contains disk activities from copying approximately 400 Megabytes of files.
General Hard Disk Drive Usage: This trace contains disk activities from using several common applications.
These are:
- Opening a Microsoft® Word document, performing grammar check, saving and closing
- Compression and decompression using Winzip
- Encrypting and decrypting a file using PowerCrypt
- Scanning files for viruses using F-Secure® Antivirus™.
- Playing an MP3 file with Winamp
- Playing a WAV file with Winamp
- Playing a DivX video using DivX codec and Windows® Media Player
- Playing a WMV video file using Windows® Media Player
- Viewing pictures using Windows® Picture Viewer
- Browsing the internet using Microsoft® Internet Explorer
- Loading, playing and exiting a game using Ubisoft™ Tom Clancy’s Ghost Recon
Disk idle times were compressed to 50 milliseconds to speed up the playback time. Our studies showed that 50 milliseconds was the smallest idle time interval that didn’t affect the test results. The results of these tests are in Megabytes processed per second.”

By all accounts an excellent and thorough test.

 

Next: Testing results