Enterprise-Class Benchmarks
The results are also presented with cost data, which is highly relevant to the channel. Cost metrics include maintenance costs for a specific length of time as well as onetime hardware purchase fees.
The only caveat is that system vendors actually run the tests. However, over the past decade, the council has taken important steps toward ensuring fair and honest testing practices, such as clarifying and fortifying their guidelines, creating an arbitration mechanism, fining vendors for violations, and most importantly, auditing results before they are published.
The council also has added benchmarks to be more relevant to modern usage models. Solution providers can examine the results to determine if a system is right for one of five applications. For example, the TPC-C benchmark simulates a population of users executing transactions against a database. The benchmark consists of a mix of five types of concurrent transactions that simulate an order-entry application. Performance is measured as the number of new order transactions generated per minute and the price per transaction.
The TPC-H and TPC-R benchmarks emulate decision-support systems, generating a high volume of complex queries against a database and forcing the server/database pair to examine large amounts of data to provide responses to business questions. Data is also modified as the benchmarks execute. The benchmarks measure queries per hour and dollars per query per hour under various database sizes and other conditions. The key difference between the TPC-H and TPC-R benchmarks is that the latter incorporates optimizations to make the queries run faster.
The TPC-W benchmark simulates a business-oriented transactional Web server, which measures the number of Web interactions processed per second and the price per interaction. Multiple Web browser clients make requests against a full Web server performing page composition and making queries up against a database.
The published data and more information on the benchmarks and organization can be found at www.tpc.org.
Have you used the TPC benchmarks? Let me know via e-mail at [email protected].