Making Use Of Benchmarks

Recently, we decided to take a second look at what these products can do for solution providers. The results were not encouraging. After carefully looking at the methodologies used by many top benchmark ISVs, we found little evidence that solution providers can make use of the metrics generated by these tools. All of the research came down to a simple and practical question: Can the results from benchmarks be used to make real World decisions about PC systems, corporate software and end user productivity? The answer is no. Well, almost no.

Results gathered from benchmark software cannot be turned into practical information. For instance, software emulation benchmarks that run general office and spreadsheet calculations cannot predict end user productivity. The same goes for complex calculations, graphics rendering and disk I/O.

Part of the problem is that output from these products is static. Meaning, metrics generated by benchmark software are essentially snapshots of a system at a specific time. Long-term metrics like system degradation cannot be measured because these products run process-intensive calculations that lock the systems during the time they run. It's not practical to run these benchmarks all the time. Therefore, results cannot be analyzed over time.

That's not to say that all of the benchmarks out there are completely useless. Hardcore gamers, for instance, have always been interested in squeezing the most of their systems by overclocking CPUs and trying new ways to improve subsystem performance. Some of the CPU benchmarks can help gamers find ways to improve performance. But the bottom line is that these system metrics are only useful for tweaking hardware components.

Benchmarks that work:

Network performance benchmarks are one exception. Measuring network performance is analogous to CPU performance. The network benchmarks help administrators understand protocol usage, bandwidth utilization and strengths and weakness related to various types of connections.

Products such as Ixia's IxChariot and open source NetPerf and Sun's UPerf provide critical information about the health and configuration of networks. With IxChariot, for instance, solution providers can figure the most optimal position of wireless access points. Network performance tools also help solve complex application architectures that use clusters and grids.

Application benchmarks are another exception. Like network performance tools, applications benchmarks help solve bottlenecks. In this case, code bottlenecks in multi-tier applications. Application load running tools from Compuware, HP (OpenView using Mercury Interactive tools) and Parasoft are used extensively by developers and software architects to optimize application performance.

What needs improvement:

While network and application benchmarks are indispensable at providing tactical information, most of the benchmark products fail miserably at providing long term business impact analysis. To a large extent, business performance benchmarks are missing from these products.

Algorithms are all well and good but data collection and analysis at every layer of a system is the best way for IT managers to make strategic decisions on the viability of their applications and hardware platforms. Right now, long-term cost analysis is approximated, so business measures are not accurate.

What's more, most products on the market act in silos. To correlate data between different benchmarks, IT administrators have to manually massage the data and create custom reports. There are no standards for data output.

Tweaking application code, services and hardware platforms so that they work in concert is also missing from benchmarks. While some products include whitepapers on best practices, the information is narrowly focused on solving single actions. Data analyses based on steps taken using whitepapers are not measurable. Moreover, there's no way to coordinate manual steps with automatic tweaks or even with automatic validation from systems.

Our wish list is simple. In the future, software and hardware benchmark products need to be integrated so that every log can be collected and analyzed with ease. In addition to reports, benchmark products need to provide business performance measures and forecast problems using integrated data from disparate sources.

This is not an easy task because most benchmarks are designed to work with test systems and non-production hardware. Despite what the benchmark software industry touts, application servers and operating systems will have performance degradation whenever service agents are present collecting real time data. There are some exceptions. Products that bind to executables will consume far less CPU cycles than service agents.