Test Metrics Of Olympic Proportions
Once you know what data you're trying to measure, the tester's next step is to develop the methodology. This is the part that describes how to test what you're testing. Evaluating an individual laptop, app-phone or printer is one thing; take it out of the box, note the look and feel, run a few tests and take down the results. Except for the performance benchmarks, this process is largely subjective.
But measuring the speed of a network switch, router or security appliance is something else entirely. Think of it as judging an Olympic event that's a cross between figure skating and the luge. There are subjective parts as in skating, but the performance measurements require an objective precision and results that cannot be argued or debated. Objectivity should be paramount, and one way to guarantee objectivity is to develop standard test methodologies for all categories of equipment and employ them consistently.
For the reseller, such objectivity can greatly simplify decisions about what to offer customers. If there's demand only for the fastest router regardless of price (like in your dreams), your selection process might simply be to start at the top of the performance list and move down based on availability. But if the customer defers to your judgment, then ease of setup and maintenance (and, of course, profit margin) might trump bits-per-second. Either way, having the relevant data about every product is an absolute must. And that data must have been acquired completely free of bias.
Other metrics will vary with the product under test, and physical attributes and characteristics will always be subjective. The same is true when judging usability and intuitiveness of human interfaces such as software GUIs, printer controls or an app-phone. However, there's one thing that should never vary, and that's reporting of the presence -- or absence -- of features. For instance, all Apple notebooks shipped after 2006 include MagSafe, Apple's patented magnetic power connector that pops off when pulled slightly to prevent damage. This feature is so appropriate for mobile equipment that I will henceforth find fault with any laptop that does not employ this or something similar.
Perhaps the most significant aspect of test methodology is repeatability. When developing a testing process, it's important that the test deliver exactly the same results any time it's performed. The point is that your methodology should include performing every test at least twice. This pays dividends down the road because such tests generate results that can be absolutely trusted for comparison with devices in the future.
E-MAIL EDDIE CORREIA AT [email protected]