Businesses Apply New Metrics In Measuring IT's Value

At Global Crossing, CIO Dan Wagner wants IT managers and staff to attend a total of 500 sales calls in 2006 in support of the company's back-from-the-brink emphasis on business growth.

Dan Drawbaugh, CIO at the University of Pittsburgh Medical Center, expects IT developers to get it right the first time. His goal: Cut in half the time it takes to deliver a data-sharing project to hospital clinicians.

Business is all about the numbers, and information technology must be part of the equation. But the metrics used to measure the value of IT resources and investments are changing as CIOs go beyond classic return-on-investment and total-cost-of-ownership formulas to prioritize their investments. How many customer interactions failed last month? How many hours were wasted on low-value computer maintenance rather than tech-driven innovation? To what extent did software developers contribute to the bottom line?

To get answers to these kinds of questions, IT and business managers are getting more creative in what they measure and how. "We're gutting our entire metrics look--putting them to the side," says Wagner of Global Crossing, the IP-based network services provider that, after three years of financial restructuring and cost cutting, is growing again and looking for new yardsticks to measure its progress.

In addition to tracking how many times IT personnel go on sales calls, Wagner has put a priority on training and skills development, on what he calls "strategic software development," and on IT security. And he's tracking all of them. For example, information on physical and network security is combined into a color-coded security vulnerability index that's shared with board members, among others. "It gives a temperature check at the senior management level," Wagner says.

At HP, Mott is just finishing the first year of a three-year IT overhaul he was hired to engineer and oversee. The goal is to create a world-class IT organization and state-of-the-art computing infrastructure, while driving down IT costs. Mott has created an IT portfolio scorecard to set goals and monitor progress. HP's objectives include:

• Delivering 98% of IT projects on time by 2009, compared with 81% in 2006.

• Contributing $3 billion in IT organization "annual benefit" in 2009, on top of the $5.5 billion to be contributed from 2006 through 2008.

• Producing "benefit per developer" of $450,000 by 2009, three times more than in 2005.

• Increasing the IT staff time spent on innovation--developing new capabilities versus supporting existing ones--to 80% in 2009, compared with 46% in 2006.

HP's IT department conducts a cost-benefit analysis on each project to get at the "benefit" to the business. Benefit refers to projected operating profits--either expense savings that drop to the bottom line or new revenue opportunities. "We try to get everything down to a bottom-line number," Mott says.

Mott uses many metrics to gauge HP's business technology efforts, but there are a handful that he relies on most: number of active projects; on-time delivery; project phase "time boxing"; time spent on innovation versus sustaining technology; cost-benefit analyses completed; and annual benefit per project. He says these metrics work best when used together rather than focusing on just one of them. Money saved through efficiency and productivity gains can be redirected as investment in innovation.

OLD-SCHOOL THINKING

Mott's emphasis on benefits to the business and his 80/20 innovation-to-support ratio are just two examples of how business technology executives are relying less on metrics that evaluate IT internally--number of servers per system administrator, for example--and more on those that look at IT's impact on the business. Once-popular measurements such as IT investment as a percentage of company revenue miss the value IT brings to a company, says Gary Scholten, CIO of Principal Financial Group, this year's No. 1 company in the InformationWeek 500 ranking. "That's really an old-school kind of metric," he says. "It's an interesting data point, but not an area of focus for us."

Principal doesn't ignore metrics that look internally at IT performance--they're useful in benchmarking against competitors--but it sees greater value in measuring how IT impacts corporate operations. Scholten cites Principal's Work Secure and Retire Secure Web services, which help its clientele of 15 million workers employed by 100,000 small and midsize companies manage and buy retirement, investment, and insurance products. The company tracks demand for its online services compared with offline demand. For example, employee participation in Principal's services is 52% higher through Work Secure, and the "close ratio" is 100% greater.

Such metrics help Principal identify the right level of IT investment. "It's the business success criteria we're looking at, and tying our IT investments that were made in those areas to those business strategies," Scholten says.

Electronics parts distributor Avnet weighs spending on infrastructure and applications against gross profit, which, unlike revenue or expenses, reflects how efficiently the company operates. "It makes sure we keep our IT spend in line with the basic profitability of the organization and doesn't allow IT spend to get out of sync with the amount of money we spend on other operational expenses," CIO Steve Phillips says.

CIOs must strike a balance between measuring performance to squeeze costs and making sure they're spending enough on the right projects to improve the business. Vanguard Group CIO Paul Heller thinks of metrics in something of a Maslow's Hierarchy, a pyramid of six big areas:

• Security and disaster recovery are the foundation, where measurements range from fraud attempts to Web site attacks.

• System availability and responsiveness, with measures like Web site response time.

• Project delivery, including timing and budget.

• Quality, speed, and efficiency in software development, with gauges such as defects allowed into production and lines of code written per developer.

• Innovation, plotted on a grid with client impact on one axis, operational impact on the other axis, and high, medium, and low as the values.

Heller focuses most of his attention at the bottom of the pyramid, around security, and at the top, on innovation.

EXPERIENCE STILL COUNTS

For an application rollout at its offices in Brazil, storage vendor EMC measured the costs, performance, and quality of outsourcing the work versus doing it in-house. The contractors didn't need to be based in Brazil, but EMC wanted them in or near the same time zone. It commissioned 15 technologists in Mexico to complement 22 of its own managers and employees in Brazil. EMC wanted to see what effect "near-shore" contractors would have on the success of the project. Among the data points analyzed: cost of labor, project budget and schedule, and software defects (see "Staff Mix").

Not every decision goes strictly by the numbers. EMC's IT department still relies on its own employees for project planning and business analysis, areas where it prefers to maintain a direct touch with its primary customers, the company's internal business units. "I don't have the ability to tie a metric to that work," says Ken LeBlanc, EMC's senior director of business operations and portfolio management. "We just know in our gut that is the last area" where outsourcing would be considered, he adds.

Indeed, good numbers are no substitute for hard-earned experience. Metrics can "get in the way of getting customer satisfaction and getting business done," says Bill Kwelty, CIO and a 40-year veteran of fleet leasing firm Automotive Resources International. Kwelty has tried tools that measure how much of the IT staff's time is spent on development and maintenance but found them cumbersome. "The management to oversee all of this didn't seem like time well spent," he says.

Point taken. It's not enough for IT metrics to be relevant; the task of number crunching can't sap productivity.

A NEW TWIST

In some cases, IT organizations are applying well-established metrics in new scenarios. The University of Pittsburgh Medical Center has adopted Capability Maturity Model Integration, developed down the street at Carnegie Mellon University's Software Engineering Institute, as well as the institute's Standard CMMI Appraisal Method for Process Improvement, known as Scampi. CMMI isn't new; it's used by defense contractors and manufacturers. The model's use in health care, however, is novel.

CMMI helps companies identify a place to start a project, develop a framework to prioritize actions, and define how improvements will benefit the organization. It also measures the benefits of a process against those realized from similar projects previously undertaken.

Among the medical center's projects that benefited from CMMI is U-Pay, a Web system that lets staff use one program to record payments of various types, such as for patient services, parking, and donations. As components were added to U-Pay, practices embodied in CMMI allowed IT developers to get them right the first time. That saved $20,000 with each release or $300,000 over three years. "Our projects don't fail," says Chris Carmody, director of the software engineering process group within UPMC's Information Services Division.

When the University of Pittsburgh Medical Center adopted CMMI, it benchmarked its performance in developing interfaces between high-volume, data-intensive health and patient management systems against its previous development efforts (see "Development Speed"). UPMC spends more than $220 million a year on IT, including $100 million on capital projects. Competition among UPMC units for that money is stiff, and methodologies and metrics to determine how best to deliver services are indispensable. "There's a lot of consternation every year over who gets what dollars," says Mike Wrobleski, the Information Services Division's finance director. "There's only a limited amount of dollars that can go around, so we really wanted to make sure we got it right."

To be sure, established metrics like CMMI are deeply rooted in many companies and will be for years. Among InformationWeek 500 companies, about a third use the CMMI and Six Sigma methodologies in their daily IT operations, and half use the IT Infrastructure Library, or ITIL, for things such as problem resolution and change management. And CFOs and boards of directors will continue to insist on metrics such as ROI and return on capital, says Ian Campbell, CEO of Nucleus Research. "Return tends to be the driver for all investment decisions," he says.

Yet Campbell agrees that new times call for new approaches. The trend toward business process outsourcing, where work is distributed to contractors who may be based in locations around the world, calls for metrics that let business and technology executives measure the value of every partner, maybe every person, involved in that process. He suggests an outsourced customer-service call center as an example of where that might work. "It can change the way we measure things from a project to an individual," Campbell says.

CIOs must think hard about whether they've got the right tools to track IT's contribution to business objectives. In Optimize magazine's just-completed CIO Effectiveness survey of more than 700 CXOs, CIOs, IT staffers, and line-of-business managers, seven of 10 CXOs say the most important criterion in considering the effectiveness of a CIO is the CIO's ability to support companywide business strategies. Metrics demonstrate that kind of support in hard-to-argue terms. Full results and analysis of the CIO Effectiveness survey will be presented in the December issue of Optimize (InformationWeek's sister publication) and, beginning Dec. 1, at Optimizemag.com.

At United Stationers, CIO Dave Bent wants to start measuring services to the business. He plans to create a "services book" that defines all the services IT provides to the business, using ITIL definitions as the base, and to measure the performance and cost for all those. Under United Stationers' current model, Bent's organization tracks the calls received and resolved by the IT help desk for PC support; the new approach will measure how satisfied employees are with their desktop tools and related services.

"I don't think we're tracking the ultimate customer satisfaction," Bent says. "Could the service be so bad that people don't even call? Have we truly measured the root cause of these calls?" The answers to those questions may require a new tape measure.

-- with Chris Murphy and John Foley

Continue to the sidebar:
Techniques For Measuring IT's Effectiveness

Continue to the blog:
Researchers Measure E-Mail's Potency As Business Tool