Business Intelligence At Lightning Speeds

"We have this problem where there are many, many touchpoints for the customer," says Jit Saxena, co-founder and CEO of Framingham, Mass.-based start-up Netezza. "First, that creates a large amount of data. Second, it makes the analysis much more complicated than it used to be."

By all accounts, business intelligence has become the oracle that companies rely on to make crucial strategic decisions or track the labyrinth of billing procedures. Netezza believes it has built the ultimate high-end server appliance,the Performance Server 8000, which combines a server system, storage and a database into a single, monolithic machine that can store and process multiple terabytes, as well as process business intelligence queries, at jaw-dropping speeds.

Netezza is backing its claims with detailed customer examples. Take Vibrant Solutions, a Fairfax, Va.-based company that processes and analyzes call-data records so customers like Sprint and Nextel can recoup thousands of dollars in expenses incurred through intercompany billing. Normally, it would take Vibrant six hours to process one day's worth of call-data records. Netezza's performance server did the job in less than two minutes. Then the server took 120 days' worth of call data records and processed it in 20 minutes.

The smallest Performance Server stores up to 4.5 TB, while the top-of-the-line system can hold 18 TB. It connects to a customer's existing servers, while the device has full database functionality. "The customer's current infrastructure does not change at all," Saxena says. "You can connect up to a 100 or more [Sun] E10000s on this. It just depends on how many applications you want to run."

Traditionally, most companies use a general-purpose system comprised of Sun servers, EMC storage and Oracle software. Or they may use IBM servers with DB2 software and third-party storage. Systems integrators would be called in to piece together a general-purpose system that some consider a clunky solution for such a high-brow job as business intelligence.

So what makes Netezza's system unique? The answer lies in its architecture. Two key parts of the new design are what Saxena calls Asymmetric Massively Parallel Processing (AMPP) and Intelligent Query Streaming technology. The query streaming places the silicon processors in proximity to the storage, so it can filter and process records as they come off the storage disk drive,taking only the data that is relevant to the query. "You use the memory for only the data that is relevant," he says. "You don't use it for unnecessary information."

The AMPP part of the design has a unique aspect: It combines both Symmetric Multiprocessing (SMP) and Massive Parallel Processing (MPP) architectures, so data needing high-level computing at slower speeds can make use of SMP, and data needing more speed can make use of MPP. AMPP entails putting a lot of intelligence on each disk drive,meaning each disk has its own microprocessor.

"All the operations happen in parallel," Saxena says. "I may ask a query, 'Tell me all the items that are sold over the weekend that have a 10 percent margin contribution.' That query comes from the database application. It sends the query to all these processors, and they act in parallel, so every drive is searching for the answer."

Since each drive has its own computing power, the disks can do calculations on only the relevant information that needs to go over the network to the host. In other words, at the front-end of this architecture, an SMP-based host compiles queries and provides the right amount of processing power to sort and aggregate large sets of query results. On the back-end, the data is distributed across many spindles to minimize I/O latency.

"So, yes, CPUs are getting faster. The I/O is getting faster, and the drives are getting faster," Saxena explains. "But the data sizes are growing faster than the aggregate capability of all those parts."

Even in this slow economy, Saxena says he has had no problem coming into the market as a start-up trying to get customers interested in buying a high-end server appliance, which on the low-end is priced at $622,000 and the high-end is price up to $2.5 million. To date, Netezza has raised more than $28 million in funding. "I would not have a problem raising money," he says. "The capital is not a problem for us."

More important, Netezza is looking to form partnerships with best-of-breed systems integrators, such as Accenture and KPMG, to deliver this platform. "We have a huge partner-based strategy," Saxena points out. "Our focus will be to train the systems integrator, and then it is up to the integrator to deliver the entire solution."