Linux Expo Offers Peek Inside Amazon, EBay Data Centers

Virtualization is the joint show's most visible theme, with executives from a wide array of vendors and industry groups extolling the virtues of virtualized, flexible IT infrastructures built around open standards and commoditizing hardware.

Top engineers from Amazon and eBay took the stage Monday to detail the challenges they face in managing data centers able to handle the scale of their massive Web services. At that level, data center optimization becomes a critical business issue: "The efficiency in our data centers translates to dollars per quarter," Amazon Web Services CTO Werner Vogels said in his talk. "At our scale, data centers matter. They don't for everyone."

Vogels quipped about his love/hate affair with data centers. On the one hand, they're a frustrating bottleneck, sucking away management time and development resources to deal with infrastructure issues unrelated to Amazon's core retail and technology business. He estimated that only 30 percent of the staff time spent on infrastructure issues is related to value creation, with the remaining 70 percent devoted to dealing with the "heavy lifting" of hardware procurement, software management, load balancing, maintenance, scalability challenges and so on.

On the other hand, Vogels loves the way data center issues can become a point of differentiation for companies that excel at handling them. Flashing a slide with a photo of 365 Main, the San Francisco data center that went offline for nearly an hour in late July, Vogels obliquely referenced the catastrophic effects of that outage on 365 Main's clients.

"Two weeks ago, the lights were not on in this building," he said. "Many of the Web 2.0 powerhouses were no longer there."

After detailing the myriad hassles that lurk in data centers, Vogels focused his remarks on the philosophy, rather than the gritty technical details, of Amazon's solution. On-demand, pay-as-you-consume flexibility is the key, he said.

It's also the strategy behind a new service Amazon quietly released in alpha last week, the Amazon Flexible Payments Service (FPS), a developer-focused set of APIs enabling users to customize billing models and process payments through Amazon's transaction platform. Modern IT goals, which focus on shifting IT from a moneysink to a tool for enabling for business process advances, require such adaptability, Vogels said.

"Investing in data centers is not what makes you compete," Vogels said, "This is the message: in an uncertain world, you have to be able to acquire resources on demand and you have to be around to release them when you no longer need them. You pay for exactly what you use."

EBay Distinguished Research Scientist Paul Strong echoed Vogels' emphasis on flexibility: "The ultimate goal has to be to turn the data center into a machine that runs business processes," he said. EBay's strategy for reaching that goal involves virtualizing as much of its infrastructure as possible and focusing on the relationships between component processes and services. The benefit of such a set-up is flexibility; the drawback is complexity, Strong said.

Like Amazon, eBay carries a Web transaction load that's among the largest in the world: On an average day, it runs through 26 billion SQL queries and keeps tabs on 100 million items available for purchase. That load forces eBay -- along with Amazon, Yahoo, and Google, the peers and rivals Strong cited as sharing eBay's singular data-processing burdens -- to stay on the bleeding edge of the adoption curve for data center advances.

"We have to go where others have not gone, because the market demands it," Strong said. "If we don't keep up, our business is gone. The technologies we're playing with today I would expect to turn up in typical data centers now and in the years to come."

EBay realized as early as 1999 that it would have to seek unorthodox solutions for dealing with its skyrocketing data load, which was on the verge of physically exceeding the limits of database software. Over the next three years, it virtualized its database, which now spans 600 production instances residing in more than 100 server clusters. Its auction platform is similarly distributed, an architecture which allowed eBay to downgrade its data center from large SMP systems to smaller, cheaper servers.

While Strong and Vogels concentrated more on spreading the on-demand gospel of IT flexibility than on the low-level technical architecture of their data centers, conference attendees nonetheless deemed the talks helpful. Unless you have friends on the inside, it's rare to get any kind of glimpse inside such industry-leading infrastructures, said audience member Sanjay Dani, CEO of outsourced data center services provider Web Professionals.

"It gives you a flavor of the problems they're facing and what they're doing," Dani said. "The technologies they're working with are ones that will be coming to smaller companies down the road."

The biggest change coming is advancing complexity, both Strong and Vogels warned. As technology presses forward toward enabling ever greater workloads and flexibility, the number of dependencies administrators have to manage increases exponentially.

"You're constantly balancing the desire to be highly responsive to the business and deliver new value and services with the need to be stable," Strong said. "One thing is very clear about the next-generation data center: Change is constant. You're never going to be able to pull in a breath and go 'I'm done,' because you're not, ever."