Continuum Health Partners, MLB Network Handle Data Center Challenges With VMware, LTFS

Despite the numerous data center technologies that have emerged in the last decade, IT administrators are still addressing some of the same issues they've had since the first computers -- the need for more storage and the need to keep it safe.

"One of the major challenges we face every day is the need for more storage, said Jill Wojcik, corporate director of distributed architecture at Continuum Health Partners, a non-profit hospital system based on New York City. "All of our users continue to keep all their information, especially our electronic medical records, for the necessary requirements to meet HIPPA regulations and all the laws around health care."

The hospital system's users, about 10,000 of them, are spread across five hospitals and about 20 remote sites, and share 600 servers, she said.

Continuum's storage needs double every year, said Wojcik. "When we started five years ago, our storage environment was 300 gigabytes. Now it's grown to 2.5 petabytes. How do we protect that and handle compliance and make sure it's available 24/7, from anywhere for all of our users both internal and external? That's where we have the most challenges."

Wojcik said that Continuum addresses the challenge by implementing a storage virtualization solution from DataCore Software. "That allowed us to buy less storage because the DataCore [solution] makes it appear like we have more than we do," she said. She added that DataCore is vendor agnostic, which allowed them to significantly lower the company's cost-per-terabyte by purchasing raw hard drives.

As daunting as the storage challenges at Continuum might be, Tab Butler faces an even greater challenge. Butler is director of media management and post production at MLB Network, based in Secaucus, N.J. Its services include live streams and on-demand replays of all major league baseball games.

"We have tremendous storage consumption. First are the high quality assets, [which are] 50- or 100-MBps files of a single baseball game that can be up to 150 gigabytes in size," he said. "I need to record five or six versions of that from cleans and dirties, for home and away," he said. A "clean" feed is one containing just the game coverage, before stats are added by the broadcaster.

"I may consume one terabyte of storage per game. With 15 games a day and [a season] of almost eight months, that's 2400 games. You need someplace to store these for the history of baseball." When the system was built in 2009, Butler said that MLB Network put into place an archive of between two and three petabytes. By 2010, it had grown to five petabytes. "This year it will be storing in six petabytes of archived video content. We need to call it up, store it, keep continuity for all the users and keep our cost down."

For the agility and cost efficiency needed to maintain such a storage-voracious web service, Butler employed VMware, and lots of it. "We had a short time to ramp up and build this infrastructure, and for making our deadlines in 2008 and 2009, VMware was absolutely critical."

What more, since they were given just four months to design and implement the systems, they were never sure that what they started with in June and July would be what they ended up deploying in December.

Next: How Data Is Protected

"So using VMware gave us a quick and dynamic way of building out new application platforms," he said.

MLB Network is up to about 65 VMware applications running in its back office environment.

"That was extremely economical ... and made the difference of being successful and being challenged at every move," Butler said. "Today in the broadcast environment, we have migrated to that virtualization using VMware and IBM hardware platforms."

Prior to that, Butler said that the organization's infrastructure was constructed on a server-by-server basis. "We're the first site that migrated to a VMware environment for full production. We can dynamically monitor and manage the systems and scale by adding CPU and memory. And tools VMware provides give us the insight to perform proactive maintenance."

To keep those balls and strikes for posterity, MLB Network uses good-old tape backup and the not-so-old Linear Tape-Open format.

"In a baseball game with four hours of a recording, chances are the highlight a year from now from the event may have been the final out of the game," said Butler. "With the sizes of tapes, how can I get to that content quicker?"

He uses the new LTFS file system and tricks to to provide high availability and high reliability, as well as the portability. "I can make multiple copies and do off-site archival storage and deep archival storage, as well as use that same medium for fast retrieval, that's Holy Grail for me," he said.

Meanwhile back at the hospitals, Wojcik also is charged with maintaining availability of the data systems while grappling with HIPPA and other government regulations regarding medical records, which are among the industry's most heavily regulated. Again, DataCore plays a role. "We also implemented metro clustering, which that allows us to implement [high availability]. The data is replicated across DataCore's synchronous mirror between our main data center in Secausus and another site in New York."

If anything goes wrong at either site, the opposite site kicks in and users "experience no downtime or latency. The data is always up and always protected," she said, adding that because of the way it's written by DataCore, at the block level "you can't get to it unless you have their technology built in. So we know it's protected at all sites."

Butler, Wojcik and others described how they address their data center operations and solutions this week in a session at ComdexVirtual called Trends and Challenges in the Data Center 2011.