AWS, Rackspace Complete Patch Updates, Reboots Prompted By Xen Security Flaw

Two public cloud giants that use the Xen hypervisor, Amazon Web Services and Rackspace, told partners and customers Wednesday that they completed system reboots on some servers to add a security patch.

The security vulnerability in the popular open-source hypervisor was discovered early last week. A Xen Security Advisory subsequently went out to some organizations, such as cloud providers, that would need to make patches. The advisory was released this week, after those updates were performed, to the general public.

AWS, the world's largest public cloud, raised eyebrows last week when it unexpectedly and somewhat mysteriously informed its partners that urgent reboots were to begin Friday and continue throughout the weekend and into the current week.

[Related: AWS Partners Prepare For Security Patch System Reboot]

On Tuesday, Jeff Barr, AWS chief evangelist, posted a blog updating clients that Amazon had completed reboots for the 10 percent of its "EC2 fleet" affected by the Xen vulnerability.

"Because our customers’ security is our top priority and because the issue was potentially harmful to our customers, we needed to take fast action to protect them. For the reasons mentioned above, we couldn’t be as expansive as we’d have liked on why we had to take such fast action," Barr wrote, adding, "the zone by zone reboots were completed as planned and we worked very closely with our customers to ensure that the reboots went smoothly for them."

Kevin RisonChu, director of systems and infrastructure at Digiteria, a San Diego-based Rackspace and Amazon partner, told CRN one-third of the AWS incidences Digiteria manages for its customers were affected.

The maintenance windows were set between 11 p.m. to 5 a.m. Pacific Standard Time, aligning with Digiteria's maintenance windows for its clients.

"Other than our monitoring system lighting up as the underlying hardware of the affected instances were being patched, everything came back up without incident and did not affect any of our customer workflows," RisonChu said.

NEXT: Rackspace Reboots Proceed Smoothly

On the Rackspace side, newly minted CEO Taylor Rhodes sent an apologetic email to all customers and partners Wednesday morning, that said the process affected almost 200,000 customers, some of the reboots took longer than they should have and some notifications lacked clarity. Rackspace is making changes to address those mistakes, Rhodes wrote.

"Like other major cloud providers, we were forced to reboot some of our customers’ servers to patch a security vulnerability affecting certain versions of XenServer, a popular open-source hypervisor. This maintenance was especially difficult for many of you because it had to be performed on short notice, and over the weekend," Rhodes wrote.

While Rackspace wants to be transparent with its partners, secrecy was necessary, as with any security vulnerability, so data could be secured before cyber criminals could exploit the Xen flaw. The problem has been fully remediated without any reports of compromised data among Rackspace customers, according to Rhodes.

Ben Mead, cloud and infrastructure lead at Credera, told CRN the Dallas-area Rackspace partner did have some customers impacted by the emergency maintenance activities, but only a small portion of them were significantly affected.

"As soon as Rackspace began notifying customers of the upcoming changes, Credera began working aggressively and proactively with our mutual clients to minimize the impact of these changes to their business and ensure that the service restoration process was completed with as little downtime as feasible," Mead said.

Mead praised Rackspace for its rapid response in identifying a resolution path, communicating the criticality of the change and implementing the necessary changes, all of which demonstrated "a strong commitment to the broader security posture of their respective cloud platforms."

"As a security professional, I appreciate the willingness to make the hard decision to prioritize the confidentiality and integrity of client data over the negative implications of service disruption with very short notice. As a partner, I appreciate the willingness of Rackspace to continue to demonstrate open and timely communication with their clients even under challenging circumstances," Mead said.

The hypervisor vulnerability made it possible for a malicious virtualized server to access data from other virtual machines running on the same hardware. In addition to exposing data, the malicious system could have crashed a physical server.

"This particular vulnerability could have allowed bad actors who followed a certain series of memory commands to read snippets of data belonging to other customers, or to crash the host server," the Rackspace CEO wrote.

Rhodes said Rackspace engineers worked with the Xen community to develop and test the patch, which became ready Friday evening. With the technical details scheduled for public release Oct. 1, the companies were left with the least-bad option of conducting reboots over the weekends.

PUBLISHED OCT. 1, 2014