Widespread Amazon EC2 Cloud Instance Reboots Spark Questions, Concerns

AWS sent e-mails to Amazon EC2 customers this week notifying them of the reboot, which was necessary "in order to receive some patch updates." The cloud giant said the majority of reboots would take minutes, depending on users' instance configurations. The e-mail also laid out for users which instances would be rebooted and at what times. The reboots are to be performed in coming weeks.

"No action is required on your part. Each reboot will occur during the corresponding scheduled maintenance window listed above," Amazon wrote in the e-mail. "Note that when a reboot is done, all of your configuration settings are retained. You also have the option to manage these reboots yourself at any time prior to the scheduled maintenance window."

Amazon also offered users the option of managing reboots themselves.

The massive reboot, and fast turnaround, has some users miffed, especially considering the December crunch and short notice.

"AWS wants me to reboot 75 EC2 instances within 6 days. This seems to be poor timing to have this happen in the month of December," one AWS user wrote on Twitter.

Another Twitter user added: "Reboot of this many servers across multiple regions has the whiff of a security patch."

And on Amazon's discussion forums, many EC2 users are reporting issues with the reboot, ranging from instances not stopping or getting stuck to instances becoming unreachable after the reboot. The issues sparked concern that another Amazon cloud outage might be on the horizon. Amazon's AWS Service Health Dashboard, however, gave the all clear, minus some EC2 scaling and provisioning issues in the East-1 region and delays in creating new environments in AWS Elastic Beanstalk.

Randy Bias, co-founder and CTO of cloud integrator and builder Cloudscaling, live-Tweeted the Amazon EC2 reboot saga, and in a blog post said that AWS is scheduling reboots across potentially thousands of AWS EC2 instances and for 32-bit and 64-bit instances. Bias noted that "this is likely a security issue and most likely related to the hypervisor."

Later, Bias added that there's "speculation that any instance booted before [Dec. 5, 2011] is affected, which is probably most of AWS globally" and that a recent Debian/Xen security announcement may be the root cause.

Amazon, however, said the massive instance reboot was part of routine maintenance.

"We frequently upgrade our EC2 fleet, with many of our patches and upgrades being applied invisibly to customers," an Amazon spokesperson said in an e-mail to CRN. "However, some updates require instances to be restarted and we periodically reboot instances in order to apply these updates. That's what you're seeing discussed today as we've sent customers the schedule for some of these (customers can choose to do them earlier themselves if they prefer)."

Amazon continued: "This rollout schedule matches pretty closely with the maintenance schedule you might see from traditional hosting providers or internal IT groups when they roll out software patches or updates. We are careful not to perform updates to multiple Availability Zones in the same Region on the same day so that customers won't have instances in different Availability Zones update on the same day. We're also giving notice a few days in advance for this maintenance window."