Comcast said on Friday that it was grappling with a nationwide outage of its cable, voice and internet service.
In a tweet, Comcast said the outage was caused by a cut fiber affecting one of the company's "large backbone network partners." The company also said the outage appeared to affect other providers as well.
One of Comcast’s large backbone network partners had a fiber cut that we believe is also impacting other providers. It is currently affecting our business and residential internet, video and voice customers. We apologize & are working to get services restored as soon as possible
— ComcastCares (@comcastcares) June 29, 2018
Down Detector, which provides real time oversight of outages, showed a sharp spike in reported problems as of 1 pm EST with 11,058 reports of problems originating mainly from Mountain View, San Francisco, Chicago, Philadelphia, Seattle, Atlanta, Washington, Arlington, Va., New York and Pittsburgh. Sixty five percent of the reported problems were internet related, 17 percent were a total black out and 16 percent were television related.
Andrew Gregoire, CEO of Ace Technology Advisors, a nationwide managed service provider headquartered in Fall River, Ma., said most of his customers utilize Ace Technology Advisor's LTE backup service and are not affected by the outage.
"This shows the importance of having a well thought out and executed backup plan," said Gregoire. "The white glove treatment that we provide with our LTE Service is our value add to the customer."
Gregoire emphasized that LTE backup is imperative for any business. "When a customer depends on voice and data as their primary driver and they don't have a backup they are setting themselves up for failure," he said.
Gregoire said nationwide outages are uncommon and it is unclear at this point what caused the outage. "Comcast is a great partner and does a good job of getting outages like this fixed quickly," he said. "Nevertheless, it is imperative that customers have a good backup solution."
Quy "Q" Nguyen, founder and CEO of Allyance Communications, one of the top global communications solution providers, said the outage shows the importance of having redundant networks for mission critical business systems.
"Every carrier has outages no matter how big they are," said Nguyen. "The lesson to be learned from this is plan ahead and make sure you have redundancy."
Nguyen said it is not unusual for Allyance Communications to be called in for a telecom services audit at Fortune 100 companies and find no redundant system for mission critical applications.
"Problems happen when you are not designing or deploying solutions correctly," he said. "Every customer at this point has moved a product or service online that heavily relies on network connectivity."
Allyance, for its part, works with some of the top companies around the globe to design and deploy carrier neutral network solutions.
"Nothing surprises me anymore," said Nguyen about the lack of redundant systems at major corporations. "You would think Fortune 100 companies would have a plan for something like this, but whether it is lack of knowledge, a staffing issue or that they simply have too much on their plate there is often not a redundant system architecture in place. We take those headaches away for companies with our services."