MSPs Scramble To Determine Microsoft 365 Tenant Impact In ConnectWise Cloud Backup Breach

In an update Wednesday, ConnectWise said ‘service availability’ for ConnectWise Cloud Backup, SaaS Security and Cloud Management will ‘return in stages’ and take several days.

Robert Cioffi, co-founder and CTO of Progressive Computing, a Yonkers, N.Y.-based MSP, sprang into action with his team when he learned on Oct. 16 that a Microsoft 365 credential belonging to MSP platform maker ConnectWise’s CloudBackup, formerly SkyKick, had been compromised and was being used to gain unauthorized access to Microsoft 365.

“I knew in the past we had touched SkyKick on a couple of projects, but we had not been using it widely,” he said. “I was taking a stance that something potentially could be affecting us so let’s operate under the mindset we have a problem until we knew we didn’t.”

Fortunately, within several hours Cioffi and his team had determined that Progressive Computing and its customers were not impacted.

Threat researcher Huntress had alerted Progressive Computing that it had one customer with a SkyKick product, but it had already been disabled and it was not the impacted application.

For Cioffi, the ConnectWise security incident brought back the PTSD he suffered when his company was hit by the historic attack on Kaseya in 2021, which resulted in all 80 of his customers being hit with ransomware.

Fortunately, most of his customer endpoints were restored in 17 days after that Kaseya attack thanks to volunteer help from other MSPs.

After that outreach of fellow MSP support for his business, Cioffi started MSP911.org to help MSPs impacted by an attack like his company experienced.

Cioffi said every security incident like the ConnectWise breach brings back those memories from the Kaseya ransomware attack.

“It brought back the negative feelings of what occurred on Friday, July 2, 2021,” he said. I could feel that tightness in my chest that this could now be another event for us. When I realized it was not going to impact us, I still felt that way empathetically for those that would be going through this.”

Cioffi said so far no MSPs have reached out to MSP911.org for assistance but he and other MSPs stand ready to help. “This is how we fight back against the bad actors,” he said. “I want to put a dent into any impact on other MSPs. If anyone is impacted and needs help with advice, direction or coaching go to the MSP911.org website, fill out the form and we’ll contact them.”

Cioffi said the big questions for all MSPs are just how many Microsoft 365 tenants the threat actors were able to gain access to and then just how many of those tenants were subject to data exfiltration.

Every MSP Impacted Needs To Do Research To Determine If There Was Data Exfiltration

“Every MSP along with ConnectWise is going to need to do the research to determine whether data exfiltration occurred,” Cioffi said. “If they determine that data exfiltration occurred then they need to mitigate that data exfiltration. If you have clients that are subject to any kind of compliance, that means notification needs to go out to their customers. There is still a lot to be unwound from this. It could take weeks or months to determine the net impact on MSPs and the Microsoft 365 tenants. The digital forensics needs to be done on each of these tenants. Some MSPs may not even know they have been impacted. There are a lot of unanswered questions about this.”

Progressive Computing itself has changed its own internal practices to audit registered apps within each Microsoft 365 tenant that it manages as an MSP. “That means a lot of work,” he said. “For risk management practices, you want to turn off things that are not needed or used any longer. From an account management perspective, we want to make sure there is nothing that would provide any unnecessary or undue exposure.”

The ultimate victims in the ConnectWise attack are Main Street small-business companies. “If this was a large bank or a Wall Street company, this would be all over the news,” he said. “Because it is thousands of nameless and faceless organizations it does not get the headlines it deserves.”

On Oct. 17, ConnectWise shut down access to ConnectWise Cloud Backup, SaaS Security and Cloud Management for more than several thousand MSPs as part of its response to the breach.

“When we shut down the [Cloud Backup] service we revoked our access to the Microsoft 365 tenant environment, which means any credential that is out there can no longer be used to access that environment,” said ConnectWise CTO David Raissipour in an interview with CRN. “That does not mean partners are safe. It simply means that can no longer be used to access their environment. But when a threat actor gets into an environment, they create other opportunities for themselves to get in. They can create other accounts, they can elevate privileges, they can do all kinds of things in that environment.”

The ConnectWise Cloud Backup Service is composed of over 4,000 Microsoft Azure services that need to be brought back online, said Raissipour.

“To bring them back online we need to rotate certificates,” said Raissipour. “We are running as fast as we possibly can while making sure we don’t create shortcuts for ourselves in order to rush things to market. We need to make sure the service is safe, secure and the environment is clean to bring it back online.”

ConnectWise Is Reaching Out To Help Any And All MSPs Impacted

Raissipour stressed that ConnectWise is reaching out to help any and all MSPs impacted to make sure their Microsoft 365 environments are safe and secure. “This has nothing to do with Microsoft,” he said. “Microsoft is in constant contact with us. They provide us threat information and help as needed. But this is something we are working with our partners to do.”

In an update on Oct. 21, ConnectWise said its teams and independent specialists are “validating the affected environment” with restoration of the suspended services subject to security review.

“We expect restoration to take several days,” ConnectWise said in the advisory. “Service availability will return in stages, and the start of restoration does not mean all capabilities are available. We recognize the continued disruption to your business and customers and remain focused on restoring service safely.”

ConnectWise said it intends to publish step-by-step instructions on how partners should reconnect affected services including any required reauthorization and verification steps. “Keep affected application connections disabled until ConnectWise provides formal instructions for your service,” the company said.

In an email response to CRN, Jonathan Semon, a principal security operations analyst for Huntress, said Huntress cannot say with certainty how many MSPs or MSP customers were impacted. “We can only speak to what we can see, and we can’t give a complete count,” he said.

Huntress reviewed application sign-ins across more than 10,000 tenants with ConnectWise applications where it can see those logs, said Semon.

Huntress saw threat activity involving ConnectWise Cloud Backup and Cloud Management applications with sign-in activity from the “malicious infrastructure” across more than 1,000 Microsoft tenants, “more than 300 of which appeared to succeed,” said Semon. That said, he cautioned that in some of those 300 cases the application was not present so “success” does not always mean “access,” he said.

“Counts will differ between reports, ours or anyone else’s, and we’d treat any single total as incomplete until ConnectWise publishes its own,” said Semon.

Dustin Bolander, an MSP and serial entrepreneur who is a partner and CEO of CyberDrain, a Microsoft 365 management platform, said CyberDrain developers worked all through Saturday launching a feature update on Oct. 18 that identifies impacted Microsoft tenants and disables them.

The Industry Has Not Yet Discovered The Damage Done

“There is not enough information out there at this point as to the impact on MSPs,” Bolander said of the ConnectWise security incident. “We don’t know what the attackers are doing with the backups. I don’t think as an industry we have discovered all the damage yet.”

Bolander, who is also the founder and managing partner of Austin, Texas-based MSP Clear Guidance Partners, and founder and CEO of Beltex, an MSP cybersecurity insurance provider, said he believes the ConnectWise incident will ultimately change the security architecture that MSP platform makers and MSPs themselves use going forward.

“It appears that the way ConnectWise connected to every MSP client with Microsoft 365 was using the exact same application to connect and back up all of the MSPs,” he said. “So if the hacker gets into one of those, they have the keys to everyone. The way the software industry is moving is to use a unique application per MSP or even per customer of the MSP. This is going to be a huge kick to get the industry moving faster in that direction.”

The security architecture change would essentially provide each MSP customer with its own individual security key, said Bolander.

Bolander said it is going to take some time to see the ultimate impact on customers from the ConnectWise breach. “There is a longtail risk here,” he said. “The bad actors have these backups. The question is, what are they going to do with it? I am sure the bad actors are going to start phishing customers with the info they have gotten. If you are an attacker, this data is valuable. Context is key for better phishing and social engineering attacks.”

Bolander, who only slept a few hours Friday night to get ahead of the ConnectWise incident, commended Huntress for actively alerting MSPs to the attack. “We had a bunch of people working Friday to double-check our client environments,” he said.

Bolander said Beltex insures hundreds of MSPs and reached out to all MSPs that were using ConnectWise. “We sent a notice early Saturday morning,” he said.

“In these situations, the faster you can move the faster you can minimize the damage,” he said. “Everybody needs to move fast to identify and disconnect the stuff that is at risk”.

Jason Wright, CEO of Avatar Managed Services, a Houston- based MSP, said he was grateful that his company got off the ConnectWise platform in December in favor of a full Microsoft stack. “All I could think when I saw this is, ‘Thank God I got off the platform,’” he said.

Avatar Managed Services consolidated its business on the Microsoft Dynamics 365 cloud-based suite of ERP and CRM applications, said Wright.

The Problem With MSPs ‘Putting All Their Eggs’ In One Basket

MSPs are at the “mercy” of SaaS providers like ConnectWise when there is a breach, said Wright. He said too many MSPs are “putting all their eggs” in one basket when it comes to their MSP platform providers.

Just as troubling to Wright is those MSP platform providers have become attractive targets for bad actors. “The bad actors and nation-state actors know if they successfully hack the MSPs they can get access to millions of clients,” he said.

Wright said he believes his business is more “secure” with the full Microsoft stack. “You are using individual point solutions within that stack,” he said. “It provides you that parity. Additionally, it is Microsoft and they have innumerable security resources to help protect MSPs.”

Keith Nelson, CEO of Irvine, Calif.-based Vistem Solutions, which provides managed services for the Port of Los Angeles and a number of aerospace clients, said the big lesson from the ConnectWise breach is MSPs need to spend more time “parsing and reading” Microsoft 365 logs and firewall logs so they can immediately detect any suspicious network traffic activity.

“As an MSP if you were watching those logs you would have seen irregular traffic that would have alerted you to the ConnectWise breach,” he said. “You should know what traffic is normal for your tenants, and anything that is abnormal you need to react to right away. You are better off being prudent and cutting something off when something like this happens. We side on the side of caution.”

Nelson said his company uses the ConnectWise PSA and RMM, but not the ConnectWise Cloud Backup product. He said Microsoft 365 is an attractive target for bad actors. “Microsoft 365 is a significant attack vector,” he said. “If you want money, you rob banks. If you want data, you attack Microsoft 365. Your email is a treasure trove of information for hackers.”

David Stinner, founder and president of MSSP US itek, Buffalo, N.Y., said he was relieved that he moved off a number of ConnectWise products eight months ago.

“I would be shaking in my boots right now if I was using the ConnectWise Cloud Backup because of the risk it would have put on my Microsoft customer tenants,” he said. “For us that would have meant several thousand mailboxes impacted by this.”

Stinner, who is also on the global leadership committee of the GTIA trade association for MSPs, said he also canceled another two ConnectWise products, the RMM and the CPQ quotation tool, earlier this month.

“As an MSP you really need to understand the security posture of the tools that you invest in because they can make or break your MSP,” he said. “If your vendor tools expose your MSP and expose your customers’ data, it could potentially destroy your business. You have to choose your vendors very wisely and understand their security posture and their investments in their security.”

Stinner said his advice for MSPs is to “audit” their clients’ Microsoft Enterprise Apps installed on their tenants to make sure there is no access to applications no longer in use. “As an MSP we also need to make sure our clients didn’t install Enterprise Apps that could pose high risk, as any exposure could certainly come back on us as the MSP managing the enterprise,” he said.

Huntress’ Semon, for his part, said one of the lessons learned from the incident is that “a single credential for a multi-tenant application can reach many tenants at once, so third-party applications deserve the same scrutiny as any other privileged access, including regular reviews of which applications hold admin roles.”

Secondly, Semon said, “defenders need visibility into application and service principle activity, not just user sign-ins, and many organizations don't have it because that data often depends on licensing tier and on logging being enabled and retained.”

Last but not least, Semon said, “speed and information sharing between vendors, security providers and partners made a difference here.”