ConnectWise CTO: We ‘Threw The Big Kill Switch.’ Now Cloud Backup Is Back.

ConnectWise CTO David Raissipour, in an exclusive discussion with CRN, details the phased restart, enhanced security controls, partner impact and lessons about transparency, rapid communication and more effectively defending MSP environments against AI-powered threats.

ConnectWise Wednesday brought its Cloud Backup service back online after a cyberattack forced the MSP platform provider to shut down the service, completing a staged restoration aimed first at getting partners reconnected to their backup data and then restoring the rest of the platform.

ConnectWise in an Oct.17 security advisory reported unauthorized application identities associated with its Cloud Backup, formerly SkyKick, and suspended the service until the company said it was safe. Microsoft subsequently attributed the supply chain compromise to China state-aligned group Storm-0940.

ConnectWise CTO David Raissipour Wednesday told CRN in an exclusive meeting that Cloud Backup is now operational after complete testing and monitoring and use by some MSP partners who needed early access.

[Related: ConnectWise Maps MSP Path From Reactive IT To Predictive Intelligence]

“The service is back up and operational,” he said. “We asked a number of our partners early on who had expressed interest in getting access to their archive data so that they could do restores for some of their customers. They on-boarded first. They enabled the cloud services. They were able to confirm that they can do backups and restores.”

Raissipour outlined the phased process for restarting the service and the new security capabilities ConnectWise added to help prevent such an attack in the future while acknowledging that it is impossible to guarantee that any environment is perfectly safe.

“AI-powered cyberattacks operate at a speed the likes of which you’ve never seen before,” he said. “What used to be telemetry data you could pick up over a period of hours from ITDR [Identity Threat Detection and Response] services or any sort of SIEM [Security Information and Event Management] that is analyzing all logs from applications—this is now happening in microbursts of seconds. So a lot of things that you rely on to make decisions are simply too slow these days. Humans reviewing logs to determine whether something was wrong cannot keep up with AI and the pace at which they attack you.”

Raissipour also said ConnectWise will discuss the incident directly at next week’s IT Nation conference, the security changes made since the attack and what it learned about communicating faster with partners.

“We’re going to own it and address it up front,” he said. “First of all, even if we wanted to run away from it, no one’s going to let us, so let’s be honest. And if we truly did learn that lesson around transparency and communication, we better apply that lesson, and it starts next week at IT Nation.”

Here is more of CRN’s conversation with Raissipour.

What’s the key takeaway from the Cloud Backup cyberattack?

The key takeaway is the service is back online. We were waiting for the final hurdle, which was getting Microsoft to re-register the enterprise application so we can generate credentials for the service principles and then allow people to connect to it. And that unfortunately took a couple of extra days. It had nothing to do with the product. It simply had to do with the old application being registered under SkyKick. The new one is registered under ConnectWise, and they wanted to make sure that we’re not a scammer pretending to be SkyKick. But we have been testing and monitoring the system in the pre-production environment and the production environment before we allowed access to it for several days [and] have not observed any malicious activity in the environment since October 17th, which has been a good thing. We have validated that there was no access to any other product or any other environment that we have because, as you know, we have cloud services in Azure as well as in AWS and in in our own data centers so not everything was in one place.

We brought the services back online in a staged fashion. One of the reasons for that is a couple of the services are intended to notify you on things like you haven’t backed up for a few days. So we wanted to make sure that people were reconnected back to the service and started their backup process instead of the first thing they receive is hundreds of thousands of messages telling them their devices haven’t been backed up. That was intentional to cut down on the noise and to get people to not panic that there’s something wrong. And we have rotated all service keys. All service principles have new credentials with Microsoft Entra ID. We have put in additional monitoring in our systems. We have done code reviews to ensure that there was no product vulnerability.

We went back and ensured that all production and even test accounts in our environment had two-factor authentication. We knew of a couple of credentials that did not. They were in our test environment, but nevertheless, we required it now in the test environment as well.

So is the service back up now?

The service is back up and operational. We asked a number of our partners early on who had expressed interest in getting access to their archive data so that they could do restores for some of their customers. They on-boarded first. They enabled the cloud services. They were able to confirm that they can do backups and restores. And then after that we opened the system to everybody, to all of our partners, to re-enroll. We’re following up tonight. I did webinars for everybody, for all of our partners. I’m doing one again tonight for the [Asia-Pacific] region and one tomorrow morning for North America and EMEA so that we can answer any questions they have. We have provided step-by-step instructions on how to go through and re-enable the tenants. We are doing a series of webinars where our technical folks will walk partners through step by step and give them a visual guide on how to re-enable tenants. Some of them have been partners for the former SkyKick for a long time. They may not even remember how to do it, so we’re going to guide them through those steps. We have provided an FAQ on our Trust Center with what we did and why we feel confident that the service was ready to be brought back online, as well as written communications to our partners as well.

How many ConnectWise MSPs were impacted by the service not being available?

North of 2,300 partners were impacted by the service not being available. We don’t have visibility into downstream Microsoft 365 tenants because the potential attack vector is somebody spoofing the Azure Enterprise app and connecting to the Microsoft tenant directly. And if the partner does not have an ITDR, an identity threat detection and response product, between them and their Microsoft tenant, they would not have seen an IP range outside our IP ranges and Azure trying to connect to the Microsoft tenant and pretending to be an Azure enterprise app. That is why some partners noticed it right away because they actually had an ITDR product. There are a couple of security partners of ours that came out and said, ‘Hey, we detected activity’ because they are the ITDR product between that partner and their Microsoft 365 environment. But we don’t have visibility because it’s our application talking directly to the Microsoft tenant. We don’t round trip through the partner, so we don’t see that.

So if a partner had ITDR in place, its customers would not have been impacted?

They would have certainly detected the anomalous activity and been able to take steps to suspend the application that was spoofing, and that would have prevented it. What we did on the 17th was we threw the big kill switch, for the lack of a better way to describe it. We just said no one has access. And by shutting down the service, we revoked the credentials from the service principles so those credentials could no longer be used to access the tenant, whether a partner shut down access on their own or not. And that was really a last resort for us because certainly it puts our product at risk because we’re turning off being able to back up and protect people’s environments. But we felt that it was the right thing to do because many partners and their downstream customers do not have that type of security sophistication and those products in place to be able to detect that. So we threw the big kill switch.

Do you see any ConnectWise Cloud Backup partners now looking around for alternatives after the attack?

Honestly, I’m sure there are. If I was on the receiving end, I would be concerned as well. So I can’t fault anyone for saying, ‘I’m concerned. I’m going to go look for alternatives.’

I can only talk about what we did to make sure that this is a safe environment to come back in, and the fact that we stand behind the statements that we have made, which is none of your data was accessed through us. Your archive was encrypted. The keys are separate; the archives are separate. It’s not the same credentials or encryption mechanism that’s used for the archives themselves. Those were not accessed.

We also were a victim of an attack. You saw what Microsoft posted on their site, I’m sure. This is a broad-scale attack that has impacted lots of companies. Some may not even know it. We can only provide assurance and evidence of what we did, how we handled the situation, how we communicated, what we have done to make sure that the environment is safe and secure now.

All throughout this incident, we reached out to partners, and we tried to provide them guidance and assistance on how to do threat hunting in their environments. We ran webinars every two hours around the clock for partners to join to learn how to do that and do our best to make sure that we regain that trust and to bring them back. …

It’s unfortunate, but I do understand that, and I respect their decision. The best we can do, like I said, is provide all the data that we have, be transparent around what happened, tell them what we did, how we fixed it, what we checked, what we found evidence of, and what we didn’t find evidence of. And we have shared all of that with our partners.

Just to make sure, no customer data was impacted, so nobody lost their data. They just lost access to the data?

Through us, through the archive, no. There was no access to customer data through ConnectWise. Zero.

Again, this was a very sophisticated type of attack. They targeted organizations that either have access to Microsoft 365 tenant data or the tenant itself. In this case, what was compromised is the credentials to access the Microsoft 365 tenant. We also had all that data, but through us, it’s all encrypted so you can’t get it through us. So if a partner looks in their environment and they do not find any signs of anything wrong using all the instructions that we provided them, they can feel confident their environment was not compromised.

ConnectWise went through a phased restoration of the Cloud Backup service. Please explain.

So the first phases were really focused on a few things. Bring up the management portal so people can re-enroll and re-register for the service. Bring up the backup and restore services so they can start backups and they can start restoring for any of their customers that were waiting to restore an endpoint or data they were missing. Then we brought up monitoring services for backups. And last was the API access, which is generally used for resellers to do transactions directly with us on the back end. The reason we brought those up later is, first, the focus was to make sure that partners currently using the product gain access to their data. The was because the monitoring services would generate lots of noise when they came back online after a week and a half of being down, saying you haven’t backed up. We didn’t want that noise to interfere with what was going on until the partners had re-established the connection and started up their backups, and then those alerts would go away. So as we brought those services back online, they wouldn’t get flooded with what are essentially useless notifications at that time.

Last, we brought up the ability to bring new tenants on board, because again, the focus was on making sure partners are safe and secure first and then open up for new business.

So is it open now for new business?

It is.

What are some lessons ConnectWise learned from this?

The first lesson is that AI-powered cyberattacks operate at a speed the likes of which you’ve never seen before. What used to be telemetry data you could pick up over a period of hours from ITDR services or any sort of SIEM that is analyzing all logs from applications, this is now happening in microbursts of seconds. So a lot of things that you rely on to make decisions are simply too slow these days. Humans reviewing logs to determine whether something was wrong cannot keep up with AI and the pace at which they attack you. It’s something we've also been telling partners: If you’re in the business of providing service delivery, you have to be able to operate at a different speed today than you were able to operate in the past.

And lesson number two?

I will admit our communications during the first 24 hours were not great. The first communication that we had when we shut down the service, we did not provide a lot of guidance, and we rightfully got backlash for it. Lesson learned. We should have provided more guidance with that first communication. We admit that to everybody. There were thousands of partners that joined our webinars on those calls to whom we said, ‘Mea culpa, we should have provided you more detail up front.’ … Our communications will be different going forward. We were very transparent, some would say perhaps overcommunicative from that point on, but it was justified for us to be overcommunicative at that point. We constantly monitor everything that we have in production. These [attackers] got through.

Did we strengthen things? Yes. … We put in additional controls and additional security in place. We looked at APIs that had any means of accessing any of the services and hardened those APIs, just as precautionary measures. You learn in these times to turn over every stone, and if there’s even a minute possibility that something could go wrong, we’re going to put controls in place to prevent that from happening again.

Can I guarantee anybody anywhere on the planet that we will never be the victim of a nation-state trying to get into our environment? As somebody who’s worked in security for a long time, that’s not possible. So the question is, how do you detect it, and how do you respond to it? And a lot of the things that we learned about the response was timely communications.

What is ConnectWise going to do then to protect MSPs and their customers going forward in the face of agentic AI attacks?

We had been investing in this in quite a few areas. We don’t currently have an ITDR security product. That is why one of our partners, Huntress, for example, who does have an ITDR product, was able to detect and report some of these things. That uncovered some gaps in products that we make available to our customers. We’ve got to go plug those gaps. In a few days, you’ll hear news about that.

We have been investing in the security products that we do provide for our partners, such as our managed EDR and managed SIEM. We switched to agentic processing of notifications and alerts that come in in order to be able to operate at machine speed instead of human speed. Unfortunately, this is the wrong way to learn that that was a good investment. But it confirms that we made the right move in that environment, and we will continue to tell all of our partners that the world has changed. Not for the worse. It’s just changed. And certain things you took for granted that you thought you had time to react to, you don’t anymore.

So look for solutions that operate at the same speed. We have invested in that for ourselves. We’re investing in technology for our partners. We’ll be talking a lot about that at IT Nation, about the things that we have done, products that are just about to launch. And we will continue to continue to beef up our security, continue to beef up our monitoring, continue to drink our own champagne and live our own talk track, which is run faster, use technology to fight technology, use technology to improve SLAs, use technology to improve response time, and make those things available to our partners as well.

You anticipated my next question, which was, what’s the key message about this at next week’s IT Nation conference?

We’re going to own it and address it up front. First of all, even if we wanted to run away from it, no one’s going to let us, so let’s be honest. And if we truly did learn that lesson around transparency and communication, we better apply that lesson, and it starts next week at IT Nation. We’re going to acknowledge the incident. We’re going to acknowledge everything that we did about it. We’re going to show people what we did to make the environment more secure than it was before, what we learned about communication and time to communicate, and how we have changed certain things going forward. We’re going to address that right up front and get the elephant out of the room. That way we can have a conversation instead of not admitting to something that happened. And at the end of the day, partners rely on us to provide technology. When we’re impacted, they’re impacted, and they need answers.

What kind of support did you get from Microsoft to remedy the situation?

Look, Microsoft has always been a great partner. This was not on Microsoft to remedy. This was on us. And we remedied this with outside help. We do not do these things on our own, even though we’re a larger company and more technical than many other companies. I think one lesson that everyone should learn is in the middle of a security incident, bring in outside help. No matter how technical and how smart you are, get another set of eyes on it. And we actually brought in two different firms, incident response firms, and we engaged with them very early on, even before we knew what was happening in the environment, to do the investigation, and then they helped us through the cleanup and remediation effort every step of the way. We reached out to Microsoft as we were ready to come back online to establish the service to go live again, and they assisted us with that.