Colonial Pipeline And Accellion CEOs On Handling Huge Hacks
From proactively forming relationships with regulators to limiting board updates during an attack, here’s what Accellion CEO Jonathan Yaron and Colonial Pipeline CEO Joe Blount recommend when responding to a cyberattack.
Reliving Their Darkest Days
Top executives from two recent high-profile cyberattack victims relived their darkest days for the benefit of the more than 5,000 in-person and virtual attendees of the Mandiant Cyber Defense Summit, taking the audience through how they first heard of the attack, interactions with their IT department and boards of directors, the biggest lessons learned from the experience, and the role law enforcement played.
Hackers in December 2020 chained together exploits for multiple zero-day vulnerabilities in the legacy Accellion File Transfer Appliance (FTA) product and exfiltrated data, demanding payment to ensure the return and deletion of the data. The data leak site of the Clop ransomware gang was used to publish some of the stolen data to encourage payment of the ransom.
Then in May, a ransomware attack prompted Colonial Pipeline to shut down its 5,500-mile natural gas pipeline for five days, resulting in more than 10,000 gas stations across the Southeastern U.S. being out of fuel. Colonial Pipeline paid Darkside $4.3 million on May 8 with the hope of restoring operations on its pipeline sooner, although federal officials were able to seize back most of the ransom.
Accellion Chairman and CEO Jonathan Yaron and Colonial Pipeline President and CEO Joe Blount spoke with Mandiant CTO Charles Carmakal about their experience being in the line of fire. From proactively forming relationships with regulators and limiting board updates during an attack to seeking maximum transparency and prioritizing essential systems while restoring, here’s what Yaron and Blount advise.
Keep Feelings At Bay When First Learning Of The Hack
Yaron: Despite the fact that our technology was a legacy technology, it had an anomaly detector implemented into it. The anomaly detector went up in one of the clients in the Northeastern United States, an education institution. And they called us and we had sensed that it was not good. At that moment, I picked up the phone and called my head of technology and basically created a war room.
And we basically started the huddle of what‘s going on. Having gone through life, I learned don’t involve feelings at this moment. It is the moment to understand the picture. Understand how bad the situation is and do everything you can to minimize the damage and obviously help clients.
My first worry was, ‘Is it a government issue? Or is it just a commercial issue?’ We didn’t know. ‘Is it a singular event, or is it a mass event that touches every part of our client base?’ We were lucky that it was our legacy product, which at that time had about 10 percent of our client base.
But among that 10 percent were very important institutions such as banks of countries, government institutions in this country, major banks around the world, major health-care organizations. I knew there‘s a big issue here to deal with, and the first order was to understand the magnitude.
Get CEO Involved Since There Isn’t Enough Time Or People
Blount: We knew we had a threat, and we knew that threat had to be contained. And therefore, we shut the pipeline down in order to do that. There was not enough detail to know whether it was beyond the IT system, whether it hit the OT system, whether we even had a physical risk on the pipeline at that point in time. We handle 5,500 miles of pipeline, being responsible for it physically, that‘s a great effort.
After an attack like that, your CEO responsibilities immediately become to contain an attack and remediate the situation, so that becomes the focus. And although I have an extremely talented team, what you find out after an incident like this is you don‘t have enough time in the day, and you don’t have enough people. So you become actively involved yourself.
One of the roles I was assigned was to be the conduit with the federal government in order to brief them on a daily basis. And we were fortunate enough in the early hours to get an agreement with the government that we would work through one conduit, in this particular case the DOE [Department of Energy].
That really allowed us to communicate effectively and collaborate effectively with the federal government and other parties that we work with, right down to the lobbyist groups that we‘re involved with. The initial ability to share with the industry what happened to us so that if there was someone else, they already had some heads up as to what could be around the corner for them.
Expect Communications Challenges If Hack Occurs During A Holiday
Yaron: Your life changes at that moment. I‘d say, in my case, Christmas and New Year’s didn’t exist. The attackers always choose the most inconvenient time. So the whole event started the 17th of December but got their access elevated just before on the 23rd or 24th of December.
Basically we made ourselves available, the top three executives, 24x7, to every one of our clients around the world and governments. I had 3 a.m. phone calls, I had midnight phone calls, I had the middle of the Saturday with Charles [Carmakal] a discussion. And it‘s just around the clock.
There were about 300 potential institutions around the world that could have been impacted, about 90 or 95 had an impact, and about 30 or 35 had a significant impact. The biggest challenge was actually fishing everybody from their holiday dinner and holiday parties and hoping that they‘re sober and can listen to what you have to say. And it was not an easy task.
It was very, very difficult. Different countries, different culture, different languages, you find out that you don‘t necessarily have the absolute correct list, because it’s one of those things that when you need it, it’s never perfect. And suddenly you’re searching for the right person. It was not simple those first 10 days.
Determine Which Systems Are Essential And Which Can Wait
Blount: We have 5,500 miles of pipeline. We have hundreds and hundreds of delivery points around the Gulf Coast all the way to New York City. It‘s an enormous physical presence in the field. We knew we had a ransomware attack, but could we potentially have a physical attack? Could the perpetrator potentially be a nation state who was trying to cause damage to the United States?
Like everybody else, we wanted to know what happens next, what are you seeing in the systems, when can we bring the systems back on? And even though we brought that pipeline back on fairly quickly within six days, we want to get on as quickly as we possibly could. So a lot of questions about where‘s that system, what system do we need to bring on, what system can wait.
For example, our financial systems, we didn‘t have our financial systems up even bare bones for at least 10 days. So the pipeline was already up and running long before we had access to our billing system and things like that. What do we need versus what can wait? And then there’s information that we might need that a federal government agency might be asking for.
And again, lots of people want lots of answers, and they want the answers yesterday. And as you can appreciate as a cyber professional, a lot of those answers don‘t come immediately. And quite frankly, a lot of those questions aren’t critical when your primary focus is containing the risk, remediating it, and bringing the asset back online.
Expect Customer Pushback If Ordering A System Shutdown
Yaron: My first expectation was to manage the list of potential vectors. My second was to make sure we did come up with a fix in 72 hours after the first zero-day. My third was to offer customers immediate movement to the modern platform. And at that point, that‘s when I also called Mandiant in for help because a second opinion on a situation like this is critical.
We do have significant forensic internal capability, but we don‘t have everything, and you need somebody from the outside to keep you honest and look in the mirror. Maintain the list. Make sure everybody is up to date as much as possible. And offer remediation by moving. And we did have some major players move in a day or week because they had to go back online.
There was one occasion, a very interesting one, a Fortune 100 company that I actually went on the phone with the senior management, and I said, ‘You have to shut the system. You just have to shut the system.’ And they said, ’I’m sorry, it’s too important. We cannot shut the system. We’re going to monitor it second by second.’ And I said, ’that’s crazy.’
But they succeeded actually to keep the perpetrators out. And that‘s how important it was.
Limit Board Updates To Once Daily To Focus On Recovery Efforts
Yaron: First of all, I‘m privately held, which is a huge advantage in circumstances like this. I don’t have the whole world calling me. I have a few major shareholders. Particularly, 12 months before, there was one new group that made a big investment, and obviously, they don’t understand what’s going on. My first expectation was let us do our job and do our best, because everybody panicking doesn’t help.
You have to stay cool. You have to make tough decisions. Now, on the flip side, what I owe them is a daily phone call of the updates of the day. But not six times a day. Once a day, 30 minutes, here‘s where we are, here’s what we know, here’s what’s going on. Very similar to a war room in my old days when I had hair on my head. My shareholders were fantastic. We have the full trust, and they let us do our job.
In the second zero-day, we‘re like, this is by far more sophisticated. And I said, OK, the enemy knows something we don’t know. And we just have to tell people they have to give us 72 hours to figure it out and clean this thing. And at that moment, I said my responsibility is to take the high road and tell them the truth. And the truth was, at that second, we didn’t have all the answers.
There‘s a responsibility at a certain level of time, and I came out and made that decision, and the vast majority listened to us. And that’s why eventually no more than 10 percent got heavily penetrated. One victim, one client of mine is paying, and it’s bad.
Clarify Any Inaccurate Media Reports To Members Of The Board
Blount: As you can imagine, everybody wants information. We are privately held, but again, our owners have a lot of investors who want answers as well. So I‘d say our first official board meeting was very early on during the situation. But the board had a lot of confidence in us already, they’re very familiar with stop work authority, they’re very familiar with our incident command structure.
So, there weren‘t a lot of questions in the first meeting because they knew we needed to focus on the problem. They needed to be apprised of what we did know at that point in time. And again, information is very limited in the first couple of days, as you know. And I would say that they were pretty brief as a result of that.
They have trust in the team to go about the task of remediating and bringing that system back on, and they knew enough to know they needed to stay out of the way. So like we did with the federal government, I had an informal agreement with the chairman of the board. I would talk with him at least once a day. And then we would pull the board together when there were significant things to report.
And then of course we have our communications team that did an extraordinary job throughout the event. And that team had to deal with a lot of media that wasn‘t always accurate in the early hours and early days. And so a lot of communication took place with the board’s communications team in order for them to have the accurate information versus what perhaps they might see on the television.
Form Relationships With Regulators Before An Incident Occurs
Blount: You don‘t have enough time, and you can’t create more time. And people want information now, and they want information that perhaps isn’t going to be available for quite some time. So I think the big lesson was know the audience before the event ever occurs. Work your relationships before anything like this ever occurs. So with us, we spent a lot of time in Washington with our regulators.
They know who we are, they know what we do. They trust us. They regulate us, of course, but they have a relationship. So when an event like this occurs, they know who to talk to. And we certainly know who to talk to. So that was extremely helpful. But again, when you have an asset like ours, there‘s a lot of voices on the other end who want that information now.
One of the biggest takeaways was we set up that one conduit with the government, which allowed us to communicate all the way from the White House all the way through to any regulator that we were responsible up to the lobbyist groups that we use who are also tremendously helpful in disseminating information to other companies during a time when no one really knows what happened.
It‘s all about communication and trying to create as much time for yourself to cover everything that you need to cover. Your typical CEO job went out the door just a few hours ago, and it‘s not coming back for quite some time. You’re in the communications game, and you want to make sure that information is accurate because there will be a lot of inaccurate information out there that’s not helpful to anybody.
Supply Detailed Information To Avoid Inaccurate Assumptions
Yaron: When it‘s all said and done, there’s a long list of tactical things that can be done better. And we can talk about it for hours. But the No. 1 thing is, there’s never enough communication. It’s communication with your clients, it’s communication with your employees, it’s communication with the government, it’s communication with your partners, it’s communication with your shareholders.
You need to have a very clear communication protocol. Meaning, accurate information is dramatically critical. Obviously, the media will pick up whatever they pick up. And obviously, if you don‘t supply detailed information as much as possible, people make assumptions. And some will be correct, but a lot will be incorrect and mislead the public.
And there‘s never enough communication, that is the No. 1 issue. No. 2 is just don’t get hacked again. Do everything you can. My thinking now is, ’I’m fighting X government.’ I’m not fighting a random hacker. I’m fighting people who serve just like me, but for countries that probably are not democracies.
And that‘s the level, and that’s the game we need to up to keep this country safe and other democracies safe.
Tap Into Authorities To Help Mitigate Impact Of Attack
Blount: The FBI was phenomenal throughout this event. And from that first phone call, within hours of seeing the ransomware come across that screen in the control room, all the way through the process, just complete professionals, very focused, very compassionate and understanding about what we went through. As I said before, that was an interesting thing that we saw a lot of.
What you would see on the news about Colonial and the FBI and its relationship wasn‘t accurate at all. We were highly collaborative with each other from the beginning. We gave them the Bitcoin wallet within a day when we received it. And that really allowed them to successfully recover what they did of the ransom from the perpetrators and effectively shut them down for the time being.
So again, just all our interaction with the federal government, including the FBI, was spot on throughout the event. People were very helpful, things that we needed done in order to help alleviate the lack of gasoline flow in the United States were done pretty quickly.
Waivers on trucking requirements, waivers on weight limits and things like that. Businesses that we‘re not involved in but help supply the supply chain throughout the United States. The government was highly, highly focused on helping us bring the system back.