FAA Air Traffic Ground Stop Investigation: HPE Juniper Routers, Cisco Routers, And The Aftermath Of A Telecom Network Upgrade
Here’s a look at how a telecom network upgrade involving a move from 20-year-old Cisco routers to modern HPE Juniper routers led to the nationwide air traffic ground stop last Friday.
The Federal Aviation Administration (FAA) nationwide air traffic ground stop last Friday, which resulted from a telecom network upgrade involving a move from 20-year-old Cisco routers to modern HPE Juniper routers, is now zeroing in on older Cisco equipment, said U.S. Transportation Secretary Sean Duffy in an interview on CNBC.
“What happened was as we did one of those switchovers in El Dorado, Ark., that server or that router started to flood our network,” said Duffy Monday of the network upgrade from old copper wires to fiber. “Well, all of the work I had done to put in new equipment, those routers in my system didn’t go down. But we also use a lot of 2001 Cisco servers that went out of commission in 2006. It is those old servers that were flooded and started to give us inconsistency in our network.”
At this point, Duffy said, the Cisco server in question has been taken to federal contractor and technology solution provider L3Harris Technologies’ Florida laboratory as part of an investigation into the incident with FAA staff on site.
CRN reached out to Cisco for comment but had not heard back at press time.
As a result of the network issue, Duffy said the decision was made to “shut down the airspace” until the U.S. could “isolate where the flooding of the network was coming from, remedy it, get the systems working again and get air travel back to normal,” said Duffy.
In a press conference on Saturday, Federal Aviation Administrator Bryan Bedford told reporters that the FAA spends a billion dollars a year maintaining and sustaining its 1980s TDMI telco network, which he referred to as a 1G system. “We are constantly replacing 2,000 Cisco 600 routers with Juniper routers,” he said. “So, this is ordinary course-of-business stuff.”
Those Cisco 600 routers were manufactured between 2001 and 2006.
The upgrade from Cisco routers to HPE Juniper equipment was being conducted by Melbourne, Fla.-based L3Harris, which is under contract with the U.S. government, runs the telecom network.
L3Harris, which has been working with the FAA for over 20 years, is now under scrutiny on whether there was a misconfiguration issue with regard to the HPE Juniper router as part of the network upgrade.
In a November 14 statement on the incident, L3Harris said: “L3Harris has been modernizing the communications infrastructure supporting the National Airspace System in close coordination with the FAA. With safety and reliability as our highest priorities, L3Harris, together with its telco partners and the FAA, has completed more than 70 percent of the planned airspace communications conversion from legacy copper wires to a fiber optic system. We are supporting the FAA in its investigation to understand the conditions that led to the Nov. 13 ground stop of our nation’s air travel. L3Harris takes the nationwide ground stop and its impact on travelers, airlines and air traffic operations seriously.”
CRN reached out to L3Harris Technologies for further comment but had not heard back at press time.
Duffy said on ABC’s “This Week” with co-anchor Martha Raddatz that the HPE Juniper router in question is also at the L3Harris laboratory in Florida.
In a statement to CRN, HPE said: "Based on what we know and have been told by others, our equipment was working as configured. The FAA is assessing the configuration and performance of all infrastructure involved to determine the cause."
A source with knowledge of the incident said it appears the broadcasting feature on the HPE Juniper router should have been turned off because it was communicating with 20-year-old Cisco routers. “That capability should not have been turned on for a large legacy system,” said the source.
Duffy said he has asked FBI Director Kash Patel to send agents to the L3Harris lab to also participate in the review of the server to ensure it was not a cyberattack.
“Everybody is congregated in Florida, and we are going to do an analysis on the server and the methods that were used on its installation,” said Duffy.
Duffy said the U.S. is taking all the “necessary steps” to “set the American people’s mind at ease” that this was a “router or installation issue” as opposed to a “nefarious” attack.
Duffy has urged Congress to approve $50 billion in funding to upgrade the current network infrastructure.
“Every institution, not just in America but around the world, has left this old copper wire analog network, and they did it decades ago,” said Duffy. “The FAA still uses this because there has just been an historic under-investment in aviation.”
‘No Playbook’ For Legacy Systems Upgrade
Bob Venero, CEO of Fort Lauderdale, Fla.-based Future Tech Enterprise, which is a supplier to federal systems integrators, applauded both the FAA and L3Harris for responding quickly and appropriately to keep the American flying public safe.
“I applaud the FAA in keeping American citizens safe by taking a conservative approach and shutting down flights in wake of the network issue,” he said. “The most important thing is to protect American lives, and they did that.”
Venero said there is “no playbook” for upgrading decades-old legacy systems like this.
“In a case like this, you are doing as much as you can to build new infrastructure, setting it up, testing it in a lab and making it work,” he said. “There are going to be configuration issues that come up in an IT environment like this. Kudos to L3Harris Technologies for being able to identify where the challenge came from with the network.”
Venero said it is an “extremely difficult process” to move from a decades-old legacy system to modern infrastructure with modern tools and processes.
“You are always going to run into certain glitches,” he said. “The fact is the FAA became aware of it, reacted to it and isolated where it came from. Now they have pulled the equipment associated with it and have brought it to L3Harris headquarters. Now they are going to validate that it was nothing that was nefarious. Now L3Harris has a blueprint to continue its process to continue the upgrade to a more modern and scalable system.”
Venero’s advice to Congress: “Give L3Harris the money and tools they need to complete the migration to a more modern IT network infrastructure.”
The CEO for a Solution Provider 500 company, who did not want to be identified, said he was “appalled” and “shocked” that the U.S. government is relying on decades-old infrastructure for the air traffic control system.
“Think about businesses who are constantly modernizing their infrastructure to reduce technical debt, avoiding exposure to the business and shareholders,” said the CEO. “It’s surprising that we wouldn’t think the same thing when people’s lives are on the line with the network for air traffic control.”
Ultimately, refusing to upgrade is a threat to the “welfare and safety” of the hundreds of thousands of people that fly each day, said the CEO.
The CEO said it is “ironic” that while the nation is running a decades-old legacy network, it requires government contractors to meet stringent security and IT requirements. “If the government requires proven state-of-the-art technology, why wouldn’t the government require the same thing of themselves,” said the CEO. “I can’t believe this.”
Additional reporting by Gina Narcisi