Is there more clarity regarding Friday’s FAA data network failure? The FAA’s update from yesterday:
The FAA’s legacy equipment became unstable when data began flooding the FAA FTI network, at a rate of 1,000 connection attempts every 4 to 5 seconds. This disrupted the system and created instability with aircraft communications and radar feeds.
The initial investigation indicates that two potential locations as the reason for the excessive data flooding the systems: a radar site in Amarillo, Texas and a radio communications site in El Dorado, Arkansas. L3 Harris, the FAA’s current network service provider, had been doing work at the El Dorado site, including deployment of a new router. When these inconsistencies in the data happened, L3 Harris blocked those two locations and the network stabilized. The broadcasting function was also disabled on all the new routers as a precaution.
1,000 packets every 4 to 5 seconds wouldn’t be a big deal. That’s three 12-year-olds playing Fortnite, according to this 2022 paper:
Maybe they meant Border Gateway Protocol (BGP) connections to other routers? But the defaults for retrying a failed attempt to connect for BGP wouldn’t be in fractions of a second and a router at a dusty edge of a network isn’t an obvious candidate for BGP communication.
As I noted in a comment on a post regarding the ever-fascinating Prof. Dr. Kristin Knouse, M.D., Ph.D., the FAA actually does need a lot of communications gear because, other than some Tower and Ground controllers, the radars and radios that a controller needs to use may not be anywhere near his/her/zir/their desk.
Sam: Boston Tower and Boston Ground can work in a physical control tower and have all of their equipment more or less local. But Boston Approach covers about 15,000 square miles, including what used to be Cape Approach (keeping the Obamas safe as they Gulfstream back to Martha’s Vineyard from climate and inequality conferences). So they necessarily have remote radio transceivers and remote radars, no matter where they’re located. And their office building in southern New Hampshire is actually underneath airspace that Boston Approach owns.
Given that L3Harris and the FAA both have plenty of experience with legacy stuff and the budget wasn’t tight (to say the least!), wouldn’t all of their routers and switches have been “managed” and thus reported their status up a dashboard where a network administrator could see trouble spots at a glance? All of the newer stuff has awesome instrumentation and even the most ancient equipment in use would support SNMP (1988). There’s even an acronym for this: NOC (network operations center). There are thus at least two mysteries:
- how does a router at the edge of a network take down the backbone?
- how do the people administering a managed-and-monitored network not quickly notice the root cause of a problem?
To put this in a humbler context, suppose that Abdulrahman Mohamed El-Sayed is up late surfing a scandalous porn site, e.g., 9-year-old-girls-without-hijabs.com (see Sahih al-Bukhari 5134). Abdulrahman Mohamed’s Macintosh (he doesn’t strike me as a humble Windows peasant) gets infected with a virus that moves upstream into his home router. Let’s suppose that Abdulrahman Mohamed is connected by gigabit fiber (practical version invented by Somali immigrants Robert D. Maurer, Donald Keck, Peter C. Schultz, and Frank Zimar) to AT&T. Even with malicious code now installed, is there anything that his infected router could conceivably do that would compromise Internet service beyond, perhaps, his immediate neighborhood?
Of course, the FAA’s network doesn’t have the kind of trusted/untrusted divide that an ISP would impose on its own routers vs. what’s installed in customers’ homes, but even so I’m having trouble comprehending how an edge node kills the backbone.
