Code Warriors (book about the NSA)

Code Warriors: NSA’s Codebreakers and the Secret Intelligence War Against the Soviet Union (Budiansky 2016) is an interesting book despite the challenge of getting information about the NSA. It is timely because of the recent Wikileaks release regarding the CIA’s efforts to get hold of messages on smartphones before they are encrypted.

I had no idea that Edward Snowden relied on a social attack to get information:

In May 2013, a twenty-nine-year-old computer security expert who had worked for three months as a $200,000-a-year contractor for the National Security Agency in Hawaii told his employer he needed to take a leave of absence for “a couple of weeks” to receive treatment for the epileptic condition he had recently been diagnosed with. On May 20, Edward J. Snowden boarded a flight to Hong Kong, carrying with him computer drives to which he had surreptitiously copied thousands of classified intelligence documents.

It was a move he had been secretly preparing for some time, having secured the job with the specific aim of gaining access to classified NSA material. (He was ultimately able to do so only by duping more than twenty coworkers into giving him their computer passwords, which he said he needed for his duties as a systems administrator; most of the colleagues whom he betrayed were subsequently fired.)

Can it really be as easy to get a password from an NSA employee with a top-secret clearance as it is to get one from a 93-year-old AOL user? Apparently the answer is “yes”!

Since “women doing jobs involving numbers” is newsworthy today…

More than 70 percent of the staff at Arlington Hall were civilians, and by the war’s end more than 90 percent of those were women. A similar balance of the sexes quickly took hold at the Navy’s signals intelligence headquarters, across the Potomac River. The Navy had a deep tradition of never permitting a situation to arise where an officer might have to take orders from a civilian, and insisted on putting all of its new hires in uniform. But with its establishment in summer 1942 of the WAVES—Women Accepted for Voluntary Emergency Service, which allowed women to serve in the Navy as officers and enlisted personnel—the service was also able to freely recruit women for codebreaking duty, and some 80 percent of its cryptanalysts by the war’s end were female.

It is doubtful that Uber will be resurrecting one particular NSA tradition:

A photograph in NSA’s historical files from this period showed the finalists in the annual Miss NSA beauty pageant, the contestants in evening gowns and each wearing a sash bearing the number of the section they worked in.

Contrary to Hollywood portrayals, the smartest people may also be the nicest…

Von Neumann had been an intellectual prodigy as a child, able to divide eight-digit numbers in his head at age six. Throughout his life he could effortlessly recite entire books verbatim after a single reading, and equally effortlessly provide a running translation in any number of languages. Years later, after he got to know him well, Goldstine tried to test von Neumann by asking him how Charles Dickens’s Tale of Two Cities begins. He was still going fifteen minutes later, without pause, when Goldstine finally stopped him. As a scientist, von Neumann had made seminal contributions to a bewildering array of fields, including game theory, quantum mechanics, economics, topology, and the theory of shock waves.

That day on the train platform the younger man, with some temerity, approached his world-famous colleague and introduced himself: Fortunately for me von Neumann was a warm, friendly person who did his best to make people feel relaxed in his presence.

The exciting age of code-breaking turns out to have mostly ended during World War II. The NSA funded a lot of powerful computers, but combinatorics worked against them.

The IBM 701, which IBM originally called the “Defense Calculator,” was much more of a number-cruncher designed to meet the needs of Los Alamos’s nuclear weapons designers, meteorologists at the U.S. Weather Bureau, and ballistics engineers at the Army’s ordnance labs. The new IBM machine that the company was now proposing was turning into the same bait and switch. In the summer of 1955, NSA agreed to provide IBM the $800,000 in funding it needed to develop the high-speed core memory that was to be the heart of the new “Stretch” computer. But meanwhile IBM also negotiated a deal with the Atomic Energy Commission to supply Los Alamos with a Stretch computer, too, for a fixed price of $4.3 million; then the company’s top management began to insist that whatever the final design, it had to be marketable to commercial users as well. “As usual the agency has a firm hold on the IBM leash and is being dragged down the street,” an NSA engineer assigned to keep tabs on the company’s work reported as the project progressed.

By the time the first machine was delivered to NSA in 1962, the price of the project had ballooned to $19 million, which did not include $1 million for supplies such as magnetic tapes and cartridges; $4.2 million for training, additional personnel, and software development; $196,045 for “installation”; and $765,000 a year in rental fees. IBM had resolved the problem of building a computer that could simultaneously serve scientific, cryptanalytic, and commercial customers by designing a flexible central processor, a high-speed arithmetic add-on unit for the AEC, and an add-on streaming unit for NSA, modeled on Abner’s “Swish” function. The special NSA add-on was called “Harvest,” which eventually became the name of the whole system; its official designation was the IBM 7950.

“There is not nearly enough energy in the universe to power the computer” that could test every setting of such a rotor machine, which had an effective cryptanalytic keyspace on the order of 1044. Even the “more modest undertaking” of recovering the setting of an individual message enciphered on such a machine whose internal configuration has already been recovered, which would involve testing about 1016 possibilities, would cost $2,000,000,000,000,000,000,000 per message for the electricity required to power any known or projected computing devices.55 (In 1998 a $250,000 machine built with 1,856 custom-made chips successfully carried out an exhaustive key search on the 56-bit key DES encryption system—a keyspace slightly greater than 1016—in two days. But a 128-bit key, with a keyspace of the order 1038, can be shown to resist an exhaustive search even by the most theoretically energy-efficient computer that the laws of physics permit.)

With the exception of a short-lived and still-classified 1979 breakthrough using Cray-1 supercomputers against Soviet codes, the modern age is all about sifting through massive volumes of plain-language communications, planting bugs to get plaintext prior to encryption, and recruiting spies.

For decades, standard histories of the air war in Korea attributed the sudden improvement in mid-1951 in the kill ratio achieved by American fighter pilots against Chinese MiG-15 jets to the arrival of the new and more capable American F-86. During the final year of the war U.S. fighters shot down 345 MiGs in air battles with a loss of only 18 F-86s, a kill ratio of 19 to 1. In fact, the real breakthrough had come from pulling together all of the signals intelligence sources in one center so that they could be rapidly correlated and passed on to fighters in the air. “The present top-heavy success of the F-86 against MiG-15s dates almost from the day of the inception of the new integrated [signals intelligence] service,” reported an officer involved in the operation. On one day, a visiting ASA colonel observed the system in action as 15 MiGs were shot down without a single loss by U.S. F-86s. With more enthusiasm than originality, the colonel said it was “just like shooting ducks in a rain barrel,” but it was an unmistakable demonstration of the incredible force multiplier that the signal interception and reporting system had provided: not a single one of the MiGs was tracked on U.S. radar during the course of the battle; all of the information passed to U.S. pilots had come from listening, in real time, to the communications of the enemy controllers and planes.39 An analysis of ground control traffic in June 1952 concluded that more than 90 percent of MiGs engaged in air operations over Korea were being flown by Russians.

The most famous penetration of the U.S. embassy was the Great Seal bug, also discovered during Kennan’s ambassadorship. Having requested a thorough sweep of his residence and the embassy, Kennan was sent a security team from Washington. To check for any voice-activated bugs, one of the technicians asked the ambassador to sit at his desk at Spaso House after hours and go through the motions of dictating a letter to his secretary. Kennan, with a certain touch of humor, chose to read from his 1936 cable in which he did nothing but recycle his predecessor’s dispatches from czarist Russia to show that nothing had changed under the Communist regime. Suddenly detecting a UHF signal coming from behind Kennan’s desk, the technician began hacking at the wall behind a wooden replica of the Great Seal of the United States that hung there. He then turned his hammer to the seal itself and pulled from behind the carved eagle’s beak a three-quarter-inch-diameter diaphragm-covered cylinder, attached to a short rod antenna.10 The seal had been presented as a gift from Russian schoolchildren to Ambassador Averell Harriman in 1945 and had hung there ever since. The American engineers who discovered it dubbed it “the Thing.” Its principle of operation was ingenious. The Thing was entirely passive, requiring no power supply and giving off no signal itself until it was illuminated by a microwave radio beam aimed from an adjoining building. As the diaphragm vibrated in and out in response to sound waves coming from the room, it minutely changed the shape, and thus the resonant frequency, of the cavity formed by the small cylinder. That slight distuning of a resonant frequency around 1800 MHz caused the strength of one of the harmonics of the incoming illuminating signal to fluctuate, producing a modulated radio signal of the same kind generated by an AM radio transmitter. The resulting signal could be picked up from a nearby location outside the building.

How did Americans find Soviet spies in their midst? “The science was settled” on the polygraph:

“The Director has repeatedly emphasized his firm conviction that the polygraph is more reliable and more protective of security than the background investigation,” his deputy for administration wrote in a 1956 memorandum that argued for periodically polygraphing existing civilian employees as well, to probe for “membership in subversive organizations,” “association with known or suspected subversives,” and unauthorized disclosure of classified information. … The trouble, aside from the abuse of privacy and due process inherent in the whole business, was that conscientious but perfectly innocent people tended to show a “deceptive” response in the standard polygraph examination, while pathological liars sailed through. In their zeal to clear the initial backlog of pending clearances, NSA scoured police departments and private detective agencies around the country to hire supposed polygraph experts to administer the tests, which took place in hastily erected soundproof rooms at the U Street building.

How well did it work?

Staff Sergeant Jack E. Dunlap was the holder of a Purple Heart and Bronze Star for “coolness under fire and sincere devotion to duty” in the Korean War. On July 22, 1963, he was found sitting dead in his car at his home near NSA headquarters, a length of radiator hose from the exhaust pipe running through the right front window and the engine idling. A month later his widow turned over to Army investigators a pile of classified documents from the attic of their home. She said her husband had told her that since mid-1960 he had been meeting a member of the Soviet embassy staff at rendezvous around Washington; in exchange for $40,000 he had supplied documents and hundreds of rolls of film containing pictures he had taken of classified material.

Dunlap’s motive was money pure and simple. He had walked into the Soviet embassy to offer his services, and the air attaché, Mikhail N. Kostyuk, had been all too happy to make the deal on behalf of the GRU.

Three months before his suicide, after applying for conversion to civilian employment at NSA, Dunlap admitted on a polygraph examination to having had “immoral sexual relations” with women and was moved to a “nonsensitive” position.

On an Army sergeant’s salary of $100 a week, he owned two Cadillacs, a baby-blue Jaguar sports car, a thirty-foot cabin cruiser, and a world-class racing hydroplane; he told coworkers a series of contradictory and patently fantastic stories to account for his sudden wealth, including that his father owned a large plantation in Louisiana, that he had made a successful investment in filling stations, that he owned land containing a valuable mineral used to make cosmetics, and that he had won the money as prizes in boat races. Nor did it exactly require a polygraph examination to uncover the fact that a married NSA employee who had begun dating an NSA secretary was possibly engaging in “immoral sexual relations.”

It seems to be tough to keep secrets in a country where people will do anything for cash. One of the big sellouts was “Ronald Pelton, an NSA cryptanalyst and Russian linguist who had worked on the agency’s most sensitive collection projects. … In exchange for $35,000 (he had asked for $400,000), he had arranged to meet with KGB officials at the Soviet embassy in Vienna on two occasions, submitting to lengthy interrogations. He told them about A Group’s success in breaking Soviet cipher machines, U.S. SIGINT satellites that targeted microwave telephone links throughout the Soviet Union, the U.S. embassy listening post, and an extremely secret Navy-NSA project that had deployed a submarine to install a tap on an undersea cable used by the Soviet Pacific Fleet’s headquarters in Vladivostok for its operational communications. The Soviets responded in 1981 by making an across-the-board change in their military encryption systems, bombarding the U.S. embassy with microwave jamming signals, and dispatching a salvage vessel to retrieve the cable tap from the floor of the Sea of Okhotsk.” Even after adjusting for CPI, that’s still only $93,502 in today’s mini-dollars, less than a tenth of what a Massachusetts child support plaintiff could get after having sex with someone earning $250,000 per year.

The author blames our entry into the Vietnam War, to some extent, on misinterpretation of signals intelligence:

Lyndon Johnson was fascinated by signals intelligence. Like no world leader since Winston Churchill, Johnson constantly demanded to see the actual translations of individual intercepted messages.

The South Vietnamese government was led by a corrupt regime that refused to hold elections and was made up largely of refugees from the North who had fled Ho’s Democratic Republic of Vietnam; nearly all were Catholics and former soldiers or police officers of the French colonial government, and to many of the indigenous and primarily Buddhist South Vietnamese, they represented nothing more than a continuation of the hated colonial rule.

Then, on the night of August 4, [1964] eighteen hours after the initial skirmish—“the darkest night I’d ever seen at sea,” in the words of one of the Maddox’s radar operators, in rough seas with a heavy chop, with a low overcast sky—the Maddox and a second destroyer, the Turner Joy, fired hundreds of rounds in a wild, four-hour-long zigzagging encounter in which their crews claimed to have seen gun flashes, searchlights, torpedo wakes, and radar and sonar contacts indicating attacks by multiple enemy boats that fired twenty-six torpedoes. A welter of confusing and contradictory evidence in the ensuing few hours cast doubt on the whole incident. For one thing, the entire known North Vietnamese force of twelve torpedo boats could have fired at most twenty-four torpedoes. The Turner Joy’s far more experienced sonarman had detected no torpedo contacts. Neither ship had suffered any visible damage. The radar contacts had appeared and disappeared at all points of the compass; not a single continuous track was followed. The white streaks in the water that some crewmen reported, Herrick quickly determined, had been nothing but the churning created by the American ships’ own wild evasive maneuvers, dodging nonexistent torpedoes. Air patrols reported they had not seen any enemy vessels or wakes.

It was at that moment, with orders for the retaliatory airstrikes pending, that McNamara decided to become his own intelligence analyst in earnest, seizing on two signals intelligence reports that had just come in: one was a Critic from Phu Bai issued the night of August 4 reporting POSS DRV NAVAL OPERATION PLANNED AGAINST THE DESOTO PATROL TONITE 04 AUG. The second, which arrived at the White House just two hours after Herrick’s message casting doubt on the whole business, appeared to be an after-action report from an unidentified North Vietnamese naval authority: SHOT DOWN TWO PLANES IN THE BATTLE AREA. WE HAD SACRIFICED TWO SHIPS AND ALL THE REST ARE OKAY. THE ENEMY SHIP COULD ALSO HAVE BEEN DAMAGED.

The information NSA provided on the August 2 attack had shown the agency at its nimble best: it had decoded messages in virtual real time, flashed an alert to the commander on the scene in time to give him tactical warning, and had sent the White House within hours crucial additional evidence that the attack might have been an unauthorized adventure by an overly aggressive North Vietnamese patrol. Its reporting on the August 4 phantom attack that precipitated America’s large-scale military intervention in Vietnam was another matter. McNamara undeniably seized and ran with the evidence he wanted to believe, but NSA’s inexperience in intelligence analysis and frantic efforts to supply the White House with information in the heat of crisis was what allowed him to do so. “Everybody was demanding the SIGINT; they wanted it quick, they didn’t want anybody to take any time to analyze it,” said Ray Cline, the CIA deputy director at the time.14 In fact, it had been a leap of complete guesswork on the part of the analyst at Phu Bai who issued the Critic on August 4 that a new attack on the Desoto patrol was about to take place: the actual intercepted North Vietnamese message, which McNamara did not see, referred only to unspecified “operations” by patrol boats that night. And as for the second message, the seemingly even more decisive after-action report, analysts at the NSA watch center later acknowledged that there had been a difference of opinion whether this referred to the earlier August 2 attack or a new incident.

NSA’s subsequent efforts to cover up its mistake turned its sin from venal to mortal; what began as an innocent lapse became an act of deliberate falsification as the agency systematically concealed the truth, issuing a series of summary reports over the following days that backed with obedient certainty the administration’s position even as the evidence pointed completely the other way. Within days NSA analysts were privately convinced that no second attack had occurred. The evidence was overwhelming: unlike on August 2, there had been no tracking reports transmitted by any of the North Vietnamese coastal radar stations on the night of August 4. At the very time the August 4 “attack” message was intercepted, other messages from North Vietnamese boats repeated orders to steer clear of the Desoto patrol altogether and left little doubt that the only “operation” taking place that night was a salvage operation to recover two boats damaged in the August 2 skirmish.

A classified, searingly honest accounting by NSA historian Robert J. Hanyok in 2001 found that in bolstering the administration’s version of events, NSA summary reports made use of only 15 of the relevant intercepts in its files, suppressing 122 others that all flatly contradicted the now “official” version of the August 4 events. Translations were altered; in one case two unrelated messages were combined to make them appear to have been from the same message; one of the NSA summary reports that did include a mention of signals relating to a North Vietnamese salvage operation obfuscated the timing to hide the fact that one of the recovered boats was being taken under tow at the very instant it was supposedly attacking the Maddox and Turner Joy

The book also chronicles our failures during the Vietnam War to anticipate enemy actions such as the Tet Offensive. The Vietnamese were good at intercepting our own signals:

[warnings based on intercepted American radio traffic] had been sent ahead of 90 percent of Rolling Thunder strikes that targeted the northeast quadrant of the country. The warnings were giving North Vietnamese MiG pilots time to scramble and be waiting—and add to the toll of more than nine hundred U.S. aircraft shot down during the three years of Rolling Thunder.

My summary: We can use unbreakable codes, but it is probably better not to try to do anything secret because eventually a spy will rat us out by revealing the plaintext.

More: Read Code Warriors: NSA’s Codebreakers and the Secret Intelligence War Against the Soviet Union