Venona project

From Wikipedia, the free encyclopedia

The Venona project was a long-running and highly secret collaboration between intelligence agencies of the United States and United Kingdom that involved the cryptanalysis of messages sent by several intelligence agencies of the Soviet Union, mostly during World War II. There were known to be at least 13 code words for this effort used by the U.S. and UK. "Venona" was the last one used. It has no known meaning. (In the decrypted documents issued from the National Security Agency, 'VENONA' is written in full capitals; authors writing on the subject generally capitalize only the first letter.)

In the early years of the Cold War, Venona would be an important source of information on Soviet intelligence activity for the Western powers. Although unknown to the public, and even to presidents Franklin D. Roosevelt and Harry Truman, the closely guarded program was of critical importance behind many famous events of the early Cold War, such as the Rosenberg spying case and the defections of Donald Maclean and Guy Burgess.

Most of the messages which would later prove to be decipherable were intercepted between 1942 and 1945, and they were decrypted beginning in 1946 and continuing until 1980, when Venona was cancelled.

Contents 1 Background 2 Breakthrough 3 Results 4 Significance 5 Bearing of Venona on particular cases 5.1 Julius and Ethel Rosenberg 5.2 Alger Hiss and Harry Dexter White 5.3 Donald Maclean and Guy Burgess 5.4 Soviet espionage in Australia 6 Usability in prosecutions 7 Public disclosure 8 Critical views 9 See also 10 Notes 11 References and further reading 11.1 Books 11.2 Online sources

[edit] Background

The Venona Project was initiated in 1943, under orders from the deputy Chief of Military Intelligence (G-2), Carter W. Clarke.^[1] Clarke distrusted Joseph Stalin, and feared that the Soviet Union would sign a separate peace with the Third Reich, allowing Germany to focus its military forces against Great Britain and the United States. Code-breakers of the U.S. Army's Signal Intelligence Service (commonly called Arlington Hall) analyzed encrypted high-level Soviet diplomatic intelligence message intercepted in large volumes during and immediately after World War II by American, British and Australian listening posts.^[2]

This traffic, some of which was encrypted with a one-time pad system, was stored and analyzed in relative secrecy by hundreds of cryptanalysts over a 40-year period starting in the early 1940s. Due to a serious blunder on the part of the Soviets—reusing pages of some of the one-time pads in other pads, which were then used for other messages—some of this traffic was vulnerable to cryptanalysis.

[edit] Breakthrough

The Soviet systems in general used a code to convert words and letters into numbers, to which additive keys (from one-time pads) were added, encrypting the content. When used correctly, one-time pad encryption is provably unbreakable. Cryptanalysis by American and British code-breakers revealed that some of the one-time pad material had incorrectly been reused by the Soviets (specifically, entire pages, although not complete books), which allowed decryption (sometimes only partial) of a small part of the traffic.

Generating the one-time pads was a slow and labor-intensive process, and the outbreak of war with Germany in June 1941 caused a sudden increase in the need for coded messages. It is probable that the Soviet code generators started duplicating cipher pages in order to keep up with demand.

It was Arlington Hall's Lt. Richard Hallock, working on Soviet "Trade" traffic (so called because these messages dealt with Soviet trade issues), who first discovered that the Soviets were reusing pages. Hallock and his colleagues (including Genevieve Feinstein, Cecil Phillips, Frank Lewis, Frank Wanat, and Lucille Campbell) went on to break into a significant amount of Trade traffic, recovering many one-time pad additive key tables in the process.

A young Meredith Gardner then used this material to break in to what turned out to be NKVD (and later GRU) traffic, by reconstructing the code used to convert text to numbers. Samuel Chew and Cecil Phillips also made valuable contributions. On 20 December 1946, Gardner made the first break into the code, revealing the existence of Soviet espionage in the Manhattan Project.^[3] Venona messages also indicated that Soviet spies worked in Washington in the State Department, Treasury, Office of Strategic Services, and even the White House. Very slowly, using assorted techniques ranging from traffic analysis to defector information, more of the messages were decrypted.

Claims have been made that information from physical theft of code books (a partially burned one was recovered by the Finns) to bugging embassy rooms in which text was entered into encrypting devices (analyzing the keystrokes by listening to them being punched in), contributed to recovering much of the plaintext. These latter claims are less than fully supported in the open literature.

One significant aid (mentioned by the NSA) in the early stages may have been work done in cooperation between the Japanese and Finnish cryptanalysis organizations; when the Americans broke into Japanese codes during World War II, they gained access to this information. There are also reports that copies of signals purloined from Soviet offices by the Federal Bureau of Investigation (FBI) were helpful in the cryptanalysis.

[edit] Results

The NSA reported that, according to the serial numbers of the Venona cables, thousands were sent, but only a fraction were available to the cryptanalysts. Approximately 2,200 of the messages were decrypted and translated; some 50 percent of the 1943 GRU-Naval Washington to Moscow messages were broken, but none for any other year, although several thousand were sent between 1941 and 1945. The decryption rate of the NKVD cables was:

• 1942 1.8%
• 1943 15.0%
• 1944 49.0%
• 1945 1.5%

Out of some hundreds of thousands of intercepted encrypted texts, it is claimed that under 3,000 have been partially or wholly decrypted. All of the duplicate one-time pad pages were produced in 1942, and almost all of them had been used by the end of 1945, with a few being used as late as 1948. After this, Soviet message traffic reverted to completely unreadable.^[4]

The existence of Venona decryptions became known to the Soviets within a few years of the first breaks. It is not clear whether the Soviets knew how much of the message traffic, or which messages, had been successfully decrypted. At least one Soviet penetration agent, British Secret Intelligence Service Representative to the U.S., Kim Philby, was told about the project in 1949, as part of his job as liaison between British and U.S. intelligence. Since all of the duplicate one-time pad pages had been used by this time, the Soviets apparently did not make any changes to their cryptographic procedures after they learned of Venona. However, this information did allow them to alert those of their agents who might be at risk of exposure due to the decryptions.

[edit] Significance

The decrypted messages gave important insights into Soviet behavior in the period during which duplicate one-time pads were used. With the first break into the code, Venona revealed the existence of Soviet espionage^[5] at Los Alamos National Laboratories.^[6] Identities soon emerged of American, Canadian, Australian, and British spies in service to the Soviet government, including Klaus Fuchs, Alan Nunn May and Donald Maclean, a member of the Cambridge Five spy ring. Others worked in Washington in the State Department, Treasury, Office of Strategic Services,^[7] and even the White House.

The decrypts show that the U.S. and other nations were targeted in major espionage campaigns by the Soviet Union as early as 1942. Among those identified are Julius and Ethel Rosenberg; Alger Hiss; Harry Dexter White,^[8] the second-highest official in the Treasury Department; Lauchlin Currie,^[9] a personal aide to Franklin Roosevelt; and Maurice Halperin,^[10] a section head in the Office of Strategic Services.

The identification of individuals mentioned in Venona transcripts is sometimes problematic, since people with a "covert relationship" with Soviet intelligence are referenced by code names.^[11] Further complicating matters is the fact that the same person sometimes had different code names at different times, and the same code name was sometimes reused for different individuals. In some cases, notably that of Alger Hiss, the matching of a Venona code name to an individual is disputed. In many other cases, a Venona code name has not yet been linked to any person. According to authors John Earl Haynes and Harvey Klehr, the Venona transcripts identify approximately 349 Americans who they claim had a covert relationship with Soviet intelligence, though less than half of these have been matched to real-name identities.^[12]

The Office of Strategic Services, the predecessor to the CIA, housed at one time or another between fifteen and twenty Soviet spies.^[13] Duncan Lee, Donald Wheeler, Jane Foster Zlatowski, and Maurice Halperin passed information to Moscow. The War Production Board, the Board of Economic Warfare, the Office of the Coordinator of Inter-American Affairs and the Office of War Information, included at least half a dozen Soviet sources each among their employees. In the opinion of some, almost every American military and diplomatic agency of any importance was compromised to some extent by Soviet espionage.^[14]

Some scholars and journalists dispute the claims by Haynes, Klehr, and others concerning the precision of the matching of code names to actual persons. Also contested is the implication that all 349 persons identified had a witting covert relationship with Soviet intelligence; it is argued that in some cases the individual may have been an unwitting information source or a prospect for future recruitment by Soviet intelligence. See "Critical views" below.

[edit] Bearing of Venona on particular cases

Venona has added information—some of it unequivocal, some of it ambiguous—to several espionage cases. Some known spies, including Theodore Hall, were neither prosecuted nor publicly implicated, because the Venona evidence against them was not made public.

[edit] Julius and Ethel Rosenberg

Main article: Julius and Ethel Rosenberg

Venona has added significant information to the case of Julius and Ethel Rosenberg, making it clear that Julius was guilty of espionage, but also showing that Ethel was probably no more than an accomplice, if that. Additionally, Venona and other recent information has shown that while the content of Julius' atomic espionage was not as vital as was alleged at the time of his espionage activities, in other fields it was extensive. The information Rosenberg passed to the Soviets concerned the proximity fuze, design and production information on the Lockheed P-80 jet fighter, and thousands of classified reports from Emerson Radio. The Venona evidence indicates that it was unidentified sources codenamed "Quantum" and "Pers" who facilitated transfer of nuclear weapons technology to the Soviet Union from positions within the Manhattan Project.

[edit] Alger Hiss and Harry Dexter White

Main articles: Alger Hiss, Harry Dexter White

According to the Moynihan Commission on Government Secrecy, the complicity of both Alger Hiss and Harry Dexter White is settled by Venona.^[15][16] In his 1998 book, Senator Moynihan expresses certainty about Hiss's identification by Venona as a Soviet spy, writing "Hiss was indeed a Soviet agent and appears to have been regarded by Moscow as its most important."^[17] However, some current authors consider the Venona evidence on Hiss to be inconclusive.^[18]

[edit] Donald Maclean and Guy Burgess

When Kim Philby learned of Venona in 1949, he obtained advance warning that his fellow Soviet spies Donald Maclean and Guy Burgess were in danger of being exposed. The FBI told Philby about an agent code named Homer, whose 1945 message to Moscow had been decoded. As it had been sent from New York and had its origins in the British Embassy in Washington, Philby deduced that the sender was Donald Maclean, now resident in London (Philby had not known Maclean's code name). By early 1951, Philby knew that US Intelligence would soon also conclude that Maclean was the sender, and he advised that Maclean be called in. This led to Maclean and Guy Burgess' famous flight to Russia in May, 1951.^[19]

[edit] Soviet espionage in Australia

In addition to the British and Americans, Venona intercepts were collected by the Australians at a remote base in the Australian Outback. However, the Russians were not aware of this base even as late as 1950.^[20] The founding of the Australian Security Intelligence Organisation by Labor Prime Minister Ben Chifley was considered highly controversial within Chifley's own party. Until then, the left-leaning Australian Labor Party had been hostile to domestic intelligence agencies on civil liberties grounds, and a Labor government actually founding one was a surprising about face. Venona material has now made it clear that Chifley was motivated by evidence that Soviet agents were operating in Australia. Investigation had revealed that Wally Clayton (codenamed KLOD), a Soviet agent within the Communist Party of Australia, was forming an underground network within the CPA so that the party could continue to operate if it was banned.

[edit] Usability in prosecutions

On February 1, 1956, Alan H. Belmont prepared an FBI memorandum on the significance of the Venona project and the prospects of using decryptions in prosecution. It considered that, although decryptions might corroborate the testimony of Elizabeth Bentley and enable successful prosecution of such suspects as Judith Coplon and the Perlo and Silvermaster groups, a careful study of all factors compelled the conclusion it would not be in the best interests of the United States to use Venona project information for prosecution.^[21]

The Memo gives a number of reasons why it was uncertain whether or not the Venona project information should be revealed and admitted into evidence.

A major hurdle was a question of law. A defense attorney might immediately move to dismiss the evidence as hearsay, since neither the Soviet official who sent the message, nor the one who received it was available to testify. The FBI reasoned that decrypts probably could have been introduced, on an exception to the hearsay rule, based on the expert testimony of cryptographers.

In addition, according to Belmont, "the fragmentary nature of the messages and the extensive use of cover names therein make positive identification of the subjects difficult."^[21] Once an individual had been considered for recruitment as an agent by the Soviets, sufficient background data on him was sent to Moscow. Cover names were used not only for Soviet agents but other people as well. President Roosevelt, for example, was called "Kapitan" (Captain), and Los Alamos the "Reservation". Cover names also were frequently changed, and a cover name might actually apply to two different people, depending on the date it was used. Several subjects, notably Alger Hiss, Harry Dexter White, Maurice Halperin, and Lauchlin Currie, denied the accusations in open Congressional Hearings based on information from sources other than Venona. Assumptions made by cryptographers, questionable interpretations and translations placed reliance upon the expert testimony of cryptographers, and the entire case would be circumstantial.

Defense attorneys also would probably request to examine messages which cryptographers were unsuccessful in breaking and not in evidence, on the belief that such messages, if decoded, could exonerate their clients. The FBI determined that this would lead to the exposure of Government techniques and practices in the cryptography field to unauthorized persons, compromise the Government's efforts in communications intelligence, and hinder other pending investigations.

Before any messages could be used in court they would have to be declassified. Approval would have to come from several layers of bureaucracy, and probably the President, as well as notification to British counterparts working on the same problem. In an election year, the Bureau feared it would be caught between two sides of a venomous political dispute.

[edit] Public disclosure

For much of its history, knowledge of Venona was restricted even from the highest levels of government. Senior Army officers, in consultation with the FBI and CIA made the decision to restrict knowledge of Venona within the government (even the CIA was not made an active partner until 1952). Army Chief of Staff Omar Bradley, concerned about the White House's history of leaking sensitive information, decided to deny President Truman direct knowledge of the project. The president received the substance of the material only through FBI, Justice Department and CIA reports on counterintelligence and intelligence matters. He was not told the material came from decoded Soviet ciphers. To some degree this secrecy was counter-productive; Truman was distrustful of FBI head J. Edgar Hoover, and suspected the reports were exaggerated for political purposes.

Some of the earliest detailed public knowledge that Soviet code messages from WWII had been broken came with the release of Robert Lamphere's book, The FBI-KGB War, in 1986. Lamphere had been the FBI liaison to the code-breaking activity, had considerable knowledge of Venona and the counter-intelligence work that resulted from it.

Many inside the NSA had argued internally that the time had come to publicly release the details of the Venona project, but it was not until 1995 that the bipartisan Commission on Government Secrecy, with Senator Moynihan as chairman, released the Venona project materials. Moynihan wrote:

"[The] secrecy system has systematically denied American historians access to the records of American history. Of late we find ourselves relying on archives of the former Soviet Union in Moscow to resolve questions of what was going on in Washington at mid-century. [...] the Venona intercepts contained overwhelming proof of the activities of Soviet spy networks in America, complete with names, dates, places, and deeds."^[22]

One of the considerations in releasing Venona translations was the privacy interests of the individuals mentioned, referenced, or identified in the translations. Some names were not released because to do so would constitute an invasion of privacy.^[23] However, in at least one case, independent researchers identified one of the subjects whose name had been obscured by the NSA.

The dearth of reliable information available to the public—or even to the President and Congress—may have helped to polarize debates of the 1950s over the extent and danger of Soviet espionage in the United States. Anti-Communists suspected that many spies remained at large, perhaps including some that were known to the government. Those who criticized the governmental and non-governmental efforts to root out and expose communists felt that these efforts were an overreaction (in addition to other reservations about McCarthyism). Public access—or broader governmental access—to the Venona evidence would certainly have affected this debate, as it is affecting the retrospective debate among historians and others now. As the Moynihan Commission wrote in its final report:

"A balanced history of this period is now beginning to appear; the Venona messages will surely supply a great cache of facts to bring the matter to some closure. But at the time, the American Government, much less the American public, was confronted with possibilities and charges, at once baffling and terrifying."

[edit] Critical views

Although widely accepted by historians and academics, the relevance, accuracy, and even the authenticity of Venona decrypts have been questioned by some. Many critics of the released Venona papers claim the material to be unverifiable, with some, such as William Kunstler, going so far as to claim that the NSA had forged Venona material in its entirety in order to discredit the reputation of the Communist Party of the United States of America and its members.^[24] Research in Soviet Archives has added to the corroboration of some Venona material, including the identities of many codenamed individuals.^[25]

Some remain skeptical of both the substance and the prevailing interpretations made since the release of the Venona material. Victor Navasky, editor and publisher of The Nation, has written several editorials highly critical of John Earl Haynes' and Harvey Klehr's interpretation of recent work on the subject of Soviet espionage. Navasky claims that the Venona material is being used to “distort … our understanding of the cold war” and that the files are potential “time bombs of misinformation”.^[26] Commenting on the list of 349 Americans identified by Venona that Haynes and Klehr published in an appendix to Venona: Decoding Soviet Espionage in America (see above), Navasky wrote: "The reader is left with the implication--unfair and unproven--that every name on the list was involved in espionage, and as a result, otherwise careful historians and mainstream journalists now routinely refer to Venona as proof that many hundreds of Americans were part of the red spy network."^[26] Navasky goes further in his defense of the listed people and has claimed that a great deal of the so called espionage that went on was nothing more than “exchanges of information among people of good will” and that “most of these exchanges were innocent and were within the law”.^[27]

According to Ellen Schrecker, "Because they offer insights into the world of the secret police on both sides of the Iron Curtain, it is tempting to treat the FBI and Venona materials less critically than documents from more accessible sources. But there are too many gaps in the record to use these materials with complete confidence."^[28]

Schrecker agrees that the documents have genuinely established the guilt of many prominent figures, but is still critical of the hardline interpretation of the materials by scholars such as Haynes, arguing that "...complexity, nuance, and a willingness to see the world in other than black and white seem alien to Haynes' view of history."^[29]

Writing about Alger Hiss, Hiss's lawyer John Lowenthal criticized the accuracy and methodology of the Venona analysts, charging that they employed false premises and flawed comparative logic to reach the desired conclusion that Alger Hiss was the spy "Ales". Lowenthal states this conclusion was psychologically and politically motivated but factually wrong.^[30]

Nigel West on the other hand, expressed confidence in the decrypts: "Venona remain[s] an irrefutable resource, far more reliable than the mercurial recollections of KGB defectors and the dubious conclusions drawn by paranoid analysts mesmerized by Machiavellian plots."^[31]

[edit] See also

• History of Soviet and Russian espionage in the United States
• List of Americans in the Venona papers

[edit] Notes

[edit] References and further reading