On 23 October 1642, two armies clashed in the English fields north of Oxford. One belonged to King Charles, the other to Parliament. The battle of Edge Hill was the first in the bloody English civil war. The fight was messy. Parliament forces fired their cannons; the royalists led a cavalry charge; inexperienced soldiers on both sides ran away. Some were keener on looting than defeating the enemy. Neither side really won. The war dragged on for another four years.
Two centuries later, on 21 July 1861, another skirmish took place. This time the Union Army was fighting the Confederates, in the first major land encounter of the American civil war. The location was Bull Run, a tributary of the Potomac in Virginia. The Northern forces expected a quick victory. Instead, the Confederate army launched a ferocious counter-attack. Brigadier General Irvin McDowell and his Union soldiers fled in the direction of Washington DC. The battle revealed there would be no easy knockout.
Many years later, American and British spies were mulling over names for two top-secret programs. Their new battles were electronic rather than territorial. It was the growing practice of encryption that was their enemy. The names they chose for their new battles were BULLRUN and EDGEHILL. Did the emphasis on civil wars have a special significance? Certainly, the spies were now about to declare war on their own domestic corporations.
Cryptography was first used in ancient Egypt and Mesopotamia. The aim, then as now, was to protect secrets. During the first and second world wars, military cryptography and cryptanalysis – the ability to decrypt coded information on enemy movements – played a key role. But it was largely the preserve of embattled nation states. Typically, those interested in codes were the British mathematicians working in secret to defeat the Nazis at wartime Bletchley Park, and the Soviets subsequently.
By the 1970s, however, encryption software such as Pretty Good Privacy (or PGP) was available to private individuals, as well as commercial organisations. Encryption thus posed an obvious challenge to western intelligence agencies, anxious to continue reading their adversaries’ messages. The Clinton administration responded by trying to insert a back door into commercial encryption systems. This would let the NSA in. The attempt met with political defeat. A bipartisan group of senators and tech executives argued this would be bad for the Valley. Plus it would violate the fourth amendment.
By 2000, as encryption was increasingly employed by service providers and individuals in everyday online communications, the NSA was spending billions of dollars finding ways to get round it. Its encrypted targets included web searches, internet chats, emails, personal data, phone calls, even banking and medical records. The challenge was to convert ‘ciphertext’ – what encrypted data looks like in its raw form: that is, mathematical nonsense – into ‘cleartext’.
In 2010 a British GCHQ document warned that over time the allies’ capacities could degrade as ‘information flows change’ and ‘widespread encryption becomes more commonplace’.
At first, the eavesdroppers seemed to face defeat, or at least stalemate. One of the leaked documents from 2006 shows that, at that date, the agency had only broken the encryption of one foreign state’s nuclear ministry, a single travel reservation system, and three foreign airlines.
It was not until 2010 that the NSA made dramatic progress, thanks to BULLRUN and EDGEHILL. It used super-computers to crack algorithms, encryption’s basic building blocks. (Algorithms generate the key which can encrypt and decrypt messages. The longer the key, the better the encryption.)
But most importantly, the Snowden files show that the NSA cheated. Despite the political defeat on back doors, the agency simply went ahead and secretly introduced ‘trapdoors’ into commercial encryption software used by millions of people. It collaborated with developers and technology companies to insert deliberate, exploitable flaws into both hardware and software. Sometimes this co-operation was voluntary; sometimes bullying legal orders enforced it. The NSA, if necessary, would steal encryption keys, almost certainly by hacking into servers where the keys were kept.
Unsurprisingly, the NSA and GCHQ were keen to keep details of these most shadowy of programs under wraps. A 2010 document from Snowden shows just how restricted knowledge was of BULLRUN – and how effective it was. The PowerPoint was used to brief British staff in Cheltenham on the NSA’s recent breakthroughs, as a result of which decrypted internet traffic was suddenly streaming across the desks of analysts.
It says: ‘For the past decade the NSA has led an aggressive, multi-pronged effort to break widely used internet encryption technologies. Cryptanalytic capabilities are now coming online. Vast amount of encrypted internet data which up to till now have been discarded are now exploitable.’
The slide says ‘major new processing systems’ must be put in place ‘to capitalise on this opportunity’. GCHQ staff previously kept in the dark about BULLRUN were astonished by the NSA’s formidable new capabilities. One internal British memo reports: ‘Those not already briefed were gobsmacked.’
Snowden’s first batch of published files did not disclose details of which companies work with the NSA on counter-encryption. Or which commercial products may have back doors. But the files do give some idea of BULLRUN’s massive dimensions. A budget report for the entire US intelligence community says that 2013 funding for the program was $254.9m. (PRISM, by contrast, costs just $20m annually.) Since 2009, the agency has splashed more than $800m on ‘SIGINT [signals intelligence] enabling’. The program ‘actively engages US and foreign IT industries to covertly influence and/or overtly leverage their commercial products’ designs’, the report says.
The joy of the program, the NSA says, is that ordinary citizens have no idea that their everyday encrypted communications are now hackable. When the NSA inserts ‘design changes’ into commercial encryption systems, the 178-page report for the fiscal year notes, ‘To the consumer and other adversaries… the systems’ security remains intact.’
James Clapper, the director of national intelligence, stresses the importance of crypto. ‘We are investing in groundbreaking cryptanalytic capabilities to defeat adversarial cryptography and exploit internet traffic,’ he writes.
The agency is not lacking in ambition. The files show the NSA is breaking the encryption systems of 4G phones. It targets online protocols used in secure banking and business transactions, such as HTTPS and Secure Sockets Layer (SSL). It wants to ‘shape’ the worldwide encryption marketplace. Soon it expects to get access to ‘data flowing through a hub for a major communications provider’ and to a ‘major internet peer-to-peer voice and text communications system’. That sounds like Skype.
Meanwhile, the British were pressing on with their own parallel EDGEHILL project. One file shows that the British spies have succeeded in breaking into three internet providers and 30 types of Virtual Private Networks (VPN) used by businesses to access their systems remotely. By 2015 it hoped to have penetrated 15 internet companies and 300 VPNs.
The spy agencies insist that their ability to defeat encryption is essential to their mission, and that without it they would be unable to track terrorists or gather valuable foreign intelligence. The problem, as the New York Times points out, is that the NSA’s anti-encryption stealth campaign may have disastrous unwanted consequences.
By inserting deliberate weaknesses into encryption systems, the agency has made those systems exploitable. Not just by government agencies, who may be acting with good intentions, but by anybody who can get hold of encryption keys – such as hackers or hostile intelligence agencies. Paradoxically, in its quest to make Americans more secure, the NSA has made American communications less secure; it has undermined the safety of the entire internet.