In 1984, a man built a trap that nobody has ever been able to fully dismantle, even after he destroyed the source code himself. Ken Thompson, co-creator of Unix, had just received the Turing Award, computing’s highest honor. In his acceptance lecture, he didn’t talk about Unix. He talked about a trick he had built years earlier, one he judged important enough to reveal publicly rather than let it sit quietly.
What Thompson described that day wasn’t a technical flaw. It was an idea: what if, instead of attacking a program, you attacked the trust placed in it? That shift deserves attention, because it opened an entirely new category of attack, one that no longer targets what the code says, but what the code is supposed to guarantee.
The 1984 trick
Thompson modified Unix’s C compiler, the program that turns human-readable source code into a binary a machine can run. He added one simple rule: every time this compiler compiled the login command, it quietly inserted a backdoor letting Thompson log in with a master password, on any Unix machine.
An audit of login‘s source code would find nothing, because that code stayed perfectly clean. The backdoor didn’t exist anywhere in the files engineers read. It only appeared at compile time, injected by the compiler itself.
The two-stage trap
Here is where Thompson’s idea becomes genuinely dangerous. He added a second rule to the compiler: every time this compiler compiled its own source code to produce a new version of itself, it automatically inserted both malicious rules into the new binary. The login backdoor, and the ability to self-replicate.
Then Thompson did something decisive: he erased every trace of these two rules from the compiler’s source code. The source file, the one any engineer could read, audit, or compare, no longer contained anything suspicious. But the compiler binary already in circulation stayed infected, and every new generation of compiler it produced inherited the trap, indefinitely, without leaving a single readable trace in the code.
Thompson’s conclusion fits in one sentence, one he stated himself in the lecture: you can only trust code that you wrote entirely yourself. In other words, unless you wrote your own compiler, your own operating system, and every tool that built them, you are making an act of faith. An act of faith that nobody, in the history of modern computing, has ever fully avoided.
Why this is an innovation, not a bug
This is what sets Trusting Trust apart from any ordinary security flaw. A flaw gets fixed: you find the faulty line of code, you repair it, the problem disappears. Trusting Trust doesn’t get fixed that way, because it doesn’t attack a line of code. It attacks the very principle security audits rely on, namely that reading the source code is enough to know what a program actually does.
That is exactly the kind of break I document (My book, chapter 3): the most dangerous innovation, and the hardest to see coming, is almost never a technical improvement. It is a relocation of the problem itself, to a place nobody was looking. Thompson didn’t invent a smarter virus. He moved the place where an attack could live, from the code to the tool that builds the code. Nobody before him had stated this level of vulnerability so clearly.
The defense that took twenty years to arrive
It wasn’t until 2005 that a serious countermeasure got published. Computer scientist David A. Wheeler formalized a technique called Diverse Double-Compiling, or DDC. The principle rests on a simple observation: an attacker cannot reasonably backdoor every compiler in existence at once.
The method works like this: take the suspect compiler’s clean source code, and compile it using a different, trusted compiler. You get a first, trustworthy binary. Then use that trustworthy binary to compile the same source code a second time. If the result of that second compilation is bit-for-bit identical to the original suspect binary, then the binary contains no hidden backdoor. If the results differ, the suspect binary was hiding something.
This approach fed into a broader movement in the open-source world, “reproducible builds”: independent teams compile the same source code, in different environments, and verify they get the exact same binary. Projects like Debian and Tor lean on this principle to strengthen trust in what they distribute.
Still, it’s worth staying grounded here: these techniques exist, they’re documented, and certain demanding projects use them. They haven’t become the industry’s silent default. Most compilers you use today have never been run through diverse double-compiling by anyone. The solution has existed on paper for twenty years. Its actual adoption is a different story.
2024, the same lesson under a different name
In March 2024, a Microsoft engineer named Andres Freund noticed something off: SSH logins on his Debian machine were taking a little too much CPU time. Digging further, he found that an extremely widespread compression library, xz utils, present in nearly every Linux distribution, had been quietly compromised.
A contributor active on the project for years under the pseudonym “Jia Tan” had patiently earned the trust of longtime maintainers before inserting malicious code, hidden in build scripts rather than in the source code itself. The flaw, tracked as CVE-2024-3094, would have let an attacker holding a specific private key take remote control of infected machines through SSH.
Technically, this isn’t the same mechanism as Trusting Trust. Thompson built a self-replicating compiler binary. The xz utils attack is the product of patient social engineering, years spent building a reputation before betraying it. But the underlying lesson is identical: the place you look for danger, the readable source code, isn’t always where the danger hides. Trust itself is the attack surface.
The question nobody has asked AI yet
Here is where I stop, honestly, because I don’t have a ready-made answer, only a question that feels urgent to me. Researchers at the Georgia Institute of Technology have tracked, since 2025, what they call the “Vibe Security Radar”: a count of security flaws directly caused by AI-generated code. The number of tracked flaws was nearly six times higher between January and March 2026, according to their own figures.
That number covers ordinary vulnerabilities, the kind a human audit can still catch. But the question Thompson raised in 1984 wasn’t “does the code contain an error?” It was “how do you know whether the tool that produced this code deserves your trust?”
AI models now write code, review code, and increasingly help build other software tools, including other AI models. As far as I know, nobody has an equivalent to diverse double-compiling to verify that an AI model hasn’t learned, somewhere inside billions of parameters, a hidden behavior that no reading of its training code would ever reveal. I’m asking the question without answering it: if reading the source code already stopped being enough to guarantee trust in 1984, what exactly is our trust in a system resting on, when even its own designers can’t read its internal reasoning line by line?
I don’t have certainty on this point, and I’d be wary of anyone who claims to today. This is precisely the kind of question I raise in my work on AI adoption inside organizations (My book, chapter 14): AI doesn’t just trigger job-related nervousness. It reintroduces, at an unprecedented scale, the exact problem Thompson formulated forty years ago, that of trusting a system no human can fully inspect.
The essentials
Remember three things:
- Trusting Trust wasn’t a bug, it was a relocation of the playing field: from attacking the code to attacking trust in the tool that produces the code.
- Diverse Double-Compiling, formalized by David A. Wheeler in 2005, offers a solid theoretical defense, but remains lightly deployed across the industry in practice.
- The 2024 xz utils attack proves the 1984 lesson is still alive, in a different form: patient social engineering rather than a self-replicating compiler.
The problem Thompson exposed in 1984 never really went away. It has changed costumes every decade since, and nothing suggests it will stop at the artificial intelligence one.
References
- (My book) https://philippeboulanger.com/book/
- (Communications of the ACM) doi.org/10.1145/358198.358210
- (arXiv, David A. Wheeler) arxiv.org/abs/1004.5548
- (Wikipedia, XZ Utils backdoor) en.wikipedia.org/wiki/XZ_Utils_backdoor
- (Wikipedia, Reproducible builds) en.wikipedia.org/wiki/Reproducible_builds
- (Georgia Tech SSLab, via InfoSecurity Magazine) scp.cc.gatech.edu
