By Magali Gruet, USC Viterbi School of Engineering | Reviewed by Irfan Ahmad
The USC Viterbi professor who built systems for nuclear missiles and ambulance networks explains why today’s AI engines are structurally unfit for the infrastructure that keeps society running.
Neil Siegel has spent his career building systems where mistakes cost lives. Ambulance dispatch networks, air traffic control, the Minuteman missile system — 450 nuclear warheads, 60 years, zero accidental launches. So when he says AI is not ready to run the world’s critical infrastructure, he’s not speculating. He’s pattern-matching against decades of hard-won engineering judgment.
“The algorithms used in these AI engines are based on mathematics that cannot — absolutely cannot — bound either the frequency or the magnitude of the errors they make,” said Siegel, IBM Professor of Engineering Management and Professor of Engineering Practice at the USC Viterbi School of Engineering, in an interview filmed for The BlackVeil Files. “You ask 10 questions, and nine times it makes a pretty reasonable answer. And the tenth is wildly wrong. And if you depend on the tenth, you would die.”
The full interview, available on YouTube, cuts to the core question animating the AI debate: not whether these systems are impressive, but whether they are trustworthy enough for the infrastructure society depends on most.
Two Worlds
Siegel draws a sharp line between what he calls the “toy business” — Silicon Valley’s dominant culture, built around speed-to-market and features — and the world of mission-critical systems where he built his career. The distinction isn’t a slight; it’s structural.
“If Facebook goes down for a while, it’s an annoyance. If an ambulance dispatch system goes down, people die.” The companies now building AI engines, he argues, grew up in the first world — optimized for consumer delight, not societal reliability. “They send out a patch to their software every 17 seconds. In our business, we don’t do that. Because if you patch an airplane, you have to redo the airworthiness certificate. That costs $100 million.”
The mismatch becomes dangerous, he warns, not because Google or OpenAI will deploy their engines in a hospital — but because someone else will. “My worry isn’t that the people who build these AI engines will put them in an ambulance dispatch system. My worry is that somebody else will. And they probably already are.”
The AGI Question
Asked about artificial general intelligence — the idea that AI will eventually surpass human cognition across the board — Siegel is characteristically blunt: “I have a lecture I give my students where I go through the mathematics and physics of why I don’t think that’s possible.”
His argument draws on Kurt Gödel’s incompleteness theorems, Alan Turing’s proof that some problems cannot be solved by any algorithm, and Roger Penrose’s theory that the human brain operates at a quantum scale that is inherently non-deterministic. “Whatever the brain is,” Siegel says, “it’s not a computer. It does not work that way.”
He acknowledges he’s in the minority. But the implications matter: if the brain can’t be replicated algorithmically, then AI — however sophisticated — will keep guessing when it doesn’t know. “The AI engines are financially incentivized to guess and not tell you they’re guessing. If you had a multiple-choice test where getting it wrong cost you five points, you’d leave it blank. We build society’s critical systems more like that: no guessing, and being explicit about when we are unsure. Psychology teaches us that humans make better decisions when they are aware of the uncertainties involved, than when they are hidden.“
The Regulation Imperative
What would responsible AI deployment look like? Siegel’s answer is unambiguous: a regulatory body with teeth, modeled on the FDA or the FAA. Specifications, failure thresholds, testing protocols — the same infrastructure that governs drugs and aircraft. The regulators, not the builders, would decide when something is good enough.
“You can’t even sensibly talk about an AI infrastructure without it. It would be like saying we’re going to allow a prescription drug business with no qualification procedure.” He pauses. “Two hundred years ago, anyone could be a physician and anyone could make and sell drugs. We decided that wasn’t good enough.”
The AI industry, he predicts, will fight regulation hard. But the scaffolding will come — the question is how much gets built before something serious goes wrong.
Watch the full interview below.
This article may feature some AI-assisted content for clarity, consistency, and to help explore complex scientific concepts with greater depth and creative range.
DIW Editor's Note: Our review found that some passages presented as direct quotations differ from the available interview transcript, including changes in wording and attribution. The article's overall account reflects the interview's main themes, but readers should not assume every quoted passage is verbatim.
Read next: Don't Want Your Data Used for AI Training? Here's How to Opt Out Across Apps and Devices
