We Shot the Wrong People: Autonomous Weapons and the Case for Human Judgment - Modern War Institute

Former Google CEO Eric Schmidt and other prominent defense leaders have argued that AI will define the future of warfare, framing such gaps as temporary hurdles solvable with more data and computing power. However, some problems are not data problems. Modern AI struggles with what philosophers and cognitive scientists describe as epistemic limits, fundamental boundaries on acquiring and understanding knowledge about the world. Many battlefield challenges involve ambiguity that exists in actual reality. No sensor can directly observe hostile intent. Social context, deception, fear, civilian behavior, and adversarial adaptation all exist outside purely quantitative representations.

The danger compounds when paired with human cognitive tendencies. Recent research on drone strike simulations found that humans exhibit a strong bias toward AI competence, deferring to machine recommendations even with AI explicitly introduced as fallible. When given completely randomized targeting advice, operators still reversed their own correct life-or-death decisions in the majority of cases simply because the machine disagreed with them. Dr. Terry Oroszi describes the underlying phenomenon as “plausible mediocrity,” outputs that appear authoritative while masking fundamental flaws. Basically, beautifully polished bullshit.

The danger is not obvious failure. It’s plausibility. In a lethal system, plausible errors become operational consequences. As Dr. Kristin Milchanowski notes in Return on Intelligence, behavioral science demonstrates that people often adopt what feels right rather than what works best. AI systems are exceptionally capable of generating outputs that feel intelligent. Ground combat cannot afford this confusion.