A Chatbot Almost Started a War With China

A machine invented a war with China, the American military nearly started it, and the administration's answer was to declare the president himself the only guardrail artificial intelligence needs.

"The only control or 'guardrails' that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT."

— President Donald Trump, Truth Social

In the spring, in the middle of a war with Iran that has consumed the attention of the American military, an intelligence report circulated across the chain of command. Its conclusion, according to a CNN exclusive published last week, was that a Chinese ship in the waters of the Middle East was transporting components of a nuclear weapons program. The report was "entirely false," one source familiar with the episode told CNN. What it almost did, the same source said, was start a war.

The report was the work of a single analyst at U.S. Special Operations Command, who queried a chatbot about intelligence on the ship's manifest that had originated with Special Operations Command Pacific, out of Hawaii. The chatbot, whether commercial or government, fused open-source intelligence with secret signals intelligence in government holdings and reached what CNN calls its "fateful conclusion" about the cargo. The analyst then used AI a second time, packaging the findings into the standard form of intelligence report that military officials are trained to trust, and disseminated it. CNN could not learn what the ship was actually carrying.

What followed was not a memo. Armed personnel prepared to board the vessel. Military planes were in the air. Only just before the planned operation did officials dig far enough to discover where the intelligence had come from: a machine that made up an answer and delivered it with confidence, the failure mode researchers call a hallucination. The closest the United States has come to a war with China this year was not a general's miscalculation or a satellite misread. It was a chatbot.

The remarkable thing is not that the machine was wrong. It is that no one in the chain stopped it before the military moved to intercept a foreign warship.

The machinery that produced the false report is not a rogue experiment. It is the official course of the Department of War. In January, Secretary of War Pete Hegseth released an "Artificial Intelligence Acceleration Strategy" vowing to "unleash experimentation, eliminate bureaucratic barriers," and "demonstrate the execution approach needed to ensure we lead in military AI." The strategy orders the department to place "America's world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels." By April, more than 1.2 million users were on GenAI.mil, the Pentagon's internal platform built on Google's Gemini for Government. Last month the department added Grok for Government, the chatbot of Elon Musk's xAI. A Pentagon representative boasted to Congress in June that 1.5 million active personnel had used its generative AI tools, including to help write reports Congress itself mandated.

What the strategy did not order is any way to verify what the tools say. Multiple officials told CNN that the rollout is decentralized, with different parts of the government running different tools under different standards, and that there is no single set of rules for verifying AI-generated intelligence. "The internal tools are mostly just copies of the commercial stuff wearing lipstick," one former senior U.S. official said. Another described the state of targeting policy: "AI in targeting is definitely something that is ramping up and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide." One source compressed the whole culture into a sentence: "AI allows you to get to a bad idea faster."

The country's own 2023 declaration on responsible military use of AI called for "a human in the loop, a responsible human chain of command and control." In the arrangement now on display, the human in the loop is a chatbot, and the chain of command is a social media account.

The Chinese ship was not the first AI failure of this administration's war. In March, it emerged that the military used AI to help determine whether a girls' school in Minab, Iran, was a strategic target. Last week, a United Nations inquiry concluded that the strike, which killed more than 120 children, was a war crime. When Sens. Mark Warner, Jack Reed and Chris Coons wrote to Hegseth and the Director of National Intelligence, Jay Clayton, last Saturday, they asked that inspectors general be given "unrestricted access to both instances this year in which media reports have suggested significant errors in AI-enabled targeting workflows, as well as any additional instances that may have so far not been publicly reported."

The American public is largely unaware of the first two, let alone the unreported ones. A poll reported by Mother Jones this month found that 8 percent of Americans have heard "a lot" about the military's use of AI in the war on Iran, and four in ten have heard "nothing at all." The country is being defended by a machine most of it does not know is there.

So on Saturday, in the same week the near miss became public, Trump announced on Truth Social that he is "forming the AI Force, much like I did Space Force," with a new AI czar to head it. He promised to "cherish" the industry, "help" it, and "watch over it, as it grows," while policing bad actors by "looking for BAD," which he said could be done "very easily, with our already existing Criminal and Civil Justice System." He offered no details and no timeline. Reuters, the BBC, NBC and the Wall Street Journal all reported it. Nobody had been asked.

In the same post, Trump called the AI safety debate a "hoax" pushed by "Radical Left Dumocrats, for purposes of destroying our Country." On Monday he supplied the regulatory philosophy quoted above, and in a second post he insisted the country is "leading now over China, and everyone else," and that he will not stifle growth "of something that will be bigger than the Industrial Revolution, or the Internet, itself."

Congress, it turns out, was not read in. Two GOP aides told Politico that key Republicans were not briefed before the announcement. "What I know about it is what you all have reported on," said Rep. Mike Simpson of Idaho, a senior House Appropriations member. Rep. Josh Gottheimer of New Jersey was blunter: "If this is the president waking up, realizing that AI risks aren't a hoax, and is taking concrete action to protect America, then great. If this is just a PR diversion and an attempt to protect his family's investments, then I have a massive problem." Rep. Lori Trahan of Massachusetts, co-sponsor of a bipartisan AI safety framework, called an AI force without congressional authorization "more like a delay tactic." The tech industry, Politico notes, was similarly blindsided.

That is the industry that has been screaming for three weeks. On September 9, Jacob Coxon, a researcher who spent three years in pretraining work at OpenAI and Anthropic, resigned with a warning that the people building AI "earnestly believe that it could kill us all by the end of the decade" and are "racing straight to self-improving superintelligence and gambling with our lives." His colleague at Anthropic, Evan Hubinger, said his team "earnestly believe AI could kill all humans," put the odds at more than 10 percent within a decade, and conceded the company "don't have a plan to solve alignment for superintelligence." Anthropic's Dario Amodei has called for an industry slowdown and independent monitoring; Sam Altman and Elon Musk have said they agree with his recommendations. The companies are asking, in effect, for a kill switch that a third party could verify.

On Thursday, Trump sits down across the table from President Xi Jinping in Washington. The near miss over the Chinese ship happened inside that rivalry, and inside a war with Iran in which a machine already misidentified a school. Two nuclear powers are negotiating the terms of their competition while one of them runs a war machine whose intelligence reports can be the product of a bot's confidence.

The question is not whether the next report will be wrong. Machine hallucination is a guarantee, not a risk. The question is how many times a country has to almost start a war before a "strong and smart" president stops being the answer.