False AI report almost sparked US military operation against Chinese ship linked to nuclear weapons

It has come to light that a military report produced with the aid of artificial intelligence (AI) almost led to a Chinese ship being boarded by US forces. The mission was aborted shortly before launch once the potential “false positive” report was scrutinized in more detail.

First reported by CNN, a military analyst reportedly came across some concerning information about a cargo manifest for a Chinese ship heading to the Middle East. The unnamed analyst then used AI to help analyze and interpret that information.

After combining classified and open-source information, the AI concluded that the ship was likely carrying components for nuclear weapons. Nothing really new here; AIs are known to suffer from hallucinations from time to time.

However, the really concerning part is what happened next. The analyst then used the AI to formalize its conclusions into a conventional military intelligence report without double- or triple-checking the validity of its conclusions.

AI “false positive” taken seriously

This report was then distributed through the regular channels, clearly raising alarms enough to plan a potential interdiction mission. According to CNN, its sources told them that armed personnel were preparing to board it and aircraft were already airborne.

Before the operation happened, someone scrutinised the underlying intelligence and discovered that the identification of the cargo was wrong. If the operation had gone ahead, it could have had very serious repercussions.

So, when looking at the bigger picture, this isn’t really an example of an AI failure leading to a military blunder, but rather a failure of verification before circulation.

But how could this have happened? Well, as CNNreports, AI hallucination is a known issue but can be harder to detect when it includes genuine information. In this case, a real ship, real cargo manifest, schedule, etc.

The next issue is something we could call “authority laundering.” The analysts in question were able to turn the questionable AI output into abona fide credentialed report for circulation. Anyone in receipt of this would have naturally assumed the validity of its content and acted accordingly.

Another issue, and perhaps the bigger one, is that the US currently doesn’t operate a uniform multi-service verification standard for this kind of thing. Different units use different models and have their own in-house policies and procedures for using them.

Human-in-the-loop didn’t help

Interestingly here is that this event also appears to be a major failing of the “human-in-the-loop” concept of AI use. Technically, there were humans throughout this process, yet according to CNN, the error still travelled remarkably far.

While it was caught on this occasion, US military intelligence analysts will need to remain vigilant to prevent future “false positives” in the future. This is especially important as US military forces are looking to lean more on AIto help make better decisions faster.

But, as this event highlights, faster may not be a great idea if its compreses the time taken to check and then check again before acting.

“AI in targeting is definitely something that is ramping up, and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide,” a “source familiar with the military’s current policies” told CNN.

“AI allows you to get to a bad idea faster,” one unnamed source told them.