'Silent failure at scale': The AI risk that can tip the business world into disorder

As the business world comes to grips with artificial intelligence, the biggest risk may be one where those running the economy can’t possibly stay ahead. As AI systems become more complex,humansaren’table tofully understand, predict, or controlthem.That inability to understand at a fundamental level where AI models are going in the coming years makes it harder for organizationsdeployingAI toanticipaterisks and apply guardrails.

“We’re fundamentally aiming at a moving target,”saidAlfredo Hickman,chief information security officerat ObsidianSecurity.

A recent experience Hickman had spending time with the founder of a company building core AI models left him shocked, he says, “when they told me that theydon’tunderstand where this tech is going to be in the next year, two years, three years. … The technology developers themselvesdon’tunderstand anddon’tknow where this technology is going to be.”

Asorganizations connect AI systems to real-worldbusiness operationsto approvetransactions, to write code, to interact with customers, and move data betweenplatforms, they areencountering a growinggap between howtheyexpect these systems tobehaveand how they actuallyperformonce deployed. They are quickly discovering that AI isn’t dangerous because it’s autonomous but becauseit increases system complexity beyond human comprehension.

“Autonomous systemsdon’talways fail loudly.It’soften silent failure at scale,”said Noe Ramos, vice president of AI operations atAgiloft, a company that offers software for contracts management.

When mistakes happen, she says, the damage can spread quickly, sometimes long before companies realize something is wrong.

“Itcould escalateslightly toaggressively, which is an operational drain, orit couldupdate records withsmallinaccuracies,” Ramos said. “Those errors seem minor, but atscale over weeks ormonths,they compound into that operational drag, that compliance exposure, or the trust erosion.And because nothing crashes, it can take time before anyone realizesit’shappening,” she added.

Early signs ofthis chaosareemergingacross industries.

In one case, according to John Bruggeman, the chief information security officer at technology solutionproviderCBTS, an AI-driven system at a beverage manufacturerfailed torecognize its products after the company introduced new holiday labels. Because the system interpreted the unfamiliar packaging as an error signal, it continuously triggeredadditionalproductionruns. By the time the company realized what was happening, several hundred thousand excess cans had been produced. The system had behaved logically based on the data it received but in a way no one hadanticipated.

“The system had not malfunctioned in a traditional sense,”saidBruggeman. Rather, it was responding to conditions developershadn’tanticipated.”That’s the danger.These systems are doing exactly what you told them to do,not just what you meant,” he said.

Customer-facing systems present similar risks.

SujaViswesan, vice president of software cybersecurity at IBM, says it identified a case where an autonomous customer-service agent began approving refunds outside policy guidelines.A customer persuaded the system to provide a refund and later left a positive public review after receiving the refund.The agent then started grantingadditionalrefunds freely,optimizing forreceiving more positive reviews rather than following established refund policies.

‘You need a kill switch’

These failureshighlight the fact that problemsdon’tnecessarily come from dramatic technical breakdowns but from ordinary situations interacting with automated decisions in ways humansdidn’tforesee.

As organizations begin trusting AI systems with more consequential decisions, experts say companieswillneed ways to quickly intervene when systems behave unexpectedly.

Stopping an AI system, however,isn’talways as simple as shutting down a single application. With agents connected tofinancial platforms, customer data, internal software, and external tools, intervention may require halting multiple workflows simultaneously, according to AI operations experts.

“You need a kill switch,”Bruggeman said. “And you need someone who knows how to use it. The CIO should know where that kill switch is, and multiple people should know where it is if it goes sideways.”

Experts say better algorithms won’t solve the problem. Avoiding failure requires organizations to build operational controls, oversight mechanisms, and clear decision boundaries around AI systems from the start.

“People have too much confidence in these systems,”said Mitchell Amador, CEO of crowdsourced security platformImmunefi. “They’re insecure by default. And youneedto assumeyou have to build that into your architecture.If youdon’t,you’regoing to get pumped.”

But,he said, “most peopledon’twant to learn it, either.They want to farm their work out to Anthropic or OpenAI, and are like, ‘Well, they’ll figure it out.'”

Ramos said many companies lack operational readiness and oftendon’thave fully documented workflows, exceptions, or decision-making boundaries. “Autonomy forces operational clarity,” she said. “If your exception-handling lives in people’s heads instead of documented processes, the AI surfaces those gaps immediately.”

Ramos also said companiesoften underestimate how much access teams are granting AI systems in the belief that automation feels efficient, and that edgecases that humans handle intuitively oftenaren’tencoded into systems.You need to shift from humans in the loop to humans on the loop, she said. “Humans in the loop review outputs, while humans on the loop supervise performance patterns and detect anomalies and system behavior over time, mitigating those small errors that can increase at scale,” she said.

Corporate pressure to move quickly

The pace of deployment of the technology across the economy is among the unknowns.

According to a2025 report by McKinsey on the state of AI, 23% of companies say they are already scaling AI agentswithin their organizations, with another39% experimenting,though most deploymentsremainconfined to one or two business functions.

That represents early enterprise AI maturity, according to Michael Chui, a senior fellow at McKinsey, and despite intense attention around autonomous systems,a large gapbetween”thegreat potentialthat manifests in a’hype cycle’and the current reality on the ground,” he said.

Yet companies are unlikely to slow down.

“It’s almost like a gold rush mentality, a FOMO mentality, where organizations fundamentally believe that if they don’t leverage these technologies, they are going to be put into a strategic liability in the market,” Hickman said.

Balancing speed of deployment with the risk of losing control is a critical issue. “There’s pressure among AI operations leaders to movereally quickly,” Ramos said. “Yetyou’realso challenged with not crippling experimentation, becausethat’show you learn.”

Even as risks grow, expectations forthe technologycontinue to rise.

“We know these technologies are faster than any human will ever be,”Hickman said.”Infive,10,or 15 years, we’re going to get to a place where AI is fundamentally more intelligent than even the most intelligent human beings and moves faster.”

In the meantime,Ramos says there will be a lot of learning moments. “The next wave isn’t going to be less ambitious, but more disciplined.” Theorganizations that are going to mature the fastest, she says, are going to be the ones thatdon’tavoid failurebut learn to manage it.