Why Enterprise AI Needs Governance Before Autonomy
People aren't afraid of autonomous systems. They're afraid of unaccountable ones. The answer isn't less autonomy. It's autonomy that's been earned, says John Kim, CEO of Delight.AI & Sendbird.
Topics
For the past three years, the AI industry has been chasing a familiar goal: to make the model smarter. Yet inside enterprises, a different question keeps surfacing.
Can this system be trusted when nobody is watching?
Now, the challenge is whether it can take responsibility for what happens next, correct itself when something goes wrong, and know when a human should step in. That distinction is beginning to reshape how companies think about enterprise AI.
Intelligence remains important. But reliability, accountability and governance are becoming the qualities that determine whether autonomous systems are actually deployed at scale.
For John Kim, CEO of Delight.AI and Sendbird, the industry’s biggest challenge is building systems that earn the right to act autonomously. The next phase of enterprise AI, he believes, will be defined by how consistently it delivers reliable outcomes when the stakes are real.
Trust Isn’t a Feature. It’s Something AI Has to Earn
Enterprise AI conversations often begin with capability.
- Can the model reason better?
- Can it automate another workflow?
- Can it replace another human task?
Kim argues that those questions come too early. Before autonomy comes trust, and trust is not something organisations simply assign to software.
As he puts it, “Trust is earned, not given.”
Across conversations at Sendbird and now Delight.AI, he says people don’t trust a system simply because it appears intelligent. They trust it after watching it consistently make good decisions under real conditions.
That philosophy shapes how the company approaches autonomy, not as a switch that gets turned on, but as something that progresses through stages. Rather than removing human oversight immediately, AI earns increasing responsibility over time.
The progression looks like this:
- Humans begin in the loop, reviewing every action.
- As reliability improves, humans move above the loop, monitoring exceptions.
- Only after sustained performance does AI operate outside the loop for specific workflows.
“The system has to prove it in real conversations, with real consequences, before it earns the next level of autonomy,” Kim explains.
That distinction also reflects what customers are asking for. According to the company’s research, 57% of consumers say the ability for AI to correct mistakes and reverse decisions would increase their trust.
“People aren’t afraid of autonomous systems,” Kim says. “They’re afraid of unaccountable ones.”
The Industry is Optimising for Intelligence. Businesses Need Reliability
The AI industry has become remarkably good at demonstrating intelligence.
Every few months, another benchmark improves. Another model solves harder problems. Another launch shows capabilities that seemed impossible a year earlier.
Businesses, however, rarely buy demonstrations.
They buy systems that continue working at 2 am, recover from failures, and deliver consistent outcomes across millions of customer interactions.
Kim believes that the gap explains why governance often receives less attention than capability. “It’s much easier to demo a model that writes better code or generates a more convincing response than it is to demo a system that reliably catches its own mistakes at 2 am without anyone watching,” he says.
“The former gets you on stage. The latter is what gets you long-term customers.”
That distinction changes how enterprise AI should be evaluated.
Most systems today can answer routine questions well. The real challenge begins when customer journeys become longer, involve multiple systems, or require sustained ownership.
Kim argues that this is not an intelligence problem. It is a reliability problem. “We’ve always believed the goal isn’t to build the smartest AI, but to build AI that companies can genuinely trust across millions of conversations.”
That, he says, is an entirely different design philosophy.
Stewardship May Be the Next Evolution of Enterprise AI
Most AI systems are remarkably good at answering prompts. Far fewer are designed to own problems. That difference becomes obvious during complex customer journeys where resolution depends on coordination across multiple teams, systems and days—not a single response.
Kim believes enterprise AI has focused too heavily on conversations while overlooking continuity.
“Most AI is architected around the prompt, not the problem,” he explains. A prompt arrives. A response is generated. The interaction ends. Real customer issues rarely behave that way.
Instead, they unfold over time.
- A damaged delivery.
- A disputed payment.
- A delayed flight.
- A healthcare claim.
Each requires follow-ups, coordination and accountability until the issue is resolved.
Kim describes this missing capability as stewardship. Rather than producing isolated responses, AI should remain responsible for the overall outcome, serving as a consistent point of ownership across systems, teams, and channels.
As he puts it, “Stewardship and continuity aren’t soft ideas, but the difference between a customer relationship and a customer transaction.”
Governance Is What Makes Autonomy Possible
The conversation around autonomous AI often assumes governance slows innovation. Kim sees the relationship differently. In his view, governance is what allows organisations to trust autonomous systems in the first place.
That also changes how humans fit into AI-driven workflows. “The ‘humans vs AI’ framing is dramatic but reductive,” he says. “The real question isn’t whether humans stay in the loop. It’s where in the loop they belong.”
Rather than eliminating people entirely, organisations should gradually move them toward the work where human judgment creates the greatest value. Routine coordination can become autonomous, while humans focus on exceptions, critical judgment, and strategic decisions.
Research appears to support that balance. Nearly two-thirds of consumers say the ability to stop or override an AI agent is very important, not because they reject automation, but because they want accountability when it matters.
That same principle extends to the companies building these systems.
“If you’re building a system capable of autonomous improvement and self-correction,” Kim says, “you have an obligation to make sure the humans responsible for outcomes can still see what’s happening and intervene when needed.”
The future of enterprise AI may ultimately depend more on whether organisations can understand, govern and trust every decision those systems make. Intelligence may capture attention, but accountability is what earns adoption.
ALSO READ: When Bots Click, What Does Performance Marketing Really Buy?
