The agent evaluation gap: Enterprise AI organizations have a reality-alignment problem, not a coverage problem — and most are shipping to production anyway

Enterprises are increasingly giving AI agents more autonomy despite significant gaps in how these agents are evaluated. Over half have deployed agents that failed customers once in production, despite passing internal checks, highlighting a disconnect between lab-based evaluations and real-world performance. This "reality-alignment problem" means that most organizations aren't fully trusting automated evaluations, even as they push for more autonomy in AI agents. This misalignment poses risks, as two-thirds are already allowing or planning for agent changes to go live without adequate verification.