Bridging the Knowing-Saying Gap: New Research Explores Why AI Can't Always Explain Itself

A new paper posted on arXiv probes a puzzle at the heart of modern AI: why can a model sometimes do a task correctly but fail to explain how? Dubbed the "knowing-saying gap," the phenomenon is familiar to anyone who has watched a large language model produce a plausible rationale for an answer that is subtly wrong — or nail a coding challenge while offering a muddled account of its approach.

The researchers behind the study offer a framework for thinking about the gap, distinguishing between a model's ability to solve a problem and its capacity to verbalize the reasoning behind that solution. Through a series of experiments, they show that the gap is not simply a matter of confidence or verbosity; it is structural. Models appear to rely on internal representations that encode more information than their output layers are trained to surface, and the mismatch grows with task complexity.

For developers, the findings resonate beyond academic curiosity. AI coding assistants routinely generate code that works, yet the explanations they provide — when they provide them — can be misleading or incomplete. The knowing-saying gap may help explain why: the model has learned to produce correct output, but the parts of its network responsible for that output are not fully aligned with the parts that generate natural-language commentary.

"Understanding this gap is critical for building tools that developers can trust," the paper argues. "If we cannot rely on an AI system to explain its own decisions, we need to design for scrutability from the outside — through better logging, better tests, and better interpretability methods."

The authors do not pretend to close the gap; instead, they map it. They propose future work that could lead to training objectives that reward both correctness and explainability, or architectures that explicitly decouple problem-solving from explanation. For now, the paper serves as a useful reminder: an AI that can do something is not the same as an AI that can tell you how.