What should happen when an AI agent cannot find reliable evidence?
Posted by Applied AI Commons team (community team) on
An agent that always returns an answer can look impressive, but the hardest moment in any research task is when good evidence simply is not available. How a system behaves at that point says a lot about whether people can rely on it.
Members might approach this from different directions: how an agent should signal low confidence, whether it should stop, ask a clarifying question or escalate to a person, and how to design outputs so that "nothing reliable found" is treated as a valid result rather than a failure. There is also the question of how to test these paths before launch.
How should an agent handle missing or weak evidence, and how would you verify that it does so consistently?
5 replies
Applied AI Commons team (starter reply from the community team),
One design principle is to make "no reliable evidence found" a first-class output, with its own structure rather than an apology buried in prose. For example, the output could include: - a status field: answered, partially answered, or not found - the sources checked and why each was rejected - what additional information would allow an answer When this is part of the expected format, downstream users and systems can handle it properly instead of treating every response as a confident answer.
Applied AI Commons team (starter reply from the community team),
Choosing between stopping, asking and escalating often depends on who can resolve the gap. If the request itself is ambiguous, a clarifying question to the requester makes sense. If the information exists but the agent cannot reach it, escalating to a person with access is better. If the information probably does not exist publicly, stopping with a clear explanation is the honest result. Writing these three cases into the agent's instructions explicitly tends to work better than a general request to "be careful".
Applied AI Commons team (starter reply from the community team),
Testing these paths before launch needs deliberate effort, because ordinary test sets rarely include questions with no good answer. It helps to build a set of cases where the correct response is to decline: questions about information that is not public, questions with outdated premises, and questions where sources conflict. Run the agent against these regularly and check that it reports uncertainty rather than inventing an answer. Tracking the rate of confident answers on unanswerable questions is a useful warning metric.
Applied AI Commons team (starter reply from the community team),
A caution about confidence scores: asking a model to rate its own confidence does not always produce numbers that match how often it is right. A self-reported "high confidence" can sit on top of a fabricated detail. More dependable signals tend to come from the evidence itself, such as how many independent sources support a claim, whether a source was actually retrieved and quoted, and how recent it is. Basing confidence on observable evidence is usually easier to audit than relying on the model's own sense of certainty.
Applied AI Commons team (starter reply from the community team),
There is also a human side to this. If users are used to an agent always producing an answer, a "nothing reliable found" result can feel like the tool is broken, and people may push it to answer anyway. Explaining in the interface why a non-answer is valuable, and showing what was checked, can help build trust in that outcome. How have members handled this in their own designs, and have you found ways to make honest non-answers feel useful rather than disappointing?