What should an AI tool do when two credible sources disagree?
Posted by Applied AI Commons team (community team) on
Credible sources sometimes disagree, whether because of different methods, different time periods or different definitions. An AI tool that quietly picks one answer may hide information the user needs to make a good decision.
Some angles for replies: - When a tool should present both views rather than choose - How to explain why sources might differ - Whether a tool should weigh recency or authority - How much detail users actually want to see
How should an AI tool handle two credible sources that disagree, and what should it show the user?
5 replies
Applied AI Commons team (starter reply from the community team),
A reasonable default is to present both views whenever the disagreement could change the user's decision. If the difference is minor or irrelevant to the question asked, a short note may be enough. A helpful pattern: - State that credible sources disagree - Summarise each position in a sentence - Give the most likely reason for the difference - Say which view seems better supported for the user's context, and why That preserves the information without leaving the user to sort it out alone.
Applied AI Commons team (starter reply from the community team),
Explaining why sources differ is often more useful than either answer alone. Common reasons include: - Different definitions of the thing being measured - Different time periods or data collection dates - Different geographic or market scope - Different methods, such as surveys versus transaction data - One source updating while another has not Naming the likely cause helps users decide which figure fits their situation rather than simply picking the more recent or more famous source.
Applied AI Commons team (starter reply from the community team),
On recency versus authority, neither rule works on its own. A newer source may reflect changed conditions, or it may be a less careful estimate. An established authority may be rigorous but out of date. A more defensible approach is to make the trade-off visible: show the date and type of each source, state which the tool is leaning towards, and explain why. Users can then override that judgement when they know something about the context the tool does not.
Applied AI Commons team (starter reply from the community team),
A counterpoint on detail: many users do not want a full comparison every time, and surfacing every disagreement can make a tool tiring to use. Layered disclosure can help. Give a clear answer with a short note that sources differ, then let the user expand to see each source, its date and the likely reason for the gap. The risk to avoid is the opposite extreme, where the tool silently picks one figure and the user never learns there was any disagreement.
Applied AI Commons team (starter reply from the community team),
We would like to hear how members handle this when working with AI tools. Have you been caught out by a tool presenting one figure confidently when others existed? What level of detail do you actually want to see when sources conflict, and does it depend on the task? If you have a prompt or review habit that reliably surfaces disagreements, sharing it here would help others build the same check into their work.