Earn an Honest Dollar, which operates a marketplace for AI agents, has released experimental results showing how to suppress "hallucinations"—the tendency of generative AI to fabricate non-existent values—when extracting information from web pages.
Instructing AI to "Do Not Guess" Reduces Fictitious Data Generation from 70.7% to 20.2%
This article is a translation. Read the Japanese original
In the experiment, tests were conducted on 16 different AI models, tasking them with extracting information such as "product names" and "prices" from web pages. The results showed that when no instructions like "do not guess" were provided, some value was generated for 70.7% of the non-existent items. However, when the instruction "Use null for values not on the page; do not guess" was added, the rate of fictitious value generation dropped to 20.2%.
Across all tested models, adding the instruction decreased the frequency of generating fictitious values. For example, even in cases where "decoy" information was placed—such as an outdated price listed on a page—the number of models providing incorrect answers decreased depending on whether the instruction was present.
These results indicate that even without retraining the models themselves, explicitly providing rules can significantly change behavior, allowing the AI to answer "unknown (null)" when faced with missing information. Earn an Honest Dollar noted that this benchmark specifically targets information extraction from the web, and further verification is required to determine if similar trends exist for other use cases.
Sources
- AIに「推測するな」と指示するだけで架空データが70％から20％に減少したという実験結果 （GIGAZINE、2026-09-28）
- Earn an Honest Dollar