Researchers have introduced ProvenanceGuard, a post-generation verification layer that addresses a critical gap in the factuality of tool-using LLM agents. As agents move from single-passage RAG to complex setups using the Model Context Protocol (MCP), they often pull information from multiple sources, such as databases, search tools, and structured records. This creates the risk of "cross-source conflation," where a claim is true within the evidence pool but is incorrectly attributed to the wrong source.
ProvenanceGuard introduces source-aware verification for MCP-based LLM agents
ProvenanceGuard operates on top of "black-box" MCP agents by analyzing the captured MCP trace, including tool outputs and their unique source IDs. The system follows a five-step process: it decomposes an answer into specific claims, identifies the most relevant source for each, verifies if that source supports the claim, checks if the source matches the one named in the answer, and finally issues a verdict.
In evaluations using a medical agent—which utilized patient records and research articles—the method demonstrated high precision. Human experts identified 139 claims that should be rejected, and ProvenanceGuard successfully caught 138 of them. The system also correctly identified the specific supporting source in approximately 86% of cases during testing.
While the method showed challenges in distinguishing between highly similar sources, it successfully detected all instances of intentional source swaps in controlled testing. The researchers noted that the verification overhead is modest, adding roughly half a second per answer in local configurations. This source-aware approach provides a way to maintain data sensitivity and accuracy in environments where knowing the exact origin of information is critical.
Sources
- Getting the Source Right, Not Just the Fact: Source-Aware Verification for MCP Agents （Hugging Face Blog, 2026-09-29）
- arXiv