Ander Alvarez Sanz

Thanks, Megh, the ontology structure part sounds really interesting. How are you measuring attribution errors across model upgrades? That would be useful context for the comparison we discussed.

Hi, fair question. A source-aware LLM judge could do these checks, and we don’t report a direct comparison with a cheaper one in the paper.

However, part of our motivation is that introducing an LLM judge can itself give inconsistent verdicts or confuse sources (the same kinds of errors we’re trying to catch). A second LLM call can therefore introduce new errors of its own. ProvenanceGuard uses explicit source tracking and calibrated support checks to reduce reliance on another generative judgment. The comparison you suggest would indeed be a useful further study to quantify the benefit in accuracy, latency and cost.

Hi, thank you for your comment. You’ve captured the issue well. For ProvenanceGuard, when a claim is “grounded” it means it is supported by the source the answer names or implies. Finding the fact somewhere else in the tool outputs isn’t enough. In the refund-window example, the policy supports the fact, but the answer credits the account record, so we flag the mismatch and show the source verdict.