Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays

Author here (Bhaskar Gurram, Zasti AI). The abstract is dense, so here's what
the paper actually argues.

The claim

"Accept a field only if the error rate among accepted fields stays under α" is the
trust contract every document-extraction system implicitly sells. The standard way
to implement it — fit a confidence score, hold out a calibration split, take the
smallest add-one-smoothed threshold with empirical selective risk ≤ α — silently
violates that contract on real documents. Not visibly. Not in a way your
dashboard catches.

Testbed: 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts, of
which only 49.0% are correct. Deliberately hard, deliberately real.

Three failure modes, each pinned by a counterfactual

1. Document clustering. Fields aren't exchangeable — errors arrive in
document-shaped clumps. Measured design effect 1.84–2.45, so your effective
calibration sample is roughly half its nominal size. A clean-fit control (score fit
on held-out docs) still overshoots at risk 0.105 in 50% of splits, which isolates
clustering from the next failure.

2. Score-refit leakage. Fit a high-capacity score and its threshold on the same
fields and the score's optimism transfers straight into the threshold: coverage
0.416 at risk 0.127, violating nominal α=0.10 in 95% of splits. It looks
spectacular. It's invalid. A 5-parameter logistic barely overfits (0.105) — which is
exactly why nobody notices until score capacity grows.

3. Tie-mass pathology. A degenerate score distribution collapses the threshold
grid. Our first capture silently shipped an all-zero entailment column; the fused
score went from 1,702 distinct values to 257, with tie masses of 221 and 183 fields
at the acceptance head. A threshold takes a tie mass whole or not at all, so no
certificate existed at any confidence level. Counterfactually reproduced by zeroing
that one signal on the intact dump: certified coverage 0.030 → 0.001.

The validity ladder

The fixes are organized as tiers, each row stating its own estimand, assumption, and
violation fraction — because these bound different functionals under different
assumptions and shouldn't be read as one price curve.

The vocabulary matters: tiers 1–2 control expected risk; only tiers 3–4 certify.
Tier 1's 47.5% violation rate isn't a bug — that's what a mean-controlled bound
sitting near its boundary looks like. If you need P(violation) ≤ δ, you buy tier 3
and pay 0.318 → 0.171.

Tier 4 is the only tier whose assumptions actually match documents, and it is
near-vacuous today — 0.060 coverage, certifying nothing in 19/40 splits. We lead
with that rather than bury it. Powered document-level PAC procedures are the named
open problem.

When conditioning pays

A two-regime law, with mechanism and scope:

• With a learned score, covariates belong in the score. A tree fusion splits on
  the covariate internally and equalizes per-group scales, so external Mondrian
  conditioning only fragments the threshold sample. It's actively harmful here
  (−0.062 / −0.036 coverage, p < 10⁻⁴).
• With a weak or frozen score, covariates belong in the taxonomy — and pay
  precisely where a pooled threshold cannot certify at the target α. On CORD,
  support-bin 0.171 vs pooled 0.091–0.098 at the rigorous tier.

Formalized at the ρ=0 boundary (Props. 1–3); the sign prediction is verified on all
three corpora, though n=3 corpora is n=3.

What we're careful not to claim

Provenance conditioning wins every rigor tier on the sonnet CORD capture (p < 10⁻⁴,
Bonferroni-corrected over taxonomies) — and the same win collapses on the identical
documents under haiku or Qwen2.5-14B, where the practical-tier winners are
field-type taxonomies instead. That counterexample is printed in §7, not hidden. It's
the boundary of the claim.

What does transfer: a frozen configuration, executed once with no tuning on
selection-untouched captures, held at both risk levels on claude-haiku-4-5
(0.167 at 0.093) and on open-weights Qwen (0.149 at 0.099). Across both untouched
captures the risk contract never failed — what varies is coverage, which tracks
signal quality.

And the guarantee survives contact with humans: three blind annotators (Fleiss'
κ=0.83) re-judged 149 fields from the accepted set. Human-verified selective risk:
2/149 = 1.3%, against a 10% budget. The automatic calibration labels err
one-sidedly pessimistic (21% of auto-flagged errors are actually correct, 0%
false-optimism), so thresholds fit against them are conservative.

The price of rigor

Certified coverage retained vs the invalid folklore baseline: 0% at α=0.05, 28% at
0.10, 42% at 0.15, 84% at 0.20. Rigor is a knob, and its cost collapses as the risk
budget grows. That's the operating guidance, not a footnote.

Artifact

Apache-2.0, seed-pinned, regression-gated: https://github.com/bhaskargurram-ai/verifydoc

pip install verifydoc
make results   # regenerates every number in the paper from configs/

40 fixed document-level splits (seed 7), bit-exact sanity gates chained across
experiments, a 463/624-cells-bit-identical regression check — and full forensics on
a capture-stage data defect in Appendix B, including how it was found and what it
invalidated. Try it with no install: 🤗 https://huggingface.co/spaces/bhaskar1225/verifydoc

Companion benchmark paper (VerifyDocBench) owns the datasets, labeling protocol,
reliability audit, and the per-model measurement study. This paper owns the
procedures, guarantees, diagnoses, and characterization.

We claim no new conformal theory — the machinery is classical (Vovk 2003;
Angelopoulos 2021, 2024). The contribution is the diagnoses, the protocol, the
certified application, and the two-regime characterization.

Open question I'd genuinely like input on: tier 4 is the assumption-honest tier
and it's nearly vacuous at 800 documents. Has anyone had luck with
variance-adaptive or clustered document-level bounds that stay powered at this scale?
That's the bottleneck between "honest" and "useful," and I don't think it's a data
problem.