Post
81
The checker was right. Its own report about itself was lying.
Round 31 of an outside audit taught a governance checker a new rule: a claim's location in a source can count as known even before anyone's pinned the exact spot, if a co-cited sibling already has one. Correct call, shipped it.
Two rounds later the checker's end-of-run summary was still silently using the old math — scoping its percentages against the wrong denominator. The one record round 31's own fix had produced never showed up in the checker's account of itself.
Fixed by giving locator_precision and locator_ceiling separate denominators instead of pretending they still meant the same thing.
Six rounds of an outside reviewer (@dipankarsarkar ) finding gaps like this so far. None of them made the checker bigger. Each one made it worse at lying to itself.
That's the actual bet: not a stronger model in the loop. A stricter loop around whatever model you already have.
Credentials: @dipankarsarkar
Checker: scripts/check_locator_precision.py, commit 7841ee7
Dataset: huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governance
Source: github.com/soulinpsyabstract/sipa-os-governance
Round 31 of an outside audit taught a governance checker a new rule: a claim's location in a source can count as known even before anyone's pinned the exact spot, if a co-cited sibling already has one. Correct call, shipped it.
Two rounds later the checker's end-of-run summary was still silently using the old math — scoping its percentages against the wrong denominator. The one record round 31's own fix had produced never showed up in the checker's account of itself.
Fixed by giving locator_precision and locator_ceiling separate denominators instead of pretending they still meant the same thing.
Six rounds of an outside reviewer (@dipankarsarkar ) finding gaps like this so far. None of them made the checker bigger. Each one made it worse at lying to itself.
That's the actual bet: not a stronger model in the loop. A stricter loop around whatever model you already have.
Credentials: @dipankarsarkar
Checker: scripts/check_locator_precision.py, commit 7841ee7
Dataset: huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governance
Source: github.com/soulinpsyabstract/sipa-os-governance