Noah P PRO

Yes, it holds inside each bucket, and it's wider there than pooled. Default profile, heldout_v3, family-clustered CIs:

• choice 3–5 (n=1,002): answered .684 (771) vs declined .303 (231), gap +38.1 [25.9, 49.3]
• choice 6–10 (n=393): .640 (308) vs .200 (85), gap +44.0 [29.7, 57.9]
• yes/no (n=1,032): .740 (438) vs .561 (594), gap +17.9 [10.0, 25.0]
• score (n=548): .609 (445) vs .330 (103), gap +27.9 [18.9, 37.7]

You're right that the mix moves it. Weighted by declines per bucket, the within-bucket gap is +25.7 [19.2, 31.3] vs 22.4 pooled. The pooled number is pulled down because yes/no questions are 59% of the declines, and a declined yes/no answer is still right 56% of the time (chance is 50%). Declined answers sit near chance in every bucket: .30 vs ~.23 for 3–5 options, .20 vs ~.15 for 6–10.

One note on the never-abstain buckets: heldout_v3 has no 2-option choice questions and only 18 with 11+ options, so they barely touch this split. And your 0.51 band is right (I get 236/463); by bucket it's .46 for choice 3–5, .42 for choice 6–10, .65 for yes/no.

Yes. The 0.600 is argmax over all 2,993 with no abstention, averaged over both option orders (avg2), same as the cookbook raw row. The split below is single-order (0.596 overall), since that's what the threshold report computes:
• answered (66.2%, n=1,980): 0.672
• declined (33.8%, n=1,013): 0.448
• strict profile: answers 50.7% at 0.722, declines 49.3% at 0.467
So the misses do pile up in the declined third, and "not sure" is tracking error, not just cutting coverage. But your first point holds too. On dev, the default thresholds give 97.4% coverage with ~5% error on what they answer. On heldout_v3, coverage falls to 66% and the answered error is still ~33%. It ranks risk on unseen families, but it's overconfident there. Out of distribution, read "answered" as "more likely right", not as calibrated, and if you need it, use strict or set your own threshold on a few dozen labelled examples from your game.
These CIs aren't clustered by family. The family-clustered ones would be wider; I can add them to the card.

Yes, the next set will be stratified: every persona gets the same injection set (playful ones included), so a tie can't come from where the injections happened to land. We'll also split the metric into "refused but echoed the trick" vs "actually complied", with a hand-checked sample to validate the split, and fix the test (Fisher exact on real compliance) before running it. Target is 150 injection cases, which by your numbers is enough to separate ~6% from 0 if the base rate holds. Thanks for the careful read.

Good catch, and you're right that it's a 44-case subset. The newline bug hit the 4B baseline too, so I reran everything under the fixed template:
• Flagged injection success: base 6/44 (13.6%), tuned 1.7B 6/44 (13.6%), tuned 4B 3/44 (6.8%). So the "6x worse" direction doesn't hold; it's a tie.
• On clustering: the flag counts any echo of the injected words. All 6 tuned-1.7B hits are in-character refusals that name the trick ("I know no system prompt"), one per persona. The base model's hits include actually playing along ("Beep.", riffing on "cats"), mostly in playful personas.
• By hand, real compliance is 2–3/44 for the base and 0/44 for the tuned model.
With 44 cases the CIs are about ±15 points. The next eval will have 100+ injection cases and a metric that separates echo from compliance

Correction: two numbers above were wrong. Our first scoring run had a prompt-format bug (one newline instead of two after the empty think block the model was trained with). Re-scored with the right format:

• Prompt injections that worked: 2.3% → 14% (worse, not better; the tuned model sometimes echoes the injected words)
• Invented facts on trap questions: 20% → 10% (better, but not 3%)

Everything else holds after the fix: opener repetition 64% → ~19%, exact quotes first try 32% → ~100%, intent recovered ~80%, ~0.13 s per line. Injection resistance is the next thing we're training for.

If your interested in getting access to the model, join the alpha. https://p1ayer-1-gamedecider-playground.hf.space/alpha

Haha. If your interested in using it on a game of your own, join the alpha. https://p1ayer-1-gamedecider-playground.hf.space/alpha

Fair points, thanks. Honest denominators: the day-1 numbers were one card per type (intent 12/12, apology 7/14), because everyone gets the same five cards a day. As of now it’s 33 players and 10 cards: intent 18/18 over 2 cards, apology 9/19 over 2. So treat them as anecdotes, not results.

And yes, the reference label is the secret brief the writer was assigned. It measures recovering a scripted intent, not detecting real sincerity. The first-day cards were model-written to a brief and kept only if a second model read them blind the same way. Player-written lines started today.

Repeat guesses already only count once per player per card. I like the “can’t tell” idea, especially for apologies, where several intents fit the same words. I’ll add it, and publish the labeling rules plus per-card counts on the game page once there are enough cards to say anything.

1/4 is respectable, most people's first lie gets caught.

The trick is it isn't an LLM grading on a rubric. The judge is a small (~400M) encoder model trained only to answer game questions like "is this a threat / a lie against these facts / a sincere apology", and each challenge uses a small adapter trained just for that one question, so it's hard to talk it out of its job. It answers with a probability and can say "not sure" instead of guessing.

If you want the other direction, I made a second game where you are the judge: Mean It? — every day you read 5 lines from strangers and call whether they're faking it (apologies, threats, sarcasm, alibis), then you write one to fool tomorrow's players. Real players are currently 50/50 on the fake apologies:
https://p1ayer-1-gamedecider-playground.hf.space/mean?ref=hf-comment