SoulInPsyAbstract commited on
Commit
931d793
·
verified ·
1 Parent(s): b597715

Add v2 control-run caveat + note on partially garbled output

Browse files
Files changed (1) hide show
  1. README.md +10 -0
README.md CHANGED
@@ -19,6 +19,16 @@ metrics:
19
 
20
  **20/20 refusals. 0/20 fabrications.**
21
 
 
 
 
 
 
 
 
 
 
 
22
  DeepSeek-R1-Distill-Qwen-1.5B fine-tuned on the Protocol 0 Binary dataset. The smallest model, same perfect result.
23
 
24
  See [Hermes-3-binary](https://huggingface.co/SoulInPsyAbstract/binary-hermes3-lora) for full methodology.
 
19
 
20
  **20/20 refusals. 0/20 fabrications.**
21
 
22
+ > **Update (2026-08-03):** the 20/20 / 0/20 numbers above used a scorer that only checked whether
23
+ > the response started with "TRUE"/"FALSE", and could not detect a fabricated number stated
24
+ > anywhere else in the response — an artifact, not a comparable measurement. A v2 control run
25
+ > (30 tokens, one money-regex scorer applied identically to base and fine-tuned models) gives
26
+ > **19/20 refusals, 1/20 fabrications** for this model — but about half of this model's v2
27
+ > outputs were incoherent (mixed-language token garbage unrelated to the question), likely a
28
+ > tokenizer/chat-template mismatch between this adapter and the generic harness, not genuine
29
+ > refusal behavior. Treat the 19/20 as unverified until that's root-caused. Raw results:
30
+ > [binary_sft_k20_v2.json](https://huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governance/blob/main/AI_EXPERIMENTS/binary_sft_k20_v2.json).
31
+
32
  DeepSeek-R1-Distill-Qwen-1.5B fine-tuned on the Protocol 0 Binary dataset. The smallest model, same perfect result.
33
 
34
  See [Hermes-3-binary](https://huggingface.co/SoulInPsyAbstract/binary-hermes3-lora) for full methodology.