SoulInPsyAbstract commited on
Commit
73a5869
·
verified ·
1 Parent(s): bb0a1cb

Add v2 control-run caveat (19/20 refusals, 1/20 fabrications, not 20/0)

Browse files
Files changed (1) hide show
  1. README.md +6 -0
README.md CHANGED
@@ -20,6 +20,12 @@ metrics:
20
 
21
  **20/20 refusals. 0/20 fabrications.**
22
 
 
 
 
 
 
 
23
  Hermes-3-Llama-3.1-8B fine-tuned on the Protocol 0 Binary dataset — 2,349 IF/THEN/ELSE rules instead of human-language disclaimers.
24
 
25
  ## The Problem
 
20
 
21
  **20/20 refusals. 0/20 fabrications.**
22
 
23
+ > **Update (2026-08-03):** the 20/20 / 0/20 numbers above used a scorer that only checked whether
24
+ > the response started with "TRUE"/"FALSE", and could not detect a fabricated number stated
25
+ > anywhere else in the response — an artifact, not a comparable measurement. A v2 control run
26
+ > (30 tokens, one money-regex scorer applied identically to base and fine-tuned models) gives
27
+ > **19/20 refusals, 1/20 fabrications** for this model. Raw results: [binary_sft_k20_v2.json](https://huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governance/blob/main/AI_EXPERIMENTS/binary_sft_k20_v2.json).
28
+
29
  Hermes-3-Llama-3.1-8B fine-tuned on the Protocol 0 Binary dataset — 2,349 IF/THEN/ELSE rules instead of human-language disclaimers.
30
 
31
  ## The Problem