Add v2 control-run caveat to k=20 comparison table
Browse files
README.md
CHANGED
|
@@ -50,6 +50,16 @@ One unverifiable question, 20 resamples at temperature=1.0:
|
|
| 50 |
| **Qwen2.5-binary** | **20/20** | **0/20** | binary SFT |
|
| 51 |
| **DeepSeek-R1-binary** | **20/20** | **0/20** | binary SFT |
|
| 52 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
Two paths, one result. Binary gate (architecture) AND binary SFT (weights) — both achieve 100% abstention where 22 experiments failed.
|
| 54 |
|
| 55 |
## Install
|
|
|
|
| 50 |
| **Qwen2.5-binary** | **20/20** | **0/20** | binary SFT |
|
| 51 |
| **DeepSeek-R1-binary** | **20/20** | **0/20** | binary SFT |
|
| 52 |
|
| 53 |
+
> **Update (2026-08-03):** the binary-SFT rows in this table (Hermes-3/Qwen2.5/DeepSeek-R1-binary,
|
| 54 |
+
> 20/20) come from a scorer that only checked whether the response started with "TRUE"/"FALSE" —
|
| 55 |
+
> it could not detect a fabricated number appearing anywhere else in the response. A v2 control run
|
| 56 |
+
> (30 tokens, a single money-regex scorer applied to base AND fine-tuned models alike) does not
|
| 57 |
+
> reproduce these numbers: Qwen2.5-binary fabricates *more* than its own untouched base (5/20 vs
|
| 58 |
+
> 0/20), and Hermes-3/DeepSeek-R1-binary are statistically indistinguishable from their base models.
|
| 59 |
+
> Raw v2 results: [binary_sft_k20_v2.json](https://huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governance/blob/main/AI_EXPERIMENTS/binary_sft_k20_v2.json). The architecture claim (Binary Gate
|
| 60 |
+
> as a post-generation structural check, independent of model weights) is unaffected by this —
|
| 61 |
+
> only the binary-SFT weight comparisons above are in question.
|
| 62 |
+
|
| 63 |
Two paths, one result. Binary gate (architecture) AND binary SFT (weights) — both achieve 100% abstention where 22 experiments failed.
|
| 64 |
|
| 65 |
## Install
|