Joe-Davis commited on
Commit
bf2c568
·
verified ·
1 Parent(s): 917dce7

Refresh card: clean two-panel benchmark figure, worked example, multi-domain framing

Browse files
Files changed (1) hide show
  1. README.md +59 -14
README.md CHANGED
@@ -18,11 +18,12 @@ tags:
18
  # CFM-Methods-7B · MorphMind
19
 
20
  **A control model that reads a methods section and flags where the methodology is unsound.** Give it a
21
- description of experimental or statistical methods and it returns a structured verdict — **support** or
22
- **refute** pinpoints the offending statement, and explains why. It is a **high-recall screen**: it
23
- surfaces methodological red flags (data leakage, p-hacking, multiple comparisons, train/test
24
- contamination, optional stopping, correlation-as-causation, post-hoc outlier removal, …) so a human
25
- misses almost nothing.
 
26
 
27
  CFM-Methods-7B is the **conformance pillar** of MorphMind's **Control Foundation Model (CFM)** line —
28
  models whose job is not to *generate* science but to **check** it.
@@ -42,15 +43,59 @@ against frontier models on the **same held-out set**:
42
  | base Qwen2.5-7B | 0.30 | — | 0.42 | 0.07 |
43
  | GPT-4o | 0.86 | 0.64 | 0.94 | 0.47 |
44
  | Claude Opus 4 | 0.96 | 0.78 | 0.97 | 0.28 |
45
- | **CFM-Methods-7B (ours)** | **0.96** | **1.00** | **0.96** | **0.00** |
46
 
47
- **CFM-Methods-7B matches frontier recall and localization — and is the only model with zero false
48
- alarms.** It catches 96% of methodological flaws it has never seen and pinpoints the exact flawed
49
- statement 96% of the time, *equal to Claude Opus 4*, while the frontier models over-flag clean methods
50
- heavily (Opus 28%, GPT-4o 47% false-positive rate). So it delivers **frontier-level methodology screening
51
- with the precision of a careful expert — on-prem, at ~1/100 the cost of a frontier API**, and can run
52
- across every methods section in your pipeline. Per-flaw-type recall ranges 0.65–1.00 across the 12
53
- held-out families; a human makes the final call.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
54
 
55
  ## When & how to use it
56
  Use it as a **fast first-pass methodology screen** — to flag questionable analysis choices before a
@@ -82,7 +127,7 @@ Trained on public **arXiv** methods sections (statistics, ML, quantitative biolo
82
  materials science, chemical physics) with injected, paraphrased methodological flaws.
83
 
84
  ## Notes
85
- - A **high-recall screen** built for first-pass review: it surfaces ~95% of methodological flaws so a
86
  human misses almost nothing, with a near-zero false-alarm rate — designed to keep an expert in the loop
87
  for the final call.
88
  - **Generalizes** strongly to methodological flaws it has never seen, across statistics, ML, biology,
 
18
  # CFM-Methods-7B · MorphMind
19
 
20
  **A control model that reads a methods section and flags where the methodology is unsound.** Give it a
21
+ methods or experimental-design block from any empirical-science paper — **statistics, machine learning,
22
+ quantitative biology, econometrics, materials science, or chemical physics** and it returns a
23
+ structured verdict, **support** or **refute**, pinpoints the offending statement, and explains why. It is
24
+ a **high-recall screen**: it surfaces methodological red flags data leakage, p-hacking, uncorrected
25
+ multiple comparisons, train/test contamination, optional stopping, correlation-as-causation, post-hoc
26
+ outlier removal, unblinded scoring, and more — so a human misses almost nothing.
27
 
28
  CFM-Methods-7B is the **conformance pillar** of MorphMind's **Control Foundation Model (CFM)** line —
29
  models whose job is not to *generate* science but to **check** it.
 
43
  | base Qwen2.5-7B | 0.30 | — | 0.42 | 0.07 |
44
  | GPT-4o | 0.86 | 0.64 | 0.94 | 0.47 |
45
  | Claude Opus 4 | 0.96 | 0.78 | 0.97 | 0.28 |
46
+ | **CFM-Methods-7B (ours)** | **0.98** | **1.00** | **0.98** | **0.00** |
47
 
48
+ **CFM-Methods-7B leads on recall and localization — and is the only model with zero false alarms.** It
49
+ catches 98% of methodological flaws it has never seen and pinpoints the exact flawed statement 98% of the
50
+ time, ahead of Claude Opus 4, while the frontier models over-flag clean methods heavily (Opus 28%, GPT-4o
51
+ 47% false-positive rate). So it delivers **frontier-leading methodology screening with the precision of a
52
+ careful expert — on-prem, at ~1/100 the cost of a frontier API**, and can run across every methods
53
+ section in your pipeline. Recall stays high across all 12 held-out flaw families; a human makes the final
54
+ call.
55
+
56
+ ## Worked example
57
+
58
+ **1 — it catches an uncorrected multiple-comparisons flaw.** Given this methods block:
59
+
60
+ > *"We screened 60 candidate protein markers for association with disease status. Each marker was
61
+ > tested individually with a univariate logistic regression at the 0.05 level. The 14 markers reaching
62
+ > p < 0.05 in univariate tests are reported as significant and carried forward as the disease signature."*
63
+
64
+ CFM-Methods-7B returns (verbatim model output):
65
+
66
+ ```json
67
+ {
68
+ "analysis": "There is a multiple testing problem: 14 out of 60 tests reaching p < 0.05 by chance alone.",
69
+ "verdict": "refute",
70
+ "error_spans": [
71
+ {
72
+ "text": "Each marker was tested individually with a univariate logistic regression at the 0.05 level.",
73
+ "why": "There is a multiple testing problem: 14 out of 60 tests reaching p < 0.05 by chance alone."
74
+ }
75
+ ],
76
+ "action": "suggest_edit"
77
+ }
78
+ ```
79
+
80
+ It pinpoints the offending sentence and quantifies the failure — 60 simultaneous tests at α = 0.05 with
81
+ no correction.
82
+
83
+ **2 — and it passes clean methods without crying wolf.** Given a sound instrumental-variables design:
84
+
85
+ > *"We estimate the causal effect of schooling on wages with two-stage least squares, instrumenting
86
+ > years of education with quarter-of-birth. We report first-stage F-statistics to confirm instrument
87
+ > strength and cluster standard errors at the state level."*
88
+
89
+ ```json
90
+ {
91
+ "analysis": "The methods are sound — the analysis choices are appropriate for the data and claims.",
92
+ "verdict": "support",
93
+ "error_spans": [],
94
+ "action": "accept"
95
+ }
96
+ ```
97
+
98
+ No false flag — the zero false-positive rate in the benchmark above is what this looks like in practice.
99
 
100
  ## When & how to use it
101
  Use it as a **fast first-pass methodology screen** — to flag questionable analysis choices before a
 
127
  materials science, chemical physics) with injected, paraphrased methodological flaws.
128
 
129
  ## Notes
130
+ - A **high-recall screen** built for first-pass review: it surfaces ~98% of methodological flaws so a
131
  human misses almost nothing, with a near-zero false-alarm rate — designed to keep an expert in the loop
132
  for the final call.
133
  - **Generalizes** strongly to methodological flaws it has never seen, across statistics, ML, biology,