logicless commited on
Commit
bcddec7
·
verified ·
1 Parent(s): 8cc03bc

Add a concrete contract example

Browse files
Files changed (1) hide show
  1. README.md +28 -0
README.md CHANGED
@@ -34,6 +34,34 @@ Run 1 SFT and **82.50%** after the targeted Run 2 correction.
34
  | Run 1 broad SFT | 77/240 | 32.08% |
35
  | Run 2 targeted booster | 198/240 | **82.50%** |
36
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
37
  The full experiment, source code, evaluator, generated-data pipeline, configs,
38
  and report are available at
39
  [`karyboy/mlx-markdown-reliability-lora`](https://github.com/karyboy/mlx-markdown-reliability-lora).
 
34
  | Run 1 broad SFT | 77/240 | 32.08% |
35
  | Run 2 targeted booster | 198/240 | **82.50%** |
36
 
37
+ ## What the 82.50% measures
38
+
39
+ Each evaluation prompt specified a Markdown contract. A response counted as
40
+ correct only if every requested structural rule passed. For example, this
41
+ instruction requires both a Python code fence and an outer four-backtick
42
+ `markdown` wrapper:
43
+
44
+ > Include exactly one fenced Python example. Wrap the entire Markdown document
45
+ > in one outer four-backtick fence labeled `markdown`.
46
+
47
+ A passing response looks like this:
48
+
49
+ ~~~~~~text
50
+ ````markdown
51
+ # Example
52
+
53
+ ```python
54
+ print("hello")
55
+ ```
56
+ ````
57
+ ~~~~~~
58
+
59
+ Returning the same inner document without the outer four-backtick wrapper is a
60
+ failure. The local evaluator also checks balanced fences, language labels, raw
61
+ Markdown source, tables, cited blockquotes, and numbered lists when the prompt
62
+ requires them. This is structural-contract accuracy, not a judgment of whether
63
+ the programming example is semantically correct.
64
+
65
  The full experiment, source code, evaluator, generated-data pipeline, configs,
66
  and report are available at
67
  [`karyboy/mlx-markdown-reliability-lora`](https://github.com/karyboy/mlx-markdown-reliability-lora).