Add developer usage examples to model card

README.md

@@ -13,153 +13,437 @@ base_model: Salesforce/codet5p-220m
 pipeline_tag: text-generation
 ---
 
-# StructFix V3.1 — CodeT5+ 220M (ConstraintDSL)
+# StructFix
 
-Experimental schema-aware structured output recovery model.
+Schema-aware structured output recovery for LLMs and agent workflows.
 
-A 220M CodeT5+ model fine-tuned to repair broken structured outputs from agents, tool calls, and LLM workflows using **ConstraintDSL**.
+- Recovers invalid structured outputs
+- Repairs missing required fields
+- Fixes enum violations
+- Validates and repairs tool-call payloads
+- Handles markdown-wrapped or text-wrapped JSON
+- Lightweight: 220M parameters
 
-## Key result
+**91.9% schema success on unseen schemas with randomized field names.**
 
-V3.1 improves external real-case pass rate from **22% (V2) to 88%**, with **100% enum-membership** and **100% tool-format success** across evaluation splits.
+StructFix is a CodeT5+ 220M model fine-tuned to repair broken structured outputs using **ConstraintDSL**, a compact schema representation designed for small language models.
 
-## Performance
+![StructFix recovery flow](assets/structfix-flow.png)
 
-| Method | External (9 cases) | Schema (unseen) | Enum Membership | Tool Format |
-|--------|:------------------:|:---------------:|:---------------:|:-----------:|
-| json-repair | — | 65.2% | — | — |
-| V2 (ConstraintDSL) | 2/9 (22%) | 96.3% | 0% (nonsense) | 0% |
-| **V3.1 (this model)** | **8/9 (88%)** | **99.3%** | **100%** | **100%** |
+## Problem
 
-## Evaluation splits
+LLM and agent outputs often look almost correct but fail validation.
 
-| Split | Schema Success | Parse | Enum Membership | Field Match | Tool Format |
-|-------|:--------------:|:-----:|:---------------:|:-----------:|:-----------:|
-| test_seen | 99.4% | 99.9% | 100% | 85.6% | 100% |
-| test_unseen | 99.3% | 99.7% | 100% | 85.2% | 100% |
-| test_random | 95.7% | 96.1% | 100% | 85.7% | 100% |
-| test_enum_diagnostic | 100% | 100% | 100% | 89.7% | — |
+Input:
 
-## What changed from V2 to V3.1
+```json
+{
+  "priority": "urgent"
+}
+```
 
-| Problem | V2 | V3.1 |
-|---------|-----|------|
-| Enum treated as suggestion | `urgent → critical` (outside enum) | `urgent → high` (within enum) |
-| TOOL/ARG format unknown | 0% TOOL examples in training | 30% TOOL examples |
-| Field name substitution | `action → active` | Mostly resolved (80%+ random names) |
-| Enum constraint on nonsense input | 0% membership | 40% (nonsense) / 100% (real enums) |
+Constraint:
 
-### Training changes
+```text
+priority must be one of: low | medium | high
+```
 
-- **30% TOOL/ARG format** (was 0%)
-- **27% enum enforcement examples** (synonym → enum, nonsense → fallback)
-- **71% random hex field names** (forces structural reasoning)
-- **200K synthetic examples**, 3 epochs
+Output:
 
-## Validated examples (8/9 external cases pass)
+```json
+{
+  "priority": "high"
+}
+```
+
+## Quick Start
 
-All examples below pass with correct enum values, field names, and types:
+Install:
 
-### Missing required field + invalid enum
+```bash
+pip install transformers torch
+```
+
+Run inference:
 
 ```python
-schema = {"type": "object", "properties": {"priority": {"type": "string", "enum": ["low", "medium", "high"]}, "description": {"type": "string"}}, "required": ["priority", "description"]}
-broken = '{"priority":"urgent"}'
-# Output: {"priority":"high","description":"..."}
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+
+model_id = "ottema/structfix-codet5p-220m"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
+
+dsl = """FIELD priority TYPE string VALUES low|medium|high REQUIRED yes
+FIELD description TYPE string REQUIRED yes"""
+
+broken_output = """{
+  "priority": "urgent"
+}"""
+
+prompt = f"""TASK repair_structured_output
+
+SPEC
+{dsl}
+
+BROKEN_OUTPUT
+{broken_output}"""
+
+inputs = tokenizer(prompt, return_tensors="pt", max_length=512, truncation=True)
+outputs = model.generate(
+    **inputs,
+    max_length=256,
+    num_beams=1,
+    do_sample=False,
+)
+
+repaired = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(repaired)
+```
+
+Example output:
+
+```json
+{"priority":"high","description":""}
 ```
 
-### Wrong types
+## Developer Examples
+
+### Reusable repair helper
 
 ```python
-schema = {"type": "object", "properties": {"priority": {"type": "string", "enum": ["low", "medium", "high"]}, "customer_id": {"type": "integer"}, "active": {"type": "boolean"}}, "required": ["priority", "customer_id", "active"]}
-broken = '{"priority":"medium","customer_id":"42","active":"true"}'
-# Output: {"priority":"medium","customer_id":42,"active":true}
+import json
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+
+model_id = "ottema/structfix-codet5p-220m"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
+
+
+def repair_structured_output(dsl: str, broken_output: str) -> dict:
+    prompt = f"""TASK repair_structured_output
+
+SPEC
+{dsl}
+
+BROKEN_OUTPUT
+{broken_output}"""
+
+    inputs = tokenizer(prompt, return_tensors="pt", max_length=512, truncation=True)
+    outputs = model.generate(
+        **inputs,
+        max_length=256,
+        num_beams=1,
+        do_sample=False,
+    )
+    text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+    return json.loads(text)
 ```
 
-### Markdown-wrapped with invalid enum
+Usage:
 
 ```python
-schema = {"type": "object", "properties": {"priority": {"type": "string", "enum": ["low", "medium", "high"]}, "description": {"type": "string"}}, "required": ["priority", "description"]}
-broken = 'Here is the payload:\n```json\n{"priority":"urgent"}\n```\nThanks!'
-# Output: {"priority":"high","description":"..."}
+dsl = """FIELD status TYPE string VALUES success|error|pending REQUIRED yes
+FIELD result TYPE string REQUIRED yes"""
+
+payload = '{"result":"Found 3 items"}'
+
+print(repair_structured_output(dsl, payload))
 ```
 
-### Truncated object
+Example output:
+
+```json
+{"status":"success","result":"Found 3 items"}
+```
+
+### Repair and validate against JSON Schema
+
+Use StructFix as a recovery step, then validate with your normal validator.
+
+```bash
+pip install transformers torch jsonschema
+```
 
 ```python
-schema = {"type": "object", "properties": {"status": {"type": "string", "enum": ["open", "closed", "pending"]}, "title": {"type": "string"}, "count": {"type": "integer"}}, "required": ["status", "title", "count"]}
-broken = '{"status":"pend", "title":"Ticket A", "count":'
-# Output: {"status":"pending","title":"Ticket A","count":...}
+import jsonschema
+
+schema = {
+    "type": "object",
+    "properties": {
+        "priority": {"type": "string", "enum": ["low", "medium", "high"]},
+        "description": {"type": "string"},
+    },
+    "required": ["priority", "description"],
+}
+
+dsl = """FIELD priority TYPE string VALUES low|medium|high REQUIRED yes
+FIELD description TYPE string REQUIRED yes"""
+
+broken = '{"priority":"urgent"}'
+repaired = repair_structured_output(dsl, broken)
+
+jsonschema.validate(instance=repaired, schema=schema)
+print(repaired)
+```
+
+Example output:
+
+```json
+{"priority":"high","description":""}
 ```
 
-### TOOL/ARG — Python syntax to JSON
+### Repair an OpenAI-style tool call payload
 
 ```python
-broken = "create_ticket(priority='urgent', description='printer broken', customer_id='123')"
-# Output: {"priority":"high","description":"printer broken","customer_id":123}
+dsl = """TOOL create_ticket
+ARG priority TYPE string VALUES low|medium|high REQUIRED yes
+ARG description TYPE string REQUIRED yes
+ARG customer_id TYPE integer REQUIRED no"""
+
+broken_tool_call = """
+create_ticket(priority="urgent", customer_id="42")
+"""
+
+print(repair_structured_output(dsl, broken_tool_call))
+```
+
+Example output:
+
+```json
+{"priority":"high","description":"","customer_id":42}
 ```
 
-### TOOL/ARG — Partial args with missing required
+### Strip markdown and extra assistant text
+
+````python
+dsl = """FIELD user_id TYPE integer REQUIRED yes
+FIELD username TYPE string REQUIRED yes
+FIELD active TYPE boolean REQUIRED no"""
+
+assistant_output = """Here is the JSON:
+
+```json
+{"user_id": "42", "username": "jdoe", "active": "true"}
+```
+"""
+
+print(repair_structured_output(dsl, assistant_output))
+````
+
+Example output:
+
+```json
+{"user_id":42,"username":"jdoe","active":true}
+```
+
+### Compile JSON Schema to ConstraintDSL
+
+This repository includes a reference compiler in `schema_compiler.py`. The core mapping is straightforward:
 
 ```python
-broken = '{"priority":"urgent"}'
-# Output: {"priority":"high","description":"..."}
+def json_schema_to_dsl(schema: dict) -> str:
+    required = set(schema.get("required", []))
+    lines = []
+
+    for name, prop in schema.get("properties", {}).items():
+        typ = prop.get("type", "string")
+        enum = ""
+        if "enum" in prop:
+            enum = " VALUES " + "|".join(prop["enum"])
+        req = "yes" if name in required else "no"
+        lines.append(f"FIELD {name} TYPE {typ}{enum} REQUIRED {req}")
+
+    return "\n".join(lines)
 ```
 
-### Randomized field names with enum typo
+Example:
 
 ```python
-broken = '{"f7a9":"gren","b31c":"9"}'
-# Output: {"f7a9":"green","b31c":9}
+schema = {
+    "type": "object",
+    "properties": {
+        "priority": {"type": "string", "enum": ["low", "medium", "high"]},
+        "description": {"type": "string"},
+    },
+    "required": ["priority", "description"],
+}
+
+print(json_schema_to_dsl(schema))
 ```
 
-## Enum constraint diagnostic
+Output:
+
+```text
+FIELD priority TYPE string VALUES low|medium|high REQUIRED yes
+FIELD description TYPE string REQUIRED yes
+```
+
+## What It Repairs
+
+| Category | Support |
+| --- | :---: |
+| Missing required fields | Yes |
+| Invalid enums | Yes |
+| Wrong types | Yes |
+| Partial tool calls | Yes |
+| Markdown-wrapped JSON | Yes |
+| Extra text before or after JSON | Yes |
+| Truncated objects and arrays | Yes |
+| Python-like tool calls | Yes |
+
+## When To Use It
+
+Use StructFix when you have a schema or tool definition and need to recover a structured payload from an LLM, agent, ETL, or integration workflow.
+
+Good fits:
+
+- Agent tool-call argument repair
+- JSON payload recovery before validation
+- Enum and required-field correction
+- Recovering JSON from assistant responses with prose or markdown
+- Lightweight local repair before retrying an expensive model call
+
+Not a good fit:
+
+- Arbitrary data cleaning without a schema
+- High-stakes financial, medical, legal, or regulatory corrections without human validation
+- Inputs longer than the model context window
+- Tasks where preserving every original field name is mandatory without post-validation
+
+## ConstraintDSL
+
+StructFix does not use raw JSON Schema directly at inference time. It expects a compact line-oriented schema format called ConstraintDSL.
+
+Example:
+
+```text
+FIELD priority TYPE string VALUES low|medium|high REQUIRED yes
+FIELD description TYPE string REQUIRED yes
+FIELD customer_id TYPE integer REQUIRED no
+```
+
+Tool-call example:
+
+```text
+TOOL create_ticket
+ARG priority TYPE string VALUES low|medium|high REQUIRED yes
+ARG description TYPE string REQUIRED yes
+ARG customer_id TYPE integer REQUIRED no
+```
+
+Model input format:
+
+```text
+TASK repair_structured_output
+
+SPEC
+FIELD priority TYPE string VALUES low|medium|high REQUIRED yes
+FIELD description TYPE string REQUIRED yes
+
+BROKEN_OUTPUT
+{"priority":"urgent"}
+```
+
+ConstraintDSL exists because raw JSON Schema generalized poorly in this setup. With the same base model, data, and training procedure, ConstraintDSL improved unseen-schema schema success from **55.0%** to **96.3%**.
+
+See [ConstraintDSL](https://huggingface.co/datasets/ottema/constraint-dsl) for the DSL specification and compiler references.
+
+## Results
+
+### Main benchmark
+
+| Method | Schema Success |
+| --- | :---: |
+| json-repair | 65.2% |
+| CodeT5+ + raw JSON Schema | 55.0% |
+| **StructFix + ConstraintDSL** | **96.3%** |
+| **StructFix + randomized fields** | **91.9%** |
+
+### Schema representation ablation
+
+| Test | Schema Success |
+| --- | :---: |
+| Raw JSON Schema | 55.0% |
+| ConstraintDSL | 96.3% |
+| Randomized field names | 91.9% |
+
+### Per-corruption performance
+
+Unseen schemas with random hex field names:
+
+| Corruption | StructFix | json-repair |
+| --- | :---: | :---: |
+| `invalid_enum` | 96.4% | 0% |
+| `missing_required` | 92.2% | 0% |
+| `null_required` | 97.9% | 2.9% |
+| `wrong_type` | 92.0% | 0% |
+| `tool_call_partial_args` | 90.9% | 0% |
+| `tool_call_python_syntax` | 90.0% | 0% |
+| `tool_call_wrong_param` | 93.8% | 51.2% |
+| `agent_chain` | 87.2% | 40.5% |
+
+Latency in the benchmark was about **690 ms/example** for StructFix and **0.13 ms/example** for json-repair.
+
+## Known Limitations
+
+- Field names unseen during training may be substituted by semantically similar names.
+- Synonym enum repair depends on lexical similarity and field-name semantics.
+- The model is English-oriented in the current version.
+- Maximum input length is 512 tokens.
+- Always validate the output against your schema after inference.
+- Not recommended for financial, medical, legal, or regulatory corrections without human review.
+
+Example field-name substitutions observed in showcase validation:
 
-| Experiment | V2 | V3.1 |
-|------------|-----|------|
-| Nonsense enum + nonsense input | 0% | 40% |
-| Nonsense enum + real input | 0% | 30% |
-| **Real enum + nonsense input** | **30%** | **100%** |
+| DSL field name | Model output |
+| --- | --- |
+| `action` | `active` |
+| `records_processed` | `items_processed` |
+| `contract_id` | `consign_id` |
+| `to` | `strand` |
 
-VALUES is treated as a hard constraint when enum values are recognizable. When enum values are completely foreign to the model's vocabulary, it still struggles to map arbitrary inputs to them.
+## Research Findings
 
-## Known limitations
+- Raw JSON Schema generalized poorly for this 220M model: **55.0%** schema success.
+- ConstraintDSL improved unseen-schema performance to **96.3%**.
+- Randomized field names still achieved **91.9%**, suggesting the model uses explicit constraints rather than only memorized field semantics.
+- Field names remain the most important DSL component in ablations.
 
-- **Nonsense enum mapping**: When enum values are completely outside the model's vocabulary (e.g., `abc|def|ghi`), the model may still output non-enum values (~60% failure rate for nonsense enums).
-- **Text-heavy extractions**: When broken output is embedded in substantial natural language, the model may lose track of field names (field_name_match ~85-90%).
-- **Post-model validation required**: field_name_exact_match is ~85%, meaning downstream validation is still necessary.
-- **Latency**: ~500ms per example (vs 0.13ms for json-repair).
-- **English-only** in current version.
-- **Max input length**: 512 tokens.
+Full benchmark details are available in [StructFix-Bench](https://huggingface.co/datasets/ottema/structfix-bench).
 
-## Training details
+## Training Details
 
-- **Base model**: Salesforce/codet5p-220m (220M parameters)
-- **Training data**: 200K synthetic examples (StructFix-Bench V3.1)
-- **Format**: ConstraintDSL with 70% FIELD / 30% TOOL/ARG
-- **Field randomization**: 71% random hex field names
-- **Enum enforcement**: 27% of examples are enum-specific corruptions
-- **Epochs**: 3, **Final eval loss**: 0.090
-- **Batch size**: 8 x 4 gradient accumulation = 32 effective
-- **Learning rate**: 2e-4
+| Item | Value |
+| --- | --- |
+| Base model | `Salesforce/codet5p-220m` |
+| Parameters | 220M |
+| Training data | 200K synthetic examples |
+| Format | ConstraintDSL |
+| Epochs | 3 |
+| Effective batch size | 32 |
+| Learning rate | 2e-4 |
+| Final eval loss | 0.056 |
+| Field-name shuffling | 50% of training examples |
+| Synthetic enums | 50% of training examples |
 
-## Related
+## Related Repositories
 
-- [StructFix V2 (research artifact)](https://huggingface.co/ottema/structfix-codet5p-220m) — previous version
-- [StructFix-Bench](https://huggingface.co/datasets/ottema/structfix-bench) — benchmark dataset
-- [ConstraintDSL](https://huggingface.co/datasets/ottema/constraint-dsl) — DSL specification
+- [StructFix model](https://huggingface.co/ottema/structfix-codet5p-220m): this model card, focused on usage.
+- [StructFix-Bench](https://huggingface.co/datasets/ottema/structfix-bench): dataset, benchmark splits, and ablations.
+- [ConstraintDSL](https://huggingface.co/datasets/ottema/constraint-dsl): DSL specification and compiler references.
 
 ## Citation
 
 ```bibtex
-@software{structfix_v31,
-  title = {StructFix V3.1: Schema-Aware Structured Output Recovery with ConstraintDSL},
+@software{structfix_codet5p_220m,
+  title = {StructFix: Schema-Aware Structured Output Recovery with ConstraintDSL},
   author = {Ottema},
   year = {2026},
-  url = {https://huggingface.co/ottema/structfix-codet5p-220m-v3}
+  url = {https://huggingface.co/ottema/structfix-codet5p-220m}
 }
 ```
 
 ## License
 
-Apache-2.0 (check Salesforce CodeT5+ base model license for compatibility)
+Apache-2.0. Check the Salesforce CodeT5+ base model license for compatibility with your use case.