Upload README.md with huggingface_hub

README.md

@@ -39,6 +39,8 @@ It answers with a single structured JSON verdict: is this vulnerable, which CWE,
   though the training mix is heavily C-weighted (see Limitations).
 - 🧪 **Honestly evaluated**: scored on the standard split *and* on vulnerable/patched function pairs,
   which is the harder and more meaningful test.
+- 🥊 **Benchmarked against a frontier model**: matches DeepSeek v4 Flash on genuine flaw
+  discrimination while being ~10,000× faster per snippet.
 - 💻 **Runs locally**: shares the HobbyLM architecture, so it loads in the from-scratch Rust CPU engine
   (`hobby-rs`) with no Python at runtime.
 
@@ -90,6 +92,59 @@ improvement is better calibration rather than deeper understanding. **27.40% is
 of true detection ability — not the 96.97% AUC.** Very few vulnerability models publish this number;
 it is here because it is the one that predicts real-world behaviour.
 
+## Comparison against a frontier model 🥊
+
+Benchmarked against **DeepSeek v4 Flash** (via OpenRouter, reasoning enabled) on the *same* held-out
+rows, through the *same* prompts, the *same* output parser and the *same* metrics. DeepSeek answers
+zero-shot; CodeAstra is fine-tuned on this corpus.
+
+### On the standard held-out split — 1,000 rows, 93 vulnerable
+
+| Metric | DeepSeek v4 Flash | **CodeAstra-500M** |
+|---|---|---|
+| JSON parse rate | 97.10% | **99.98%** |
+| Precision | 16.56% | **75.57%** |
+| Recall | 54.84% | **76.89%** |
+| F1 | 25.44% | **76.23%** |
+| Exact CWE | 10.75% | **~55%** |
+| Latency per snippet | ~47 s | **4.3 ms** |
+
+DeepSeek flagged 308 of 1,000 snippets as vulnerable when only 93 were — a 3.3× over-prediction, since
+a general model has no way to know this corpus is 89% safe.
+
+⚠️ **Do not read this as "CodeAstra is 3× better at security."** Most of that gap is *corpus
+familiarity* — the base rate and the annotator's CWE conventions — not better security reasoning. The
+per-CWE breakdown makes it plain: DeepSeek gets CWE-89 (SQL injection) 100% detected and 80% exact,
+but 0% exact on CWE-119, a catch-all bucket whose boundaries you can only learn from the training set.
+
+### On the paired set — 894 pairs, the unconfounded test
+
+| | DeepSeek v4 Flash | **CodeAstra-500M** |
+|---|---|---|
+| **P-C — flags flawed, clears patched** | 24.05% | **27.40%** |
+| P-V — flags both | 34.68% | 33.45% |
+| P-B — clears both | 32.33% | 36.02% |
+| P-R — reversed | 8.95% | **3.13%** |
+| Recall, vulnerable half | 58.72% | **60.85%** |
+| Specificity on patched code | 56.38% | **63.42%** |
+
+Here the 3× gap **disappears**. The two are effectively tied, with CodeAstra marginally ahead and 3×
+less likely to get a pair exactly backwards — which confirms that the standard-split gap was corpus
+familiarity rather than capability.
+
+**Both are weak in absolute terms, and the card should say so.** A model answering the two halves
+*independently* at its own observed flag rate would score 33.1% (DeepSeek) and 38.6% (CodeAstra).
+**Both score below their own independence baseline**, meaning their verdicts are correlated across a
+pair — they respond to what the code looks like, which is identical in both halves, rather than to the
+flaw, which is the only thing that differs.
+
+✅ **The defensible claim:** CodeAstra-500M **matches a frontier reasoning model on genuine flaw
+discrimination**, at ~10,000× lower latency, no per-call cost, and fully local. Not that it is better
+at security.
+
+*Caveat: 4.7% of DeepSeek's paired calls truncated mid-reasoning and are scored as "not vulnerable",
+which slightly overstates its specificity and understates its recall. It does not change the ranking.*
+
 ## Intended Use
 
 CodeAstra-500M is for developers, security researchers and code auditors who want a fast first-pass
@@ -210,9 +265,15 @@ evaluation uses 894 vulnerable/patched function pairs drawn from the same held-o
 HobbyLM-Chat comparison was run through the **identical** prompts and harness, so the two columns are
 directly comparable.
 
-Numbers on this page were not copied from other model cards, and no comparison against external models
-is claimed — CodeAstra-7B was evaluated on a different corpus and protocol, so the two are **not**
-directly comparable.
+The DeepSeek v4 Flash comparison was run through OpenRouter on the same held-out rows, receiving the
+identical system/user content in native chat format so it is not penalised for our decoder-only prompt
+convention, and scored with the same parser and metrics. Reasoning was enabled at low effort: at
+default effort the model spends ~98% of its output budget thinking and 37% of replies never emit an
+answer within 8k tokens, which would have measured our token cap rather than its accuracy.
+
+No numbers on this page were copied from another model card — everything here was measured directly.
+**CodeAstra-7B is not comparable** to these figures: it was evaluated on a different corpus under a
+different protocol, so its 83% accuracy and this card's numbers do not sit on the same scale.
 
 ## Citation 📜