Upload README.md with huggingface_hub

README.md

@@ -1,24 +1,136 @@
 ---
 license: apache-2.0
+language:
+- en
+metrics:
+- accuracy
+- f1
+- roc_auc
 library_name: safetensors
+tags:
+- code
+- security
+- vulnerability-detection
+- cwe
+- owasp
+- moe
+- hobbylm
 pipeline_tag: text-generation
-tags: [security, vulnerability-detection, cwe, owasp, code, moe, hobbylm]
 ---
+# CodeAstra-500M: Laptop-Scale Vulnerability Detection 🔍🛡️
 
-# CodeAstra-500M
+## Model Description
 
-A 500M-parameter sparse-MoE **code vulnerability detector**, fine-tuned from
-[rootxhacker/HobbyLM-Chat](https://huggingface.co/rootxhacker/HobbyLM-Chat) on the full
-[`ayshajavd/code-security-vulnerability-dataset`](https://huggingface.co/datasets/ayshajavd/code-security-vulnerability-dataset)
-(140,335 train / 17,542 test rows, 100% coverage — nothing subsampled).
+CodeAstra-500M is the small sibling of [CodeAstra-7B](https://huggingface.co/rootxhacker/CodeAstra-7B) — a
+500M-parameter **sparse Mixture-of-Experts** model fine-tuned for security vulnerability detection in
+source code. Where CodeAstra-7B is built on Mistral-7B, CodeAstra-500M is built on
+[HobbyLM-Chat](https://huggingface.co/rootxhacker/HobbyLM-Chat), a MoE language model trained from
+scratch on a hobby budget — so the whole thing runs on a laptop CPU.
 
-Given a code snippet it returns one JSON object: whether the code is vulnerable, the CWE, and the
-OWASP Top-10 category.
+It answers with a single structured JSON verdict: is this vulnerable, which CWE, which OWASP category.
 
-## Prompt format
+### Key Features
 
-The model is format-sensitive — it was trained on exactly this layout (note the inner fence is part
-of the prompt):
+- 🪶 **Tiny**: 500M total parameters, only ~150M active per token thanks to top-6-of-36 expert routing.
+- 📐 **Structured output**: emits parseable JSON on 99.98% of inputs — no regex-scraping prose.
+- 🎚️ **Tunable**: returns a calibrated probability, so you pick the precision/recall trade-off at
+  inference time instead of retraining.
+- 🌐 **Multi-language**: C, C++, Python, Java, JavaScript, PHP, Go, Ruby, Swift, Kotlin, C#, Fortran —
+  though the training mix is heavily C-weighted (see Limitations).
+- 🧪 **Honestly evaluated**: scored on the standard split *and* on vulnerable/patched function pairs,
+  which is the harder and more meaningful test.
+- 💻 **Runs locally**: shares the HobbyLM architecture, so it loads in the from-scratch Rust CPU engine
+  (`hobby-rs`) with no Python at runtime.
+
+## Performance 📊
+
+Evaluated on the held-out test split (17,542 snippets, 1,887 vulnerable) of
+[`ayshajavd/code-security-vulnerability-dataset`](https://huggingface.co/datasets/ayshajavd/code-security-vulnerability-dataset).
+The base model is un-finetuned HobbyLM-Chat under identical prompts.
+
+| Metric | HobbyLM-Chat (base) | **CodeAstra-500M** |
+|---|---|---|
+| JSON parse rate | 1.20% | **99.98%** |
+| ROC AUC | — | **96.97%** |
+| Average precision | — | **82.64%** |
+| Best F1 | 0.00% | **76.23%** |
+| Precision / Recall @ best F1 | 0.00 / 0.00 | **75.57% / 76.89%** |
+| Exact CWE (28 classes) | 0.00% | **~55%** |
+| OWASP category | 0.00% | **~74%** |
+
+⚠️ **A note on accuracy.** This dataset is 89% non-vulnerable, so a model that answers "safe" every
+time scores **90.5% accuracy** — which is exactly what the un-finetuned base model does, at 0% recall.
+That is why this card leads with F1, AUC and recall rather than accuracy. Treat any headline accuracy
+figure on this dataset with suspicion, including for other models.
+
+### Paired evaluation — the number that actually matters 🎯
+
+Standard splits of CVE-derived vulnerability datasets are **confounded**: the vulnerable functions come
+from big C projects (Linux kernel, Chromium, PHP, ffmpeg) while the "safe" ones are often unrelated
+code. A model can score very well by recognising *code style* rather than *code flaws*.
+
+So CodeAstra-500M is also evaluated PrimeVul-style, on 894 pairs of a vulnerable function and **its own
+patched version** — near-identical code differing only by the security fix.
+
+| | First SFT pass | **CodeAstra-500M** |
+|---|---|---|
+| **P-C — flags the flawed one, clears the patched one** | 6.94% | **27.40%** |
+| P-V — flags both (the style shortcut) | 84.90% | **33.45%** |
+| P-B — clears both | 6.71% | 36.02% |
+| P-R — reversed | 1.45% | 3.13% |
+| Specificity on patched code | 13.65% | **63.42%** |
+| Within-pair ranking accuracy | 66.22% | **69.46%** |
+
+The released model was produced by feeding the ~7,000 patched functions back in as *safe* training
+examples — minimal pairs that differ only by the fix. This quadrupled paired accuracy and lifted
+specificity on patched code from 13.65% to 63.42%.
+
+**Read this honestly:** within-pair ranking accuracy moved only 66.22% → 69.46%, meaning most of the
+improvement is better calibration rather than deeper understanding. **27.40% is the realistic estimate
+of true detection ability — not the 96.97% AUC.** Very few vulnerability models publish this number;
+it is here because it is the one that predicts real-world behaviour.
+
+## Intended Use
+
+CodeAstra-500M is for developers, security researchers and code auditors who want a fast first-pass
+triage filter that runs locally — in a pre-commit hook, a CI step, or an editor plugin — without
+sending source code to an API. It is a **filter that decides what a human looks at**, not an oracle.
+
+## Threshold — please read 🎚️
+
+The model returns a probability. **The default 0.5 is not the balanced operating point.**
+
+| Threshold | Precision | Recall | Use case |
+|---|---|---|---|
+| 0.3346 | 75.57% | 76.89% | **Balanced (best F1 = 76.23%)** |
+| 0.3775 | 80.02% | 70.27% | Triage |
+| 0.4378 | 85.03% | 59.88% | |
+| 0.5156 | 90.01% | 45.36% | CI gating |
+| 0.6514 | 95.07% | 23.53% | High-confidence alerts only |
+
+At the naive 0.5 you get 88.79% precision but only **48.70% recall** — it will quietly miss half the
+vulnerabilities. Set the threshold deliberately.
+
+## Training 🏋️‍♂️
+
+Fine-tuned from HobbyLM-Chat on 8×H100 GPUs via [Modal](https://modal.com), using the **full** dataset —
+all 140,335 training rows, nothing subsampled, with over-long code head+tail truncated rather than
+dropped.
+
+| | |
+|---|---|
+| Main SFT | 6,000 steps, lr 2e-5, micro-batch 8 × 8 GPUs, 176,216 examples, ~27 min |
+| Hard-negative pass | 500 steps, lr 5e-6, 7,023 minimal pairs, ~2.5 min |
+| Objective | next-token CE masked to the JSON verdict; MoE aux-free balancing bias frozen |
+| Class balance | vulnerable rows oversampled 3× (10.9% → 26.0% positives) |
+
+Architecture is unchanged from HobbyLM: 768 hidden / 16 layers, 36 experts with top-6 routing plus one
+shared expert, GQA attention with per-head QK-norm, RoPE, GPT-2 byte-level BPE, 2048-token context.
+
+## Usage 💻
+
+The model uses the HobbyLM MoE architecture, so it needs the `hobbylm` package rather than
+`transformers`. The prompt format matters — the model was trained on exactly one layout:
 
 ````text
 SYSTEM: You are a source-code security auditor. Given a code snippet, decide whether it contains a security vulnerability and reply with one JSON object: {"vulnerable": bool, "cwe": str, "cwe_name": str, "owasp": str}. Use "none" for safe code.
@@ -31,114 +143,110 @@ Language: C
 ASSISTANT:
 ````
 
-`hobbylm.security_data.detect_prompt(code, language)` builds this for you — use it rather than
-hand-assembling the string.
-
-It answers:
-
-```json
-{"vulnerable": true, "cwe": "CWE-89", "cwe_name": "Improper Neutralization of Special Elements used in an SQL Command (SQL Injection)", "owasp": "A03: Injection"}
-```
-
-## Usage
+Use `detect_prompt()` rather than assembling that by hand:
 
 ```python
-import json, torch
+import json, torch, tiktoken
 from huggingface_hub import hf_hub_download
 from safetensors.torch import load_file
-from hobbylm.config import ModelConfig          # github.com/<your>/moe-lab
+from hobbylm.config import ModelConfig
 from hobbylm.model import MoETransformer
 from hobbylm.security_data import detect_prompt, VERDICT_PREFIX, TRUE_ID, FALSE_ID
-import tiktoken
 
 repo = "rootxhacker/codeastra-500M"
 cfg_d = json.load(open(hf_hub_download(repo, "config.json")))
 cfg_d.pop("preset", None)
 model = MoETransformer(ModelConfig(**cfg_d)).cuda().eval()
 model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")))
-
 enc = tiktoken.get_encoding("gpt2")
-prompt = detect_prompt(my_code, "C") + VERDICT_PREFIX          # force the verdict position
+
+code_to_analyze = """
+$query = $_GET['query'];
+$stmt = $db->prepare($query);
+$stmt->execute();
+"""
+
+# Fast path: force the verdict position and read one probability (~4ms, no generation)
+prompt = detect_prompt(code_to_analyze, "PHP") + VERDICT_PREFIX
 ids = torch.tensor([enc.encode_ordinary(prompt)], device="cuda")
 with torch.no_grad():
     logits, _ = model(ids)
 p_vuln = torch.softmax(logits[0, -1, [TRUE_ID, FALSE_ID]].float(), -1)[0].item()
-print(p_vuln >= 0.3346)      # see the threshold note below
+print(f"P(vulnerable) = {p_vuln:.3f}  ->  {'VULNERABLE' if p_vuln >= 0.3346 else 'safe'}")
 ```
 
-Reading `p_vuln` in one forward pass is ~26x faster than generating the JSON, and it gives you a
-tunable score instead of a fixed decision. Generating the full JSON works too if you want the CWE.
-
-## Threshold — please read
-
-**The default 0.5 is not the balanced operating point.** Pick deliberately:
-
-| threshold | precision | recall | use case |
-|---|---|---|---|
-| 0.3346 | 75.57% | 76.89% | balanced (best F1 = 76.23%) |
-| 0.3775 | 80.02% | 70.27% | |
-| 0.4378 | 85.03% | 59.88% | |
-| 0.5156 | 90.01% | 45.36% | CI gating |
-| 0.6514 | 95.07% | 23.53% | high-confidence only |
+Scoring this way is ~26× faster than generating the JSON (4.3 ms vs 110 ms per snippet) and gives you
+the tunable probability. If you also want the CWE and OWASP labels, generate the completion normally
+from `detect_prompt(code, lang)` and parse the JSON with `hobbylm.security_data.parse_verdict`.
 
-At the naive 0.5 you get 88.79% precision but only 48.70% recall — a conservative detector.
+A typical answer:
 
-## Evaluation
-
-All numbers are from the held-out test split (17,542 rows, 1,887 vulnerable), scored by this repo's
-own harness. Base model = un-finetuned HobbyLM-Chat, identical prompts.
-
-| metric | HobbyLM-Chat | CodeAstra-500M |
-|---|---|---|
-| JSON parse rate | 1.20% | 99.98% |
-| ROC AUC | — | 96.97% |
-| average precision | — | 82.64% |
-| best F1 | 0.00% | 76.23% |
-| exact CWE (of 28 classes) | 0.00% | ~55% |
-| OWASP category | 0.00% | ~74% |
-
-### Paired evaluation — the number that matters
+```json
+{"vulnerable": true, "cwe": "CWE-89", "cwe_name": "Improper Neutralization of Special Elements used in an SQL Command (SQL Injection)", "owasp": "A03: Injection"}
+```
 
-Standard splits of CVE-derived datasets are confounded: vulnerable functions come from large C
-projects (kernel, Chromium, PHP) while "safe" ones are often unrelated code, so a model can score
-well by recognising *style* rather than *flaws*. We therefore also evaluate PrimeVul-style, on 894
-pairs of a vulnerable function and **its own patched version**:
+## Limitations ⚠️
+
+1. **Function-level only.** It sees a single function, so interprocedural and data-flow vulnerabilities
+   are largely invisible. Access-control bugs needing caller context are its weakest class.
+2. **Heavily C-weighted training data** (92% C). Other languages work but are out-of-distribution —
+   expect lower reliability on Go, Swift, Kotlin and TypeScript.
+3. **CWE labels confuse related classes.** CWE-89 (SQL injection, 85% exact) and CWE-94 (code
+   injection, 82%) are strong; catch-all buckets like CWE-399 and CWE-416 are weak. It frequently finds
+   the right bug and picks a sibling CWE — the detection is better than the label suggests.
+4. **Multiple vulnerabilities in one snippet** are not reliably enumerated; it returns a single verdict.
+5. **Long code is truncated.** Snippets beyond 2048 tokens are head+tail truncated, so the middle of
+   very long functions is unseen.
+6. **False positives are expected** at the recall-oriented thresholds. Results need human verification.
+7. **It is a 500M research model, not a security product.** Use it alongside code review and SAST, not
+   instead of them, and do not gate a release on it alone.
+
+## Test Apparatus
+
+All figures come from the held-out test split of `ayshajavd/code-security-vulnerability-dataset`
+(17,542 snippets, never trained on), scored with a purpose-built harness that generates the verdict and
+parses it, plus the calibrated single-forward-pass scorer for threshold-free metrics. The paired
+evaluation uses 894 vulnerable/patched function pairs drawn from the same held-out split. The base
+HobbyLM-Chat comparison was run through the **identical** prompts and harness, so the two columns are
+directly comparable.
+
+Numbers on this page were not copied from other model cards, and no comparison against external models
+is claimed — CodeAstra-7B was evaluated on a different corpus and protocol, so the two are **not**
+directly comparable.
+
+## Citation 📜
 
-| | first SFT pass | CodeAstra-500M |
-|---|---|---|
-| **P-C — flags flawed, clears patched** | 6.94% | **27.40%** |
-| P-V — flags both (style shortcut) | 84.90% | **33.45%** |
-| P-B — clears both | 6.71% | 36.02% |
-| specificity on patched code | 13.65% | **63.42%** |
-| within-pair ranking accuracy | 66.22% | 69.46% |
+```
+@software{CodeAstra-500M,
+  author = {Harish Santhanalakshmi Ganesan},
+  title = {CodeAstra-500M: Laptop-Scale Vulnerability Detection},
+  year = {2026},
+  howpublished = {\url{https://huggingface.co/rootxhacker/codeastra-500M}}
+}
+```
 
-The published model was produced by adding the ~7,000 patched functions back as *safe* training
-examples — minimal pairs that differ only by the fix. That quadrupled paired accuracy and raised
-specificity on patched code from 13.65% to 63.42%.
+## License 📄
 
-**Be honest about what this means:** within-pair ranking accuracy only moved 66.22% → 69.46%, so most
-of the gain is better calibration rather than deeper understanding. **27.40% paired accuracy is the
-realistic estimate of true detection ability — not the 96.97% AUC.**
+CodeAstra-500M is released under the Apache License 2.0.
 
-## Training
+```
+Copyright 2026 [Harish Santhanalakshmi Ganesan]
 
-8xH100 on [Modal](https://modal.com). Main SFT: 6,000 steps, lr 2e-5, micro-batch 8, 176,216
-examples (~27 min). Hard-negative pass: 500 steps, lr 5e-6 (~2.5 min). Loss masked to the completion;
-MoE aux-free balancing bias frozen. Architecture is unchanged from HobbyLM: 768 hidden / 16 layers,
-36 experts top-6 + 1 shared, GQA with per-head QK-norm, GPT-2 BPE, 2048 context.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
 
-## Limitations
+    http://www.apache.org/licenses/LICENSE-2.0
 
-- **Function-level only.** It sees one function; interprocedural and data-flow bugs are largely
-  invisible to it. Access-control flaws needing caller context are its weakest class.
-- **92% of training data is C.** Treat other languages as out-of-distribution.
-- **CWE labels confuse related classes.** CWE-89 and CWE-94 are strong (85%/82% exact); catch-all
-  buckets like CWE-399 and CWE-416 are weak. It often finds the bug and picks a sibling CWE.
-- **Training labels are noisy.** Sources include mislabeled languages and prose spliced into code.
-- **Code over 2048 tokens is head+tail truncated**, so the middle of long functions is unseen.
-- **This is a 500M research model, not a security product.** It complements review and SAST; it does
-  not replace them. Do not gate a release on it alone.
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+```
 
-## License
+## Acknowledgements 🙏
 
-Apache-2.0.
+Thanks to the HobbyLM project for the 500M MoE base model, and to
+[@ayshajavd](https://huggingface.co/ayshajavd) for compiling the vulnerability dataset this model was
+trained on.