README.md

---
library_name: transformers
license: apache-2.0
language:
- en
pipeline_tag: text-classification
tags:
- anomaly-detection
- hallucination-detection
- llm-safety
- streaming
- corruption-detection
- safetensors
- fine-tuning
model-index:
- name: simurg-pulse
  type: text-classification
  results:
    - task:
        type: text-classification
        name: streaming hallucination detection
      dataset:
        name: CorruptBench + live wahoo-1.5-preview
        type: corruptions
      metrics:
        - type: auroc
          name: held-out AUROC
          value: 0.925
          verified: false
---

# SIMURG

**SIMURG is a streaming hallucination and output-degradation detector for
LLMs.** It watches a live model response as tokens stream in and raises a
calibrated alarm the moment the output degenerates: repetition loops,
cross-lingual drift, regurgitation, and structural collapse — the failure
modes that ship hallucinated or garbage text to users. This checkpoint is the
learned deep tier of the SIMURG ensemble; it runs in single-digit
milliseconds on Apple Silicon and never blocks the stream it guards.

- Repository: https://github.com/doofzoff/SIMURG
- Paper: https://ssrn.com/abstract=7451269

## Can you train SIMURG on your own hallucination types?

**Yes.** The detector learns from any (clean, corrupted) pairs you can
produce. If your workload has a characteristic failure mode — fabricated
citations, number drift, prompt echo, domain-specific garbage — collect or
synthesize examples of it, point the trainer at your endpoint, and the
ensemble picks up the new checkpoint automatically. See the
[Fine-Tuning Tutorial](#fine-tuning-tutorial--train-simurg-on-your-own-hallucinations)
below.

## Architecture

| Component | Specification |
|:---|:---|
| Backbone | 2x TransformerEncoder (norm-first, dropout 0.05) |
| Hidden dimension | 64 |
| Attention heads | 4 |
| Feedforward | 256 (4x hidden), ReLU |
| Vocabulary | 4096 trigram-hash buckets (index 0 reserved for left padding) |
| Sequence length | 256 trigrams (~600-character tail window) |
| Positional encoding | learnable parameters |
| Pooling | masked mean over non-padding positions |
| Head | LayerNorm -> Linear(64, 1), single logit |
| Parameters | 345,665 |
| Checkpoint size | ~1.3 MB (safetensors) |

### Formulation

A checkpoint observes the stream prefix $x_1\dots x_t$ and reads only the
recent window $w_t = x_{\max(1,\,t-600)}\dots x_t$. The window is tokenized
into character trigrams and mapped deterministically into a 4096-bucket
vocabulary:

$$
\tau_i = \mathrm{blake2b}(x_i x_{i+1} x_{i+2}) \bmod (V - 1) + 1, \qquad V = 4096
$$

After embedding $E \in \mathbb{R}^{V \times 64}$ and a learnable positional
bias $P$, two norm-first transformer layers with masked self-attention
produce the representation

$$
H = \mathrm{Transformer}\big(E[\tau] + P\big), \qquad
h = \frac{\sum_i m_i\, H_i}{\sum_i m_i},
$$

where $m_i \in \{0,1\}$ masks padding positions. The head outputs a logit

$$
z = W\,\mathrm{LN}(h) + b,
$$

and the sigmoid score is linearly calibrated to a probability using two
anchors stored in the checkpoint metadata:

$$
\hat{p}_\mathrm{corrupt}(w_t) = \mathrm{clip}\!\left(
\frac{\sigma(z) - \ell_o}{h_i - \ell_o},\; 0,\; 1\right),
\qquad
\ell_o = Q_{0.95}\big[\text{clean scores}\big],\;
 h_i = \mathrm{median}\big[\text{corrupt scores}\big].
$$

Calibration anchors for this checkpoint: $\ell_o = 0.168$, $h_i = 0.367$.

### Training objective

Training minimizes class-weighted binary cross-entropy over onset-aware
windows:

$$
\mathcal{L} = -\frac{1}{N}\sum_{n=1}^{N}
\Big[
 y_n \, \alpha \, \log \sigma(z_n)
 + (1-y_n)\, \log\big(1 - \sigma(z_n)\big)
\Big],
\qquad
\alpha = \frac{N_-}{N_+}
$$

with $N_-$ and $N_+$ the counts of clean and corrupt windows. A window is
labeled corrupt only when its right edge lies at least $\Delta = 300$
characters past the true corruption onset; windows before the onset are
labeled clean. Optimizer: AdamW ($\eta = 5\times10^{-4}$, weight decay
$10^{-4}$), batch 64, 8 epochs; the best validation checkpoint is kept.

## Training Data

| Split | Source | Count |
|:---|:---|:---:|
| Clean (train) | live answers from the guarded endpoint (wahoo-1.5-preview via vLLM) plus a bundled clean corpus | 40 live + pooled |
| Corrupt (train) | CorruptBench synthetic corruptions: repetition loops, cross-lingual drift, table echo, structural garbage | 240 streams |
| Validation | held-out clean streams + 40 held-out corruptions | onset-aware windows |

## Results

![ROC curve](roc_curve.png)

*Figure: ROC re-evaluated on a held-out split at release time (292 onset-aware
windows, 9 held-out live clean documents). The headline training-time held-out
AUROC reported by the trainer is 0.925; per-split variation is expected with a
small clean set.*

| Metric | Value |
|:---|:---:|
| Held-out AUROC | 0.925 |
| Calibration anchors | $\ell_o = 0.168$, $h_i = 0.367$ |
| Inference latency | ~4 ms per window on MPS |
| CPU fallback | supported |

![Score distribution](score_distribution.png)

![Per-class response](per_class.png)

| Input | Pulse probability |
|:---|:---:|
| Clean prose | 0.000 |
| Repetition loop | 1.000 |
| Cross-lingual drift | 1.000 |
| Structural table echo | 1.000 |

![Training curves](training_curves.png)

## Production Readiness

- **Deterministic:** tokenization uses a stable blake2b hash, so training and
  live inference over the same text produce identical tensors.
- **Graceful degradation:** the detector is optional. Without torch,
  safetensors, or the weights file, the ensemble behaves exactly as the
  numpy-only core.
- **Lightweight:** 1.3 MB checkpoint, single-digit millisecond inference,
  far below the token rate of any guarded model.

## Usage Inside SIMURG

The weights ship with the SIMURG package and auto-register as the sixth
detector in the ensemble. Install with:

```
pip install "simurg[deep]"
```

To point SIMURG at a custom or retrained checkpoint:

```
export SIMURG_PULSE_WEIGHTS=/path/to/simurg_pulse.safetensors
```

## Standalone Inference

```
from safetensors.torch import load_file

tensors = load_file("simurg_pulse.safetensors")
```

The full inference plumbing (tokenization, windowing, calibration) lives in
SIMURG under `src/simurg/deep/pulse.py`; this repository ships the raw
checkpoint and the training recipe.

## Fine-Tuning Tutorial — train SIMURG on your own hallucinations

Retrain the checkpoint against any OpenAI-compatible endpoint in one command.
The trainer samples live answers (clean labels), synthesizes corruptions
(label 1 with known onset), trains on CPU in seconds, computes held-out AUROC
and calibration anchors, and writes a new safetensors file.

### Step 1: Install dependencies

```
pip install "simurg[deep]"
```

### Step 2: Configure the target endpoint

```
export SIMURG_LIVE_URL=http://your-host:port/v1/chat/completions
export SIMURG_LIVE_MODEL=your-model-name
```

For custom prompts, prepare a JSONL file where each line is
`{"text": "your prompt"}` and set:

```
export SIMURG_LIVE_PROMPTS_JSONL=prompts.jsonl
```

### Step 3: Run the trainer

```
python -m simurg.deep.train_pulse --clean 40 --corrupt 240 --epochs 8 --out ./simurg_pulse.safetensors
```

Useful flags:

| Flag | Default | Meaning |
|:---|:---:|:---|
| --clean | 40 | number of live clean answers to sample |
| --corrupt | 240 | number of synthetic corruption streams |
| --epochs | 8 | training epochs |
| --batch | 64 | batch size |
| --seed | 7 | RNG seed (reproducible runs) |
| --device | cpu | cpu, mps, or cuda |
| --out | bundled path | output safetensors path |

### Step 4: Verify the new checkpoint

The trainer prints held-out AUROC and calibration anchors before saving.
A strong fine-tune typically reaches held-out AUROC above 0.90. If AUROC
drops below ~0.75, increase --clean or add more diverse prompts to the JSONL.

### Step 5: Ship it

Replace the bundled file (`src/simurg/weights/simurg_pulse.safetensors`) or
keep the new file external and export `SIMURG_PULSE_WEIGHTS` in production.
The ensemble picks it up automatically on the next process start.

## Links

- Code: https://github.com/doofzoff/SIMURG
- Paper: https://ssrn.com/abstract=7451269

## License

Apache-2.0, matching the parent SIMURG project.

## Attribution

Trained as part of SIMURG v1.0.4 (2026-09-21) by MergenAI.