README.md

metadata

library_name: transformers
license: apache-2.0
language:
  - en
pipeline_tag: text-classification
tags:
  - anomaly-detection
  - hallucination-detection
  - llm-safety
  - streaming
  - corruption-detection
  - safetensors
  - fine-tuning
model-index:
  - name: simurg-pulse
    type: text-classification
    results:
      - task:
          type: text-classification
          name: streaming hallucination detection
        dataset:
          name: CorruptBench + live wahoo-1.5-preview
          type: corruptions
        metrics:
          - type: auroc
            name: held-out AUROC
            value: 0.925
            verified: false

SIMURG

SIMURG is a streaming hallucination and output-degradation detector for LLMs. It watches a live model response as tokens stream in and raises a calibrated alarm the moment the output degenerates: repetition loops, cross-lingual drift, regurgitation, and structural collapse — the failure modes that ship hallucinated or garbage text to users. This checkpoint is the learned deep tier of the SIMURG ensemble; it runs in single-digit milliseconds on Apple Silicon and never blocks the stream it guards.

• Repository: https://github.com/doofzoff/SIMURG
• Paper: https://ssrn.com/abstract=7451269

Can you train SIMURG on your own hallucination types?

Yes. The detector learns from any (clean, corrupted) pairs you can produce. If your workload has a characteristic failure mode — fabricated citations, number drift, prompt echo, domain-specific garbage — collect or synthesize examples of it, point the trainer at your endpoint, and the ensemble picks up the new checkpoint automatically. See the Fine-Tuning Tutorial below.

Architecture

Component	Specification
Backbone	2x TransformerEncoder (norm-first, dropout 0.05)
Hidden dimension	64
Attention heads	4
Feedforward	256 (4x hidden), ReLU
Vocabulary	4096 trigram-hash buckets (index 0 reserved for left padding)
Sequence length	256 trigrams (~600-character tail window)
Positional encoding	learnable parameters
Pooling	masked mean over non-padding positions
Head	LayerNorm -> Linear(64, 1), single logit
Parameters	345,665
Checkpoint size	~1.3 MB (safetensors)

Formulation

A checkpoint observes the stream prefix $x_1\dots x_t$ and reads only the recent window $w_t = x_{\max(1,,t-600)}\dots x_t$. The window is tokenized into character trigrams and mapped deterministically into a 4096-bucket vocabulary:

$${\tau }_i=\mathrm{b}\mathrm{l}\mathrm{a}\mathrm{k}\mathrm{e}2\mathrm{b}(x_i{x}_{i+1}{x}_{i+2})(V-1)+1,V=4096\backslash tau\_i\; =\; \backslash mathrm\{blake2b\}(x\_i\; x\_\{i+1\}\; x\_\{i+2\})\; \backslash bmod\; (V\; -\; 1)\; +\; 1,\; \backslash qquad\; V\; =\; 4096$$

After embedding $E \in \mathbb{R}^{V \times 64}$ and a learnable positional bias $P$, two norm-first transformer layers with masked self-attention produce the representation

$$H=\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}(E[\tau ]+P),h=\frac{\sum _i{m}_i{H}_i}{\sum _i{m}_i},H\; =\; \backslash mathrm\{Transformer\}\backslash big(E[\backslash tau]\; +\; P\backslash big),\; \backslash qquad\; h\; =\; \backslash frac\{\backslash sum\_i\; m\_i\backslash ,\; H\_i\}\{\backslash sum\_i\; m\_i\},$$

where $m_i \in {0,1}$ masks padding positions. The head outputs a logit

$$z=W\mathrm{L}\mathrm{N}(h)+b,z\; =\; W\backslash ,\backslash mathrm\{LN\}(h)\; +\; b,$$

and the sigmoid score is linearly calibrated to a probability using two anchors stored in the checkpoint metadata:

$${\widehat{p}}_{\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{u}\mathrm{p}\mathrm{t}}(w_t)=\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}\text{\hspace{0.17em}⁣}(\frac{\sigma (z)-{\mathrm{\ell }}_o}{h_i-{\mathrm{\ell }}_o},0,1),{\mathrm{\ell }}_o=Q_{0.95}\left[\text{clean scores}\right],h_i=\mathrm{m}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{n}\left[\text{corrupt scores}\right]\mathrm{.}\backslash hat\{p\}\_\backslash mathrm\{corrupt\}(w\_t)\; =\; \backslash mathrm\{clip\}\backslash !\backslash left(\; \backslash frac\{\backslash sigma(z)\; -\; \backslash ell\_o\}\{h\_i\; -\; \backslash ell\_o\},\backslash ;\; 0,\backslash ;\; 1\backslash right),\; \backslash qquad\; \backslash ell\_o\; =\; Q\_\{0.95\}\backslash big[\backslash text\{clean\; scores\}\backslash big],\backslash ;\; h\_i\; =\; \backslash mathrm\{median\}\backslash big[\backslash text\{corrupt\; scores\}\backslash big].$$

Calibration anchors for this checkpoint: $\ell_o = 0.168$, $h_i = 0.367$.

Training objective

Training minimizes class-weighted binary cross-entropy over onset-aware windows:

$$\mathcal{L}=-\frac{1}{N}\sum _{n=1}^N[y_n\alpha \log \sigma (z_n)+(1-y_n)\log (1-\sigma (z_n)\left)\right],\alpha =\frac{N_{-}}{N_{+}}\backslash mathcal\{L\}\; =\; -\backslash frac\{1\}\{N\}\backslash sum\_\{n=1\}^\{N\}\; \backslash Big[\; y\_n\; \backslash ,\; \backslash alpha\; \backslash ,\; \backslash log\; \backslash sigma(z\_n)\; +\; (1-y\_n)\backslash ,\; \backslash log\backslash big(1\; -\; \backslash sigma(z\_n)\backslash big)\; \backslash Big],\; \backslash qquad\; \backslash alpha\; =\; \backslash frac\{N\_-\}\{N\_+\}$$

with $N_-$ and $N_+$ the counts of clean and corrupt windows. A window is labeled corrupt only when its right edge lies at least $\Delta = 300$ characters past the true corruption onset; windows before the onset are labeled clean. Optimizer: AdamW ($\eta = 5\times10^{-4}$, weight decay $10^{-4}$), batch 64, 8 epochs; the best validation checkpoint is kept.

Training Data

Split	Source	Count
Clean (train)	live answers from the guarded endpoint (wahoo-1.5-preview via vLLM) plus a bundled clean corpus	40 live + pooled
Corrupt (train)	CorruptBench synthetic corruptions: repetition loops, cross-lingual drift, table echo, structural garbage	240 streams
Validation	held-out clean streams + 40 held-out corruptions	onset-aware windows

Results

Figure: ROC re-evaluated on a held-out split at release time (292 onset-aware windows, 9 held-out live clean documents). The headline training-time held-out AUROC reported by the trainer is 0.925; per-split variation is expected with a small clean set.

Metric	Value
Held-out AUROC	0.925
Calibration anchors	$\ell_o = 0.168$, $h_i = 0.367$
Inference latency	~4 ms per window on MPS
CPU fallback	supported

Input	Pulse probability
Clean prose	0.000
Repetition loop	1.000
Cross-lingual drift	1.000
Structural table echo	1.000

Production Readiness

• Deterministic: tokenization uses a stable blake2b hash, so training and live inference over the same text produce identical tensors.
• Graceful degradation: the detector is optional. Without torch, safetensors, or the weights file, the ensemble behaves exactly as the numpy-only core.
• Lightweight: 1.3 MB checkpoint, single-digit millisecond inference, far below the token rate of any guarded model.

Usage Inside SIMURG

The weights ship with the SIMURG package and auto-register as the sixth detector in the ensemble. Install with:

pip install "simurg[deep]"

To point SIMURG at a custom or retrained checkpoint:

export SIMURG_PULSE_WEIGHTS=/path/to/simurg_pulse.safetensors

Standalone Inference

from safetensors.torch import load_file

tensors = load_file("simurg_pulse.safetensors")

The full inference plumbing (tokenization, windowing, calibration) lives in SIMURG under src/simurg/deep/pulse.py; this repository ships the raw checkpoint and the training recipe.

Fine-Tuning Tutorial — train SIMURG on your own hallucinations

Retrain the checkpoint against any OpenAI-compatible endpoint in one command. The trainer samples live answers (clean labels), synthesizes corruptions (label 1 with known onset), trains on CPU in seconds, computes held-out AUROC and calibration anchors, and writes a new safetensors file.

Step 1: Install dependencies

pip install "simurg[deep]"

Step 2: Configure the target endpoint

export SIMURG_LIVE_URL=http://your-host:port/v1/chat/completions
export SIMURG_LIVE_MODEL=your-model-name

For custom prompts, prepare a JSONL file where each line is {"text": "your prompt"} and set:

export SIMURG_LIVE_PROMPTS_JSONL=prompts.jsonl

Step 3: Run the trainer

python -m simurg.deep.train_pulse --clean 40 --corrupt 240 --epochs 8 --out ./simurg_pulse.safetensors

Useful flags:

Flag	Default	Meaning
--clean	40	number of live clean answers to sample
--corrupt	240	number of synthetic corruption streams
--epochs	8	training epochs
--batch	64	batch size
--seed	7	RNG seed (reproducible runs)
--device	cpu	cpu, mps, or cuda
--out	bundled path	output safetensors path

Step 4: Verify the new checkpoint

The trainer prints held-out AUROC and calibration anchors before saving. A strong fine-tune typically reaches held-out AUROC above 0.90. If AUROC drops below ~0.75, increase --clean or add more diverse prompts to the JSONL.

Step 5: Ship it

Replace the bundled file (src/simurg/weights/simurg_pulse.safetensors) or keep the new file external and export SIMURG_PULSE_WEIGHTS in production. The ensemble picks it up automatically on the next process start.

Links

• Code: https://github.com/doofzoff/SIMURG
• Paper: https://ssrn.com/abstract=7451269

License

Apache-2.0, matching the parent SIMURG project.

Attribution

Trained as part of SIMURG v1.0.4 (2026-09-21) by MergenAI.