YuyaoGe/Prism_Delta

Prism-Δ: Differential Subspace Steering for Prompt Highlighting in Large Language Models

---

📄  Paper   |   💻  Code   |   📦  Projections

---

Introduction

PRISM-Δ (Projection-based Relevance-Informed Steering Method) makes a language model prioritize the spans a user has highlighted, without any training.

PRISM-Δ instead decomposes the difference between the positive and negative cross-covariance matrices, which cancels that shared structure and leaves only what discriminates. Each attention head then receives a continuous softplus weight, so a weak-but-useful head contributes at reduced strength rather than being switched off by a threshold. The same framework extends from the Key (routing) channel to the Value (content) channel, giving PRISM-ΔV.

This repository holds the precomputed projection matrices, so you can reproduce the reported numbers without rebuilding them. Building from scratch takes 3–8 minutes per model on a single GPU if you would rather do that.

Contents

projections/
├── biasbios/          # occupation prediction from highlighted biographies
├── counterfact/       # knowledge-conflict resolution
└── pronoun/           # gender-neutral pronoun rewriting
    └── <model>/
        ├── <model>_diff_proj.pt      # PRISM-Δ   — differential Key projection
        ├── <model>_kv_diff_proj.pt   # PRISM-ΔV  — differential Key + Value projection
        ├── <model>_pos_proj.pt       # positive-condition projection (baseline / ablation)
        └── <model>_neg_proj.pt       # negative-condition projection (baseline / ablation)

All three benchmarks carry the same five models: Qwen3-4B-Base, Qwen3-8B-Base, Qwen3-14B-Base, gemma-3-4b-pt and gemma-3-12b-pt.

Reach for diff_proj if you want the Key-only variant and kv_diff_proj for the dual-channel one. The pos_proj / neg_proj pair is the independent decomposition that the SEKA baseline runs on, and that the ablation isolating the differential step needs.

Quick Start

git clone https://github.com/YuyaoGe/PRISM-DELTA && cd PRISM-DELTA
pip install -r requirements.txt
git clone https://huggingface.co/YuyaoGe/Prism_Delta ./prism-projections
export PYTHONPATH=$(pwd)

PRISM-Δ (Key-only):

python benchmarks/eval_bias_gen.py \
  --model <path-to-Qwen3-4B-Base> \
  --data_path <path-to-biasbios.json> \
  --output_dir ./results/prism-k \
  --overwrite_output_dir --batch_size 256 --max_new_tokens 64 \
  --wd-seka \
  --wd-seka-proj ./prism-projections/projections/biasbios/Qwen3-4B-Base/Qwen3-4B-Base_diff_proj.pt \
  --wd-seka-gain 0.40 --layers all

PRISM-ΔV (Key + Value):

python benchmarks/eval_bias_gen.py \
  --model <path-to-Qwen3-4B-Base> \
  --data_path <path-to-biasbios.json> \
  --output_dir ./results/prism-kv \
  --overwrite_output_dir --batch_size 256 --max_new_tokens 64 \
  --kv-seka \
  --kv-seka-proj ./prism-projections/projections/biasbios/Qwen3-4B-Base/Qwen3-4B-Base_kv_diff_proj.pt \
  --kv-seka-gain-k 0.40 --kv-seka-gain-v 0.10 --layers all

Evaluation data is not redistributed here. Get BiasBios, CounterFact (pasta_bench) and Pronoun Change from the original release: waylonli/SEKA-datasets.

Gains

The variance-retention threshold γ and the discriminability floor δ_min are baked into the projections you download here, so the only thing left to set is the gain. Pass --wd-seka-gain for PRISM-Δ, and --kv-seka-gain-k / --kv-seka-gain-v for PRISM-ΔV. These are the validation-selected values behind the reported results.

Benchmark	Model	g_K (Δ)	g_K (ΔV)	g_V (ΔV)
BiasBios	Qwen3-4B-Base	0.40	0.40	0.10
BiasBios	Qwen3-8B-Base	0.40	0.40	0.10
BiasBios	Qwen3-14B-Base	0.40	0.40	0.10
BiasBios	gemma-3-4b-pt	0.50	0.30	0.10
BiasBios	gemma-3-12b-pt	0.40	0.40	0.10
CounterFact	Qwen3-4B-Base	1.90	1.90	0.02
CounterFact	Qwen3-8B-Base	2.70	2.70	0.05
CounterFact	Qwen3-14B-Base	3.00	3.00	0.05
CounterFact	gemma-3-4b-pt	6.00	6.00	0.10
CounterFact	gemma-3-12b-pt	1.10	3.00	0.50
Pronoun Change	Qwen3-4B-Base	0.15	0.15	0.05
Pronoun Change	Qwen3-8B-Base	0.05	0.05	0.02
Pronoun Change	Qwen3-14B-Base	0.05	0.05	0.02
Pronoun Change	gemma-3-4b-pt	0.30	0.30	0.10
Pronoun Change	gemma-3-12b-pt	−0.30	0.05	0.02

A negative g_K, as on gemma-3-12b-pt for Pronoun Change, is expected: the edit rescales the projected component rather than amplifying it, and which sign helps is decided per configuration on validation data.

If you are rebuilding projections rather than using these, the γ and δ_min for every configuration are in the paper's appendix.

Citation

@misc{ge2026prism,
  title         = {Prism-$\Delta$: Differential Subspace Steering for Prompt
                   Highlighting in Large Language Models},
  author        = {Yuyao Ge and Shenghua Liu and Yiwei Wang and Baolong Bi and
                   Lingrui Mei and Jiayu Yao and Jiafeng Guo and Xueqi Cheng},
  year          = {2026},
  eprint        = {2603.10705},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2603.10705}
}

Acknowledgments

Built on SEKA, SEA-LLM, PASTA, and Selective Prompt Anchoring.