lebe1/lettuceprevent-generate

LettucePrevent — custom_generate

Real-time hallucination prevention during RAG generation. A token-level detector plugs into decoding through a custom LogitsProcessor. At every step the top-k candidates are scored and hallucinated tokens are penalized before they are emitted.

🔗 GitHub · Detector model

---

Detectors

Detector	Type	Notes
number	rule-based	RegEx-verifiable; rejects numbers not in the input. No checkpoint to download.
lettuceprevent	neural (Ettin 68M)	Flags unsupported tokens for factual claims. CUDA recommended.
lettucedetect	encoder	Post-hoc hallucination scoring (e.g. KRLabsOrg/lettucedect-base-modernbert-en-v1).

Install

pip install -r custom_generate/requirements.txt
python -m spacy download en_core_web_sm

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mistralai/Mistral-7B-Instruct-v0.2"
tok   = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto",
                                             trust_remote_code=True)

# For instruction-tuned models, always apply the chat template
context  = "Revenue was 2400 million in 2021 and 3100 million in 2022."
question = "What is the percentage increase in revenue from 2021 to 2022?"
text     = tok.apply_chat_template(
               [{"role": "user", "content": f"{context}\n{question}"}],
               tokenize=False, add_generation_prompt=True)
inputs   = tok(text, return_tensors="pt").to(model.device)

out = model.generate(
    **inputs,
    custom_generate="lebe1/lettuceprevent-generate",   # or a local path
    trust_remote_code=True,
    tokenizer=tok,
    input_text=context,          # grounding context for the detector
    detector_type="number",      # or "lettuceprevent"
    max_new_tokens=300,
)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Key arguments

Argument	Default	Description
tokenizer	required	Generator tokenizer (used to decode candidates).
input_text	required	Grounding context validated against by the detector.
detector_type	"lettuceprevent"	"number", "lettuceprevent", or a baseline-* switch.
skip_threshold	1.0	Skip the HDM check when top-token probability exceeds this. 1.0 = always check. Important to play around with this parameter to understand its importance.
penalty_value	0.0	Score assigned to hallucinated tokens (float('-inf') to hard-block).
confidence_threshold	0.9	Hallucination-probability threshold for lettuceprevent.
top_k_logits	10	Candidate tokens scored per step.
last_k_tokens_to_consider	10	Context window the detector uses. For 'number' 10 is usually fine, for 'lettuceprevent' use all tokens below.
use_all_tokens	True	If False, only the last last_k_tokens_to_consider tokens are used.
model_path	detector default	Override the hallucination-detector checkpoint.
query	""	Optional query string for certain detectors.
debug_print	False	Print per-step debug information.

Notes

• Returns exactly what model.generate() would — a tensor of token ids.
• The lettuceprevent detector requires a CUDA-capable GPU.
• For instruction-tuned models, wrap the prompt with tok.apply_chat_template(...) (see the example above).

Citation

@mastersthesis{Beccard:2026,
  title  = {Real-time Prevention of Factual Hallucinations in Retrieval-Augmented Generation},
  author = {Leon Beccard},
  school = {Technische Universität Wien},
  year   = {2026},
  url    = {https://repositum.tuwien.at/handle/20.500.12708/229242}
}