chrisjcc/ask-before-answer

AskBeforeAnswer 🤖

This model is a Qwen 2.5 7B Instruct model fine-tuned using a two-stage pipeline (Supervised Fine-Tuning followed by Direct Preference Optimization) on the AmbigNQ dataset.

Model Description

The AskBeforeAnswer model exhibits "clarification-seeking" behavior. When presented with an ambiguous question, rather than hallucinating or blindly assuming an intent, the model:

1. Detects the ambiguity.
2. Explains the reasoning behind the ambiguity.
3. Identifies the missing facets of information.
4. Asks a targeted clarification question to the user.

Pipeline

• Base Model: Qwen/Qwen2.5-7B-Instruct
• Ablation Winner: The model variant promoted to Production via W&B Registry is: sft_dpo.

LLM-as-a-Judge Evaluation Leaderboard

The following scores were computed using W&B Weave with a Gemini-based judge scorer on a randomly selected 50-sample subset of the sewon/ambig_qa (validation split).

Metric	base	dpo_only	sft	sft_dpo	clarifier_lora
ambiguity_detection	0.978	0.964	0.972	0.976	0.998
clarification_quality	0.786	0.798	0.798	0.8	0.796
usefulness	0.882	0.896	0.9	0.9	0.896
model_accuracy	0.62	0.64	0.66	0.62	0.6
clarify_precision	0.617021	0.642857	0.675676	0.627907	0.6
clarify_recall	0.966667	0.9	0.833333	0.9	1
clarify_f1	0.753247	0.75	0.746269	0.739726	0.75
answer_f1	0.173913	0.357143	0.484848	0.296296	0
macro_f1	0.46358	0.553571	0.615559	0.518011	0.375
answer_accuracy	0.05	0.1	0.1	0.1	0
facet_generation_rate	0.0212766	1	1	1	0
clarify_ratio	1.56667	1.4	1.23333	1.43333	1.66667

GitHub Release: v0.0.5

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "Qwen/Qwen2.5-7B-Instruct"
adapter_model_name = "chrisjcc/ask-before-answer"

# Load Base
model = AutoModelForCausalLM.from_pretrained(base_model_name)
tokenizer = AutoTokenizer.from_pretrained(base_model_name)

# Attach AskBeforeAnswer Adapters
model = PeftModel.from_pretrained(model, adapter_model_name)