Shamima/dpo-implicit-reward

Exploring DPO's Implicit Reward

Research checkpoints for a controlled study asking whether DPO's implicit reward

r̂(x, y) = β · [ log π_θ(y|x) − log π_ref(y|x) ]

works as well as an explicitly trained Bradley–Terry reward model, under matched data, in-distribution and under distribution shift.

• H1: explicit RM > implicit on held-out in-distribution pairs. → supported
• H2: the gap widens under distribution shift. → not supported

Code: github.com/silvererudite/dpo-implicit-reward

---

⚠️ Read this before using any checkpoint

π_ref must be sft_merged/. Every adapter here was trained on top of the merged SFT checkpoint, not on raw Qwen/Qwen2.5-0.5B. Loading an adapter onto the raw base model gives a different — and wrong — model. For the implicit reward specifically, using the wrong reference silently changes the quantity you compute.

These are research artifacts at 0.5B scale, not production reward models. They are not safety-aligned and should not be used to rank or moderate content in a deployed setting.

---

What is in this repository

Reference policy — start here

Path	What it is
sft_merged/	π_ref. SFT policy with LoRA merged into the weights. A complete model. Everything else is an adapter on top of this.
sft/	The SFT LoRA adapter alone (applies to Qwen/Qwen2.5-0.5B). Provided for provenance; you normally want sft_merged/.

Main experiment — 1 epoch, 5 seeds

The headline result. Each run saw exactly 8,000 preference pairs in 500 optimizer steps.

Path	What it is
dpo_beta0.1_8k_s{0,1,2,3,4}/	DPO policies, β=0.1. The implicit reward is read off these.
rm_8k_s{0,1,2,3,4}/	Explicit Bradley–Terry reward models (LoRA + scalar head).
dpo_beta0.1_8k/, rm_8k/	Earlier single-seed run, kept for provenance. Superseded — trained before the pair-matching fix, so not comparable to the seeded runs.

Convergence experiment — 4 epochs, 2 seeds

Trained to convergence with held-out evaluation every 100 steps, to test whether the 1-epoch ranking was an artifact of stopping early. It was not.

Path	What it is
long/dpo_beta0.1_8k_s{0,1}/	DPO, 4 epochs (2,000 steps).
long/rm_8k_s{0,1}/	Reward model, 4 epochs.
long/*/checkpoint-{250,500,...,2000}/	Snapshots every 250 steps = every half epoch. Adapter weights + trainer_state.json only; optimizer state was stripped, so these are for analysis, not resuming.

Results and figures

Path	What it is
results/eval_s*.json	Per-seed metrics on all six test sets.
results/raw_s*.json	Per-pair scores. Every metric is recomputable from these without re-running any model.
results/aggregate_*.json	Cross-seed means with 95% CIs.
results/bias_over_training_*.json	Surface-form preference at each checkpoint.
results/curves/*.csv	Every training run's loss/accuracy curves.
figures/	All result figures.

Every training directory carries a budget.json recording steps, epochs, effective batch, LR, seed and wall-clock, so the compute-matching claim is auditable rather than asserted.

---

How to load each artifact

1. The reference policy π_ref

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

REPO = "Shamima/dpo-implicit-reward"
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

# subfolder= pulls just this directory out of the repo
ref = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder="sft_merged", torch_dtype=torch.bfloat16).cuda().eval()

2. A DPO policy → the implicit reward

from peft import PeftModel

backbone = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder="sft_merged", torch_dtype=torch.bfloat16).cuda()
policy = PeftModel.from_pretrained(backbone, REPO, subfolder="dpo_beta0.1_8k_s0").cuda().eval()

# r̂ = β · (log π_θ(y|x) − log π_ref(y|x)), summed over RESPONSE tokens only
# (prompt masked). See src/scoring.py in the GitHub repo for the exact implementation.

The reward is a difference of two models, so you need both policy and ref loaded. Ranking within a prompt is invariant to β, so β only affects calibration, not pairwise accuracy.

3. An explicit reward model

from transformers import AutoModelForSequenceClassification

base = AutoModelForSequenceClassification.from_pretrained(
    REPO, subfolder="sft_merged", num_labels=1, torch_dtype=torch.bfloat16)
base.config.pad_token_id = tok.pad_token_id
rm = PeftModel.from_pretrained(base, REPO, subfolder="rm_8k_s0").cuda().eval()

text = tok.apply_chat_template(
    [{"role": "user", "content": prompt},
     {"role": "assistant", "content": response}], tokenize=False)
score = rm(**tok(text, return_tensors="pt", truncation=True, max_length=1024).to("cuda")).logits[0].item()

The trained scalar head ships inside the adapter under modules_to_save, so loading the adapter restores it. Transformers will warn that score.weight was "newly initialized" when the base is created — that warning is expected and the adapter overwrites it.

4. A mid-training checkpoint

rm_at_epoch2 = PeftModel.from_pretrained(
    base, REPO, subfolder="long/rm_8k_s0/checkpoint-1000").cuda().eval()

Step → epoch mapping: 500 steps = 1 epoch (8,000 pairs at effective batch 16). So checkpoint-250 = 0.5 epochs, checkpoint-1000 = 2 epochs, checkpoint-2000 = 4 epochs.

5. Results without any GPU

from huggingface_hub import hf_hub_download
import json

p = hf_hub_download(REPO, "results/aggregate_beta0.1_8k.json", repo_type="model")
agg = json.load(open(p))["results"]

---

Which checkpoint should I use?

If you want…	Use
The best reward model here	long/rm_8k_s0/checkpoint-1000 — its held-out peak, ~epoch 2
The headline-experiment reward model	rm_8k_s0 (1 epoch, matched budget)
The best implicit reward	dpo_beta0.1_8k_s0 — DPO peaks at ~1 epoch and degrades after
To measure seed variance	all five of *_s0 … *_s4
To study overfitting dynamics	long/*/checkpoint-*

Do not use long/*/checkpoint-2000 (4 epochs) expecting the best model — both scorers are past their peak there. The reward model's held-out accuracy falls from 0.731 at epoch 2 to ~0.69 at epoch 4.

---

Results

Length-controlled pairwise accuracy, mean ± 95% CI over 5 seeds. Length control restricts scoring to pairs whose two responses are within 1.2× in length, because raw accuracy on these benchmarks is substantially a length signal.

Test set	DPO implicit	Explicit RM	Untrained baseline	pairs
UltraFeedback (ID)	0.600 ± 0.026	0.663 ± 0.009	0.529	442
RewardBench Chat	0.800 ± 0.044	0.747 ± 0.080	0.500	32
RewardBench Chat-Hard	0.535 ± 0.052	0.615 ± 0.061	0.661	62
RewardBench Safety	0.492 ± 0.036	0.549 ± 0.059	0.530	83
RewardBench Reasoning	0.850 ± 0.003	0.663 ± 0.019	0.906	908
HH-RLHF harmless	0.472 ± 0.016	0.429 ± 0.036	0.435	276

The "untrained baseline" is the length-normalized log-probability of the response under π_ref — no reward training at all.

Three findings

1. H1 holds. The explicit reward model wins in distribution, +0.062 ± 0.030, in every seed, and the effect survives correction for the six simultaneous comparisons.
2. H2 does not hold. Four of five shifted sets are ties at 5 seeds. The one clearly resolvable OOD difference runs the other way (Reasoning).
3. On the two hardest sets, reward training is worse than no reward training. The untrained baseline beats both trained scorers on Reasoning and Chat-Hard. Whatever these methods learn, they also destroy signal the base model already had.

Convergence and drift

Training both to 4 epochs: the reward model peaks at 0.731 held-out accuracy at epoch 2 then overfits; DPO peaks at 0.676 at epoch 1 and is flat thereafter. The ranking never inverts.

Scoring every checkpoint for surface-form preference produced a result that refuted our hypothesis: we expected the reward model's length bias to grow with training. It peaks at +0.069 around epoch 1.5 and then fades to +0.002. DPO's drifts the other way, from −0.134 to −0.232 — it progressively prefers shorter answers than humans do, and its accuracy falls with it.

---

Training setup

Qwen/Qwen2.5-0.5B, LoRA r=16 / α=32 / dropout=0.05 on all attention and MLP projections, bf16, single NVIDIA A10G per job, TRL 0.15.2. DPO and the reward model both initialize from sft_merged/ and see byte-identical pairs (verified by hash) so neither is advantaged.

Stage	Steps	Epochs	Effective batch	LR	Wall-clock
SFT → π_ref (32k pairs)	1998	1	16	2e-4	1h 13m
DPO β=0.1 (8k pairs)	500	1	16	5e-5	~40 min
Reward model (8k pairs)	500	1	16	5e-5	~24 min
Convergence runs	2000	4	16	5e-5	175 / 105 min

Limitations

• 0.5B scale, one β, one data budget. Do not generalize to 7B+ without checking.
• π_ref was trained once. Seeds vary only the second stage, so the intervals describe training variance given one reference, not variance of the method. The untrained baseline consequently has no error bars at all.
• Length control costs power. Three test sets retain under 100 pairs after filtering and cannot support conclusions on their own.
• Calibration is poor for every scorer (ECE 0.22–0.45), and degrades faster than accuracy during overfitting.
• Compute is data-matched, not FLOP-matched. DPO runs an extra frozen reference forward pass and took ~1.7× the wall-clock.

License

Apache 2.0, following the Qwen/Qwen2.5-0.5B base model.