TinyJLLM-Instruct-DPO — preference-tuned 100M model

Direct Preference Optimization applied to TinyJLLM-Instruct with a DPO trainer implemented from scratch in the TinyLLM repository (no TRL).

Measured effect: held-out preference accuracy improved from 64.5% (SFT reference) to 77.5% (this model) on 200 unseen pairs, and generations visibly cleaned up:

Prompt	SFT	DPO (this model)
capital of France	is a country of France.	The capital of France
fruit or vegetable?	apple, apple, apple… (loops)	Apple

Model family

Model	Stage	Link
TinyJLLM	Base	jaweed123/TinyJLLM
TinyJLLM-Instruct	SFT	jaweed123/TinyJLLM-Instruct
TinyJLLM-Instruct-DPO	DPO (this model)	jaweed123/TinyJLLM-Instruct-DPO

Usage

Same marker format and the same response-masked generation protocol as the SFT model (plain causal model.generate() degrades into repetition):

### Instruction:
What is the capital of France?

### Response:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "jaweed123/TinyJLLM-Instruct-DPO"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).eval()

@torch.no_grad()
def respond(instruction, max_new_tokens=40):
    prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
    ids = tok(prompt)["input_ids"]
    L = len(ids)
    seq = ids + [tok.pad_token_id]          # dummy keeps response positions aligned
    for _ in range(max_new_tokens):
        inp = torch.tensor([seq])
        T = inp.shape[1]
        mask = torch.tril(torch.ones(1, 1, T, T, dtype=torch.bool))
        mask[:, :, L:, :] = False           # response rows ...
        mask[:, :, L:, :L] = True           # ... see the prompt only
        nxt = int(model(inp, attention_mask=mask).logits[0, -1].argmax())
        if nxt == tok.eos_token_id:
            break
        seq.append(nxt)
    return tok.decode(seq[L + 1:])

print(respond("What is the capital of France?"))

For sampling (temperature 0.8, top-k 50, top-p 0.95, repetition penalty 1.15 — the values chosen by the project's sampling sweep), use the project's learnllm.inference.generate.generate(..., response_mask_start=...).

Training details

Base / reference	TinyJLLM-Instruct (frozen reference = same weights)
Dataset	32,648 pairs: Orca DPO (Apache-2.0) + HH-RLHF (MIT, filtered to single-turn answer-like pairs), 2 epochs
Beta / LR	0.2 / 3e-6, warmup + cosine
Compute trick	reference log-probabilities precomputed once (halves training compute)
Batch / steps	32 pairs effective, 3,882 steps
Hardware	RTX 4060 8 GB, ~2 h

Two earlier DPO runs (unfiltered data, lower LR) and the reasoning for the final recipe are documented in DPO_TRAINING.md, including a data-integrity incident that was found and fixed.

Limitations

• Still a 100M model. DPO improves preference ranking and answer shape; it does not add knowledge. Math and code remain unreliable.
• Multi-turn conversations fail. The model echoes earlier context instead of answering the last turn (measured; see the repository docs).
• Safety tuning is minimal — a small, mostly-English preference set. Do not rely on this model for safety-critical filtering.

Citation

@misc{tinyjllm,
  title  = {TinyJLLM: A 100M-Parameter Small Language Model Built From Scratch},
  author = {Jaweed, Abdul},
  year   = {2026},
  url    = {https://github.com/Abdul-Jaweed/TinyLLM}
}