ukung/semantic-lite-2-decoder-smoke-test

Semantic-Lite-2 Decoder — SMOKE TEST

Status: SMOKE TEST / EXPERIMENTAL. This is not a production model. It is an early experiment to validate the architecture, not a model ready for use.

What this is

A transformer decoder conditioned on semantic representations produced by ukung/semantic-lite-2 (frozen).

• Data A — 256-dim global semantic vector -> projected into a decoder prefix token
• Data B — 2048-dim per-token hidden states (2 backbone layers) -> key/value for cross-attention
• Decoder — 1 layer, d_model=1024, nhead=16, FFN=1024 (trainable)
• Output — tied embedding with the encoder embedding table (131072 vocab)

The encoder is fully frozen. Only the projections, the decoder layer, and the output projection are trained.

Training (smoke test)

Item	Value
Dataset	ukung/Opus-4.6-Reasoning (1000 samples)
Split	900 train / 100 eval
Target	thinking + solution (chain-of-thought)
Epochs	10
Effective batch	4 (batch 1 x accum 4)
LR	1e-4 (AdamW, warmup 100)
Final train loss	4.6753
Final eval loss	5.3534
Hardware	Tesla T4

Files

• decoder_weights.pt — decoder state dict (encoder excluded)
• config.json — architecture hyperparameters

Usage

The encoder must be loaded separately from ukung/semantic-lite-2, then the decoder weights are loaded into the SemanticConditionedDecoder class.

Notes

Eval loss of 5.35 is still high — the model has not actually learned reasoning. Generated output is largely incoherent: it reproduces the surface format of the training data ("Let me analyze this problem step by step") but the content is not grounded in the input. This repository purely validates that the pipeline (frozen encoder -> conditioning -> decoder) runs end-to-end. Further training is required.

An ablation comparing the conditioned decoder against an unconditioned baseline has not yet been run, so the contribution of the semantic conditioning is currently unverified.