TR-HASH-0.5B

Final research checkpoint

Explore how a balanced token-ID hash selects two narrow residual experts while a shared dense SwiGLU path preserves contextual computation.

492.1M parameters 20B pretraining tokens 24 layers · fixed top-2

Interactive companion — not additional experimental evidence.

Token

Contextual hidden state x_t The same contextual vector enters both branches

Token identifier t routing · tokenizer ID 22447

Shared SwiGLU Width 4,864 · all tokens · contextual computation

Fixed route table r_l(t) Balanced layer-specific hash · two of four experts

Residual experts · width 128 each E0 E1 E2 E3

Feed-forward output Shared + 2 × mean(E0, E3)

Layer-specific deterministic routes

Checkpoint-derived tokenizer IDs and fixed top-2 assignments.

E0E1E2E3 Layer 10.5——0.5 Layer 60.5—0.5— Layer 12——0.50.5 Layer 180.5—0.5— Layer 24—0.50.5—

Token “routing” follows the persisted route table of the final checkpoint. Every occurrence keeps the same expert pair at a given layer, while its hidden-state input still changes with context.

Recorded training diagnostics

Initial and final scheduled evaluations · NLL, lower is better

4.8554

2.6615

Pretraining held-out

3.4805

2.9669

Matched SFT diagnostic

Initial Final

Measurement	Initial	Final
Pretraining held-out NLL	4.8554 @ 1k	2.6615 @ 76k
Pretraining held-out PPL	128.43	14.32
Matched SFT NLL	3.4805	2.9669
Natural-gold SFT NLL	3.0644	2.6866
Training coverage	20B pretrain	31.37M supervised

Pretraining evaluation is held out. SFT diagnostics use a 672-example matched set and a 28-example natural-gold set; the latter is too small for a stable capability claim.

Prompt

TR-HASH-0.5B ready

The model response will stream here token by token.

Decode configuration

Chat endpoint · temperature 0.7 · top-k 40 · top-p 0.9 · repetition penalty 1.1 · maximum 128 new tokens.

This is a qualitative demonstration of the released SFT checkpoint, not an evaluation result.

TR-HASH-0.5B · live qualitative generation