ZappY-AI/anlp-a1

ANLP Assignment 1 — Custom Transformer Ablation (C1–C5)

Custom encoder-decoder transformers trained from scratch on an XOR-cipher-to-plaintext seq2seq task (key: ANLP2026).

Each subfolder contains the best checkpoint (best.pt) and metrics (metrics.json) for one architectural configuration.

Configs

Folder	Description
c1/	Base — Sinusoidal PE, MHA, LayerNorm, Standard Subword
c2/	RoPE — Rotary positional encoding replacing sinusoidal
c3/	GQA — Grouped-Query Attention replacing MHA
c4/	RMSNorm — RMSNorm replacing LayerNorm
c5/	BLT — Byte Latent Transformer, token-free (fixed patch size 8)

Loading a checkpoint

from huggingface_hub import hf_hub_download
import torch

path = hf_hub_download(repo_id="ZappY-AI/anlp-a1", filename="c1/best.pt")
state_dict = torch.load(path, map_location='cpu')

See the assignment report for full architecture details, ablation results, and analysis.