afoolishman/anlp-a1-assignment

ANLP Assignment 1 — Encoder–Decoder Ablations (C1–C5)

This hub entry hosts checkpoints from a controlled ablation: map encrypted binary sequences (brown_cipher.txt) to English plaintext (brown_plain.txt). Configuration C1 is the base model; C2–C5 each change exactly one component.

---

Model variants

Config	Single change vs C1	Architecture class
C1	— (base)	Seq2SeqTransformer
C2	RoPE instead of sinusoidal PE	Seq2SeqTransformer
C3	GQA (8 Q, 2 KV) instead of MHA	Seq2SeqTransformer
C4	RMSNorm instead of LayerNorm	Seq2SeqTransformer
C5	Entropy-patched BLT instead of BPE	BLTSeq2Seq

Shared hyperparameters

Hyperparameter	Value
d_model	256
Heads	8 (C3: 2 KV heads)
Encoder / decoder layers	4 / 4
d_ff	1024
Dropout	0.15
Optimizer	AdamW, lr 1e-3, wd 0.01, betas (0.9, 0.98)
LR schedule	cosine + 400 warmup steps
Batch size	64
Chunk size	256 characters (aligned cipher/plain)
Max epochs	100 (C5 early-stopped later; best @ epoch 146)
Early stopping	patience 10 on val CE
Seed	42
Precision	AMP (fp16)

Tokenization

• C1–C4: from-scratch BPE, vocab 1024 per side. Cipher: pack 8 bits → byte, then learn merges. Plain: character BPE. Artifacts: cipher_bpe.json, plain_bpe.json.
• C5: no BPE. Packed bytes + entropy patches (len 1–8, mean ≈ 4.2), hashed 2/3/4-gram features. Artifacts: entropy_ngram.json, entropy_patcher.json.

---

Intended use

• Primary: course ablation / reproducibility for ANLP Assignment 1.
• Task: greedy decode ciphertext chunks → plaintext chunks.
• Not intended for: production MT, open-domain generation, or claiming a general “decrypt any cipher” system. The mapping is specific to this homework dataset.

---

How to load weights

Checkpoints are raw PyTorch state dicts (pytorch_model.bin) plus config.json. They are not Hugging Face transformers.AutoModel checkpoints—load with the assignment code.

import json
import torch
from pathlib import Path

# after cloning / downloading this repo's `src/` onto PYTHONPATH
from models.transformer import Seq2SeqTransformer
from models.blt import BLTSeq2Seq
from tokenizer import BPETokenizer, PAD

hf_dir = Path("path/to/downloaded/hf")  # folder with config.json + pytorch_model.bin
cfg = json.loads((hf_dir / "config.json").read_text())

if cfg.get("blt"):
    model = BLTSeq2Seq(
        d_model=cfg["d_model"],
        n_heads=cfg["n_heads"],
        n_kv_heads=cfg["n_kv_heads"],
        n_encoder_layers=cfg["n_encoder_layers"],
        n_decoder_layers=cfg["n_decoder_layers"],
        d_ff=cfg["d_ff"],
        dropout=cfg["dropout"],
        patch_size=cfg["patch_size"],
        n_local_layers=cfg["n_local_layers"],
        n_local_heads=cfg["n_local_heads"],
        pad_id=PAD,
    )
else:
    model = Seq2SeqTransformer(
        src_vocab=cfg["src_vocab"],
        tgt_vocab=cfg["tgt_vocab"],
        d_model=cfg["d_model"],
        n_heads=cfg["n_heads"],
        n_kv_heads=cfg["n_kv_heads"],
        n_encoder_layers=cfg["n_encoder_layers"],
        n_decoder_layers=cfg["n_decoder_layers"],
        d_ff=cfg["d_ff"],
        dropout=cfg["dropout"],
        pos_encoding=cfg["pos_encoding"],
        attn_kind=cfg["attn_kind"],
        norm_kind=cfg["norm_kind"],
        pad_id=PAD,
    )

state = torch.load(hf_dir / "pytorch_model.bin", map_location="cpu")
model.load_state_dict(state)
model.eval()

---

Training data

• Source: assignment corpus brown_cipher.txt / brown_plain.txt (5000 line pairs).
• Split: document-level 80/10/10, seed 42, then aligned 256-character chunks (~11.5k / 1.4k / 1.4k train/val/test).
• Property: len(cipher)/8 == len(plain) so packed cipher byte i aligns with plaintext character i.

---

Evaluation

Greedy decoding only. Test metrics (1389 chunks):

Config	Params	Best val CE	Epochs	Bit acc. (%)	Seq. acc. (%)	Lev.	BLEU
C1	8.15M	0.0294	100	97.06	73.43	0.58	97.87
C2	8.15M	0.0188	82	99.01	85.89	0.81	98.75
C3	6.97M	0.0336	100	96.34	69.62	0.73	97.34
C4	8.14M	0.0283	94	97.15	73.79	0.54	97.86
C5	11.70M	0.0699	156	91.64	26.21	3.33	n/a*

*BLEU/ROUGE are secondary for token-free C5; greedy C5 BLEU ≈ 89.3 if computed for reference.

Ranking: C2 > C1 ≈ C4 > C3 > C5.

Note on C5: bit accuracy stays relatively high while sequence (exact-match) accuracy is lower because errors are mostly scattered typos on long chunks (mean Lev ≈ 3.3). Parallel patch decoding and entropy cuts on predicted bytes amplify that gap.

---

C1–C4

File	Description
pytorch_model.bin	state_dict of Seq2SeqTransformer
config.json	Architecture + hparams
cipher_bpe.json	Source BPE
plain_bpe.json	Target BPE
README.md	This model card

C5

File	Description
pytorch_model.bin	state_dict of BLTSeq2Seq
config.json	Architecture + hparams + entropy_threshold
entropy_ngram.json	Smoothed byte n-gram entropy model
entropy_patcher.json	Threshold / target patch size
README.md	This model card

---

Limitations

• Single seed (42).
• C5 is not capacity-matched to C1 (extra byte/n-gram parameters).
• Results are specific to this index-aligned local cipher, not general MT.
• Not an transformers Auto* model; requires the course src/ code to run.
• Do not treat published weights as a cryptanalysis tool beyond this homework setup.

---

References

• Vaswani et al., Attention Is All You Need, NeurIPS 2017.
• Sennrich et al., Neural Machine Translation of Rare Words with Subword Units, ACL 2016.
• Su et al., RoFormer (RoPE), 2021.
• Ainslie et al., GQA, 2023.
• Zhang & Sennrich, RMSNorm, NeurIPS 2019.
• Pagnoni et al., Byte Latent Transformer, ACL 2025. https://aclanthology.org/2025.acl-long.453