v11-tokenizer

adopted. This is the current recommended tokenizer for this line.

Loading

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("chrishayuk/v11-tokenizer")

No trust_remote_code, no clone, no training code. That is verified against the downloaded artifact at publish time, not asserted.

Identity

Identity is the content hash, not the Hub revision. Re-pushing identical bytes mints a new commit oid; a README edit does too. Join on this instead — it is the same value a checkpoint records as tokenizer_hash:

tokenizer.json sha256	10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae
Vocabulary size	71,260
Source	chrishayuk/v-tokenizers
Source commit	ee502e091e71741703847bed1a187a10702825f2

This repo is immutable: a fix ships under a new name, never as a replacement.

Special tokens

Role	Token	ID
Unknown	<unk>	1
Beginning	<s>	2
End	</s>	3
Padding	<pad>	0

Using the Rust CLI

v11.vocab.bin is the native vocabulary, shipped here because cargo install delivers the binary and not the data:

cargo install v11-cli
huggingface-cli download chrishayuk/v11-tokenizer v11.vocab.bin --local-dir .
v11 --model v11.vocab.bin vocab --blocks
v11 --model v11.vocab.bin encode --text "Once upon a time" --show-pieces

Dormant blocks — read before sizing an embedding table

This vocabulary declares blocks that are present as ids but masked out of the projection matmul and optimizer state at 115M parameters. Sizing an embedding table from len(tokenizer) will disagree with the model's active slice, and no vocabulary-length check will catch it.

See dormant_blocks.json for the machine-readable declaration.

Block	Active @115M	Active @1B	Tokens frozen
dormant: T-cell	False	True	no (schema pinned, filled at TOK-5)
dormant: T-scale	False	True	no (schema pinned, filled at TOK-5)

Golden encodings

golden_encodings.json carries 116 frames from the MSI frame battery — call operands, post-delimiter and mid-string-punctuation surfaces, not "representative strings", because those are where segmentation actually breaks. Every one is replayed against the downloaded artifact at publish time.

import json, urllib.request
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("chrishayuk/v11-tokenizer")
golden = json.load(urllib.request.urlopen(
    "https://huggingface.co/chrishayuk/v11-tokenizer/resolve/main/golden_encodings.json"))
for name, case in golden.items():
    assert tok(case["text"], add_special_tokens=False)["input_ids"] == case["ids"], name

Provenance

provenance.json points at the systems that own these facts rather than copying them: the corpus lives in the chuk-datasets catalog (dataset name, version, content sha), and results live in the chuk-experiments ledger (run id). This repo is a publication location, not a second source of truth.

Intended use

Research and training of compact language models.

Limitations

Corpus coverage, language coverage and normalisation behaviour are described in the source repository. Dormant-block ids are exposed by this artifact but inactive at 115M — see above.