scribe-tok-5k

A small byte-pair encoding (BPE) tokenizer with a 5,000-token vocabulary, trained on clean English book text. It is intended for experimentation, learning, and small language models where a compact vocabulary is preferred over a large production tokenizer.

Details

Model type	BPE
Vocab size	5,000
Pre-tokenizer	Whitespace
Min frequency	2
Corpus	kd13/bookcorpus-clean (100,000 lines)
Trained with	tokenizers 0.23.2 / transformers 5.17.0

Special tokens

Token	ID
[PAD]	0
[UNK]	1
[CLS]	2
[SEP]	3
[MASK]	4

The whitespace pre-tokenizer means tokens never include leading spaces, and no lowercasing is applied (the corpus is already lowercase).

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("coderian/scribe-tok-5k")

text = "the mysterious creature suddenly disappeared into the shadows"
encoded = tokenizer(text)

print(encoded["input_ids"])
print(tokenizer.convert_ids_to_tokens(encoded["input_ids"]))

Or directly with tokenizers:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
output = tokenizer.encode("the mysterious creature suddenly disappeared into the shadows")

print(output.tokens)
print(output.ids)

Training

The tokenizer was trained from scratch:

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

trainer = BpeTrainer(
    vocab_size=5000,
    min_frequency=2,
    special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"],
)

tokenizer.train(["corpus.txt"], trainer=trainer)
tokenizer.save("tokenizer.json")

Limitations

• Trained only on English book text; performance on other languages, code, or social/web text will be poor.
• No normalization or lowercasing is applied, so casing variations are treated as distinct tokens.
• Vocabulary is intentionally small (5k), so rare words are split into many subword pieces.

License

Released under the MIT License.