scribe-tok-50k

A byte-pair encoding (BPE) tokenizer with a 50,004-token vocabulary, trained from scratch on 2,000,000 lines of clean English book text. It is intended for experimentation, learning, and small-to-medium language models that need a compact, from-scratch vocabulary instead of a large production tokenizer.

Details

Model type	BPE
Vocab size	50,004
Merges	49,928
Pre-tokenizer	Whitespace
Normalizer	None
Decoder	None
Min frequency	2
Corpus	kd13/bookcorpus-clean (2,000,000 lines, ~151 MB)
Trained with	tokenizers 0.23.2 / transformers 5.17.0

Special tokens

Token	ID
[PAD]	0
[UNK]	1
[BOS]	2
[EOS]	3

Usage

With transformers:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("coderian/scribe-tok-50k")

text = "the mysterious creature suddenly disappeared into the shadows"
encoded = tokenizer(text)

print(encoded["input_ids"])
print(tokenizer.convert_ids_to_tokens(encoded["input_ids"]))

Or directly with tokenizers:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
output = tokenizer.encode("the mysterious creature suddenly disappeared into the shadows")

print(output.tokens)
print(output.ids)

Training

The tokenizer was trained from scratch on the English kd13/bookcorpus-clean dataset:

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

trainer = BpeTrainer(
    vocab_size=50_004,
    min_frequency=2,
    special_tokens=["[PAD]", "[UNK]", "[BOS]", "[EOS]"],
)

tokenizer.train(["corpus.txt"], trainer=trainer)
tokenizer.save("tokenizer.json")

The full pipeline is available on GitHub:

1. config.py - streams 2,000,000 lines from the dataset and writes corpus.txt
2. train.py - trains the BPE tokenizer, saves tokenizer.json, and pushes it to the Hub
3. test.py - encodes a sample sentence and prints tokens/IDs

Limitations

• Trained only on English book text; performance on other languages, code, or social/web text will be poor.
• No normalization or lowercasing is applied, so casing variations are treated as distinct tokens.
• Pre-tokenization is whitespace-only, so tokens never include leading spaces and punctuation is treated as standalone tokens.
• No decoder is configured; decode reflects the whitespace-separated format of the training corpus.

License

Released under the MIT License.