Tokenizers

A tokenizer converts text into tensors, which are the inputs to a model. It normalizes and splits text, applies the tokenization algorithm, adds special tokens, and decodes output ids back into text.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b")
tokenizer("Sphinx of black quartz, judge my vow.", return_tensors="pt")
{
    'input_ids': tensor([[     2, 235277,  82913,    576,   2656,  30407, 235269,  11490,    970,  29871, 235265]]),
    'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])
}

This guide covers loading, encoding, decoding, batch processing, and the available tokenizer backends.

Load a tokenizer

Load a tokenizer with the AutoTokenizer class or a model-specific tokenizer class.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b")

Encode and decode

The TokenizersBackend.call() method encodes text or a batch of text into input_ids, attention_mask, and other model inputs. It also controls padding, truncation, and special token insertion.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b")
tokenizer("Sphinx of black quartz, judge my vow.", return_tensors="pt")
{
    'input_ids': tensor([[     2, 235277,  82913,    576,   2656,  30407, 235269,  11490,    970,  29871, 235265]]),
    'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])
}

TokenizersBackend.encode() is similar but only returns the input_ids.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b")
tokenizer.encode("Sphinx of black quartz, judge my vow.")
[2, 235277, 82913, 576, 2656, 30407, 235269, 11490, 970, 29871, 235265]

TokenizersBackend.decode() converts a single sequence or batch of tokenized input_ids back to text.

tokenizer.decode(outputs["input_ids"])
['<bos>Sphinx of black quartz, judge my vow.']

TokenizersBackend.decode() preserves the exact tokenization spacing. Set clean_up_tokenization_spaces to remove spaces before punctuation, and skip_special_tokens to strip special tokens from the output.

tokenizer.decode(outputs["input_ids"], skip_special_tokens=True)
['Sphinx of black quartz, judge my vow.']

Special tokens

Special tokens mark structural boundaries in a sequence, like the beginning-of-sequence or padding positions. Each model defines its own set of special tokens. The tokenizer adds them when you call it.

tokenizer.encode("Sphinx of black quartz, judge my vow.")
[2, 235277, 82913, 576, 2656, 30407, 235269, 11490, 970, 29871, 235265]
tokenizer.decode(outputs["input_ids"])
['<bos>Sphinx of black quartz, judge my vow.']

Register additional named special tokens with the extra_special_tokens argument. Multimodal models use them as placeholders for images, video, or audio.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "google/gemma-3-4b-pt",
    extra_special_tokens={"image_token": "<image>"}
)

Batch processing

Batch processing tokenizes multiple sequences in a single call. TokenizersBackend handles large batches faster because its Rust-based backend parallelizes work across threads.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b")
tokenizer(
    [
        "Sphinx of black quartz, judge my vow.",
        "Pack my box with five dozen liquor jugs.",
        "How vexingly quick daft zebras jump!"
    ],
    return_tensors="pt"
)

Batch processing requires all sequences to share the same length. Padding and truncation are strategies to handle varying-length sequences.

Padding

Padding appends special tokens so shorter sequences match the longest sequence in a batch. The attention mask marks padding positions as 0 so the model ignores them. Set padding=True to pad to the longest sequence or pass max_length to pad to a fixed size.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b")
tokenizer(
    [
        "Sphinx of black quartz, judge my vow.",
        "Pack my box with five dozen liquor jugs.",
        "How vexingly quick daft zebras jump!"
    ],
    return_tensors="pt",
    padding=True,
)
{
    'input_ids': tensor([
        [     2, 235277,  82913,    576,   2656,  30407, 235269,  11490,    970,  29871, 235265],
        [     0,      2,   6519,    970,   3741,    675,   4105,  25955,  42184, 225789, 235265],
        [     0,      2,   2299,  73378,  17844,   4320, 224463,   4949,  48977,  9902, 235341]
    ]),
    'attention_mask': tensor([
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
    ])
}

Large language models pad on the left side to avoid disrupting generation, which predicts the next token from the right side.

Truncation

Truncation clips tokens so a sequence fits within a maximum length. Set truncation=True and specify max_length to enable it.

Padding and truncation work together. Short sequences gain padding tokens while long sequences lose trailing tokens. Together, they produce a packed rectangular tensor.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b")
tokenizer(
    [
        "Sphinx of black quartz, judge my vow.",
        "Pack my box with five dozen liquor jugs.",
        "How vexingly quick daft zebras jump!"
    ],
    return_tensors="pt",
    padding=True,
    truncation=True,
    max_length=5
)
{
    'input_ids': tensor([
        [     2, 235277,  82913,    576,   2656],
        [     2,   6519,    970,   3741,    675],
        [     2,   2299,  73378,  17844,   4320]
    ]),
    'attention_mask': tensor([
        [1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1]
    ])
}

Backends

Each model tokenizer is defined in a single file and supports four tokenization backends.

All backends inherit from PreTrainedTokenizerBase and share the same APIs for encoding, decoding, padding, truncation, saving, and loading. The difference is which tokenization pipeline runs underneath.

AutoTokenizer selects the best available backend when you call from_pretrained().

1. It reads the tokenizer_config.json file for the tokenizer_class field.
2. The registry matches tokenizer_class to a class name. The resolved class inherits from one of the four backends. For example, GemmaTokenizer inherits from TokenizersBackend, and SiglipTokenizer inherits from SentencePieceBackend. Some models, like GLM, map directly to TokenizersBackend because the tokenizer.json file fully describes the pipeline. GemmaTokenizer exists as a subclass since it defines additional model-specific settings in Python that tokenizer.json doesn’t capture. When a backend like mistral-common isn’t installed, AutoTokenizer falls back to TokenizersBackend.

Fallback to TokenizersBackend

Some models do not a dedicated tokenizer class, and some checkpoints have a tokenizer_class that doesn’t match their actual tokenizer.json file. AutoTokenizer handles both cases by loading the pipeline from tokenizer.json through the generic TokenizersBackend. It prioritizes the serialized tokenizer over the class name from the Hub, which fixes issues like when a mismatched tokenizer_class produces incorrect token ids.

A checkpoint resolves to a generic TokenizersBackend for one of three reasons.

The affected model types and checkpoints grow as configs are corrected on the Hub. For the current set, see the MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS and MODEL_IDS_TO_TOKENIZERS_BACKEND definitions in tokenization_auto.py.

The fallback is automatic and doesn’t change how you call from_pretrained(). The resulting tokenizer encodes and decodes exactly as tokenizer.json specifies. To override the choice, pass backend="tokenizers" or backend="sentencepiece".

Check which backend a tokenizer is using with the backend property.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b")
tokenizer.backend
'tokenizers'

Inspect the tokenizer architecture

Inspect a tokenizer’s internal components (normalizer, pre-tokenizer, model, decoder) with the _tokenizer attribute.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b")
print(tokenizer._tokenizer.normalizer)
print(tokenizer._tokenizer.pre_tokenizer)
print(tokenizer._tokenizer.model)
print(tokenizer._tokenizer.decoder)

Resources

• The Tokenization in Transformers v5 post discusses the motivation behind the new tokenization backends.
• Review the migration guide for an overview of the tokenization changes.