File size: 1,826 Bytes
78c54ec | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | # Tokenizer Notes
The key teaching point is that the tokenizer is part of the model. It is not an interchangeable preprocessing detail.
## Tokenizers In This Repo
| File | Vocab | Use |
| --- | ---: | --- |
| `tokenizers/polish_bpe_32k.json` | 32768 | Paired with `model/ckpt.pt` |
| `tokenizers/rxlm_polish_bpe_65k.json` | 65536 | Separate later custom tokenizer artifact |
## 32k Polish BPE
Properties:
- byte-level BPE,
- 32768 vocabulary entries,
- 32511 merges,
- no normalizer,
- `add_prefix_space=False`,
- `<|endoftext|>` is the special document separator token.
This tokenizer is small enough that token ids fit safely in `uint16`, which is why the training shards can be compact raw binary files.
## 65k RXLM BPE
Properties:
- byte-level BPE,
- 65536 vocabulary entries,
- 65283 merges,
- NFKC normalization,
- `add_prefix_space=True`,
- 12 added tokens.
This is a different tokenizer. It should be taught as a later design variant, not as the tokenizer for `model/ckpt.pt`.
## Why Custom BPE
For Polish, a custom tokenizer can reduce awkward fragmentation of common morphemes, diacritics, inflected forms, and domain-specific text. The lesson is not that 32k is always best; the lesson is that tokenizer choice changes:
- effective context length,
- training cost,
- model embedding size,
- output fluency,
- evaluation comparability.
## Quick Inspection Snippet
```python
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizers/polish_bpe_32k.json")
text = "Zażółć gęślą jaźń. Polska jest częścią Europy."
enc = tok.encode(text)
print(enc.ids)
print(enc.tokens)
print(tok.decode(enc.ids))
```
## Compatibility Rule
For inference:
```text
checkpoint vocab_size == tokenizer vocab size
```
For this repo:
```text
model/ckpt.pt -> tokenizers/polish_bpe_32k.json
```
|