metadata
license: other
library_name: transformers
tags:
- tokenizer
- byte-level-bpe
- traditional-chinese
- taiwan
- multilingual
PangolinTokenizer
Byte-level BPE tokenizer for Traditional Chinese, Taiwan text, multilingual text, rich transcription, OCR-style text, and generic control formats.
This revision adds the Open Formosa required control tokens as special tokens. The base BPE vocabulary size remains 114,688. The effective tokenizer length, including added special tokens, is 114,822.
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"voidful/PangolinTokenizer",
trust_remote_code=False,
)
text = "<|system|>台灣健保與注音ㄅㄆㄇ，Tailo: Tâi-uân"
ids = tokenizer.encode(text)
decoded = tokenizer.decode(ids)
Open Formosa Compatibility
- Required special tokens present: 157
- Required special tokens encode as single IDs: yes
- Standard special tokens:
<unk>,<s>,</s>,<pad> - Model max length metadata: 131,072
trust_remote_code: not required- No discrete audio codec token ranges are included.
- No dense timestamp token ranges are included.