Hugging Face's logo

voidful
/
PangolinTokenizer

Transformers
gpt2
tokenizer
byte-level-bpe
traditional-chinese
taiwan
multilingual
Model card Files
xet
 Community
PangolinTokenizer / tokenizer.json
voidful's picture
voidful
Add Open Formosa special tokens
989d8ec verified
raw
history contribute delete
9.36 MB
File too large to display, you can check the raw version instead.