Hugging Face's logo

chrishayuk
/
v11-tokenizer

Transformers
English
tokenizer
language-modeling
Model card Files
xet
 Community
v11-tokenizer

Commit History

Publish tokenizer 10dd51100331ab50 (adopted)
361c586
verified

chrishayuk commited on

Publish tokenizer 10dd51100331ab50 (adopted)
b779c03
verified

chrishayuk commited on

Publish tokenizer 10dd51100331ab50 (adopted)
fa41352
verified

chrishayuk commited on

v11: byte-safety fix (0 UNK, 0 round-trip mismatches on the 32-file gate corpus)
7e96a85
verified

chrishayuk commited on

initial commit
0c2f3cd
verified

chrishayuk commited on