Spaces:

HuggingFaceFW-Dev
/

lang-word-tokenizers

Running

App Files Files Community

lang-word-tokenizers / data

Commit History

georgian tokenizer and south azerbeijani

9a7091f
unverified

guipenedo commited on Oct 22, 2024

added khmer, tibetan and lao

baa687b
unverified

guipenedo commited on Oct 10, 2024

do not propagate to the root

49dc1e7
unverified

guipenedo commited on Oct 10, 2024

macrolanguages fix

bd41049
unverified

guipenedo commited on Oct 10, 2024

updated

0741edf
unverified

guipenedo commited on Oct 10, 2024

cleaned up unusued parent toks

79ddb0e
unverified

guipenedo commited on Sep 11, 2024

improved viz to include scripts

6a75090
unverified

guipenedo commited on Sep 11, 2024

added data

d43fb2c
unverified

guipenedo commited on Sep 9, 2024