Hugging Face's logo

ThingAI
/
DwarfGoToken

English
tokenizer
bpe
shell
code
chatml
Model card Files
xet
 Community
ThingsAI commited on
Commit
61cb92f
·
verified ·
1 Parent(s): c527fc5

Upload folder using huggingface_hub

Browse files
Files changed (5) hide show
  1. _raw/merges.txt +0 -0
  2. _raw/tokenizer.json +0 -0
  3. _raw/vocab.json +0 -0
  4. gotoken_meta.json +1 -1
  5. tokenizer.json +0 -0
_raw/merges.txt CHANGED
The diff for this file is too large to render. See raw diff
 
_raw/tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
_raw/vocab.json CHANGED
The diff for this file is too large to render. See raw diff
 
gotoken_meta.json CHANGED
@@ -2,6 +2,6 @@
2
  "tokenizer_type": "gotoken-bpe",
3
  "vocab_size": 8192,
4
  "gotoken_version": "0.1.1",
5
- "protected_tokens": 60,
6
  "target_model_size": "15M"
7
  }
 
2
  "tokenizer_type": "gotoken-bpe",
3
  "vocab_size": 8192,
4
  "gotoken_version": "0.1.1",
5
+ "protected_tokens": 22,
6
  "target_model_size": "15M"
7
  }
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff