PangolinTokenizer / config.json
voidful's picture
Upload Pangolin tokenizer
82e506e verified
{
"model_type": "gpt2",
"tokenizer_class": "GPT2TokenizerFast",
"vocab_size": 114688,
"model_max_length": 131072
}