d0rj
/

diffusion-51M-base

tiny-llm-ablation

masked-language-modeling

Eval Results (legacy)

Model card Files Files and versions

Metrics Training metrics Community

diffusion-51M-base / tokenizer_config.json

d0rj's picture

Publish evaluated diffusion v2 with PLL intervals and TensorBoard traces

80aea5b verified 4 days ago

History Blame Contribute Delete

213 Bytes

	{
	"backend": "tokenizers",
	"bos_token": "<s>",
	"eos_token": "</s>",
	"is_local": true,
	"model_max_length": 2048,
	"pad_token": "<pad>",
	"tokenizer_class": "TokenizersBackend",
	"unk_token": "<unk>"
	}