SQLGenerator / tokenizer.json

Create tokenizer.json

4387ce7 verified over 1 year ago

155 Bytes

Invalid JSON: Unexpected token 'r', "from transf"... is not valid JSON

	from transformers import BertTokenizer

	tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
	tokenizer.save_pretrained("path_to_save_tokenizer")