How to use voidful/PangolinTokenizer with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("voidful/PangolinTokenizer", dtype="auto")
voidful commited on 