english / tokenizer.json
hima06varshini's picture
Upload 6 files
827583c verified
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("your-model-name")
tokenizer.save_pretrained("model_directory")