Rename original/tokenizer.model to tokenizer.model
#6
by
winglian
- opened
tokenizer.model should be in the root to work with Huggingface Trainer
Also needs a similar PR for 70B
No because it's not supported by the standard slow tokenizers (it's tiktoken based not sentencepiece based)
winglian
changed pull request status to
closed