Add tokenizer configuration

#1

Adds tokenizer configuration and sets the chat_template from allenai/OLMo-7B-Instruct.

ref https://github.com/ggerganov/llama.cpp/pull/6910#issuecomment-2100366560.

nopperl changed pull request title from add-tokenizer-config to Add tokenizer configuration
nopperl changed pull request status to open
Scientific Software Engineering Center @ University of Washington org

Thanks @nopperl ! Looks like this is in my local directory but not here. Must be an uploading error! Great that you caught it πŸ™‚

lsetiawan changed pull request status to merged

Sign up or log in to comment