teknium

/

OpenHermes-2.5-Mistral-7B

like 784

Hello,

I'm not getting a proper prompt string from huggingfaces transformers AutoTokenizer apply_chat_template for this model when using add_generation_prompt=True as a parameter to apply_chat_template.

Here is the code that causes what I'm seeing:

from transformers import AutoTokenizer

MODEL =  "teknium/OpenHermes-2.5-Mistral-7B"

tokenizer = AutoTokenizer.from_pretrained(MODEL ,  skip_special_tokens=True, use_fast=True)

test_msg = [
        {"role": "system", "content": "You are Alex, a friendly AI assistant."},
        {"role": "user", "content": "Hello, how are you?"},
        {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
        {"role": "user", "content": "Tell me a good joke! Not a short one either."},
    ]

prompt_str = tokenizer.apply_chat_template(test_msg, tokenize=False, add_generation_prompt=True)

print(f'"""{prompt_str}"""')

Which outputs:

"""<|im_start|>system
You are Alex, a friendly AI assistant.<|im_end|>
<|im_start|>user
Hello, how are you?<|im_end|>
<|im_start|>assistant
I'm doing great. How can I help you today?<|im_end|>
<|im_start|>user
Tell me a good joke! Not a short one either.<|im_end|>
"""

Notice that its missing "<|im_start|>assistant\n" at the end. Which should be included as noted in the HF documentation: https://huggingface.co/docs/transformers/main/chat_templating#what-are-generation-prompts

I can do a dirty fix by just appending what I need to the end of the prompt string after calling apply_chat_template. But I feel that it would be best that 'add_generation_prompt' worked as intended for this model.

I'm not sure if this is a tokenizer config template that is malformed and accompanying this models repo files, or if it's something with HF's transformer tokenizer library.

I feel that I'm doing things properly and the issue is likely not on my side. If not, I'm happy to hear some feed back.

If I'm doing things properly, I hope that me pointing out this unexpected behavior helps in some way.

Thanks for taking time to read this.