mistralai

/

Mistral-7B-Instruct-v0.2

like 2.29k

Dear community,

We are using Mistral-7B-Instruct-v0.2 (off-the-shelf, no fine-tuning etc.) with a chat template in order to accept the System prompts (as user input) from various GUI chat clients such as ChatBox (https://github.com/Bin-Huang/chatbox) and ChatGPT-lite (https://github.com/blrchen/chatgpt-lite).

Problem:

The issue we observe with this chat template together with Mistral is the following: Often times (after a couple of chat turns), Mistral starts generating repetitive response and goes on for a very long time, as if it does not know when to stop. It tends to show this behavior especially on basic questions such as hi, who are you?, and tell me about yourself. Has anyone experienced such a behavior? Do you spot any potential issue with the template we are using? Any hints here would be highly appreciated!

The chat-template is below, which is the same as https://github.com/chujiezheng/chat_templates/blob/main/chat_templates/mistral-instruct.jinja only with an adaptation to remove the redundant newlines:

{% if messages[0]['role'] == 'system' -%}
    {% set loop_messages = messages[1:] -%}
    {% set system_message = messages[0]['content'].strip() + '\n\n' -%}
{% else -%}
    {% set loop_messages = messages -%}
    {% set system_message = '' -%}
{% endif -%}
{{ bos_token -}}
{% for message in loop_messages -%}
    {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
        {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
    {% endif -%}
    {% if loop.index0 == 0 -%}
        {% set content = system_message + message['content'] -%}
    {% else -%}
        {% set content = message['content'] -%}
    {% endif -%}
    {% if message['role'] == 'user' -%}
        {{ '[INST] ' + content.strip() + ' [/INST]' -}}
    {% elif message['role'] == 'assistant' -%}
        {{ ' '  + content.strip() + ' ' + eos_token -}}
    {% endif -%}
{% endfor -%}

Many thanks for your help!