openGPT-X

/

Teuken-7B-instruct-research-v0.4

like 66

Follow

OpenGPT-X 232

Hi!

I tried to fire the model with inference servers as vLLM and llama.cpp (as GGUF) and both fail when detecting the multiple chat templates for the different languages.

vllm        | ERROR 11-27 06:51:09 serving_chat.py:170] ValueError: This model has multiple chat templates with no default specified! Please either pass a chat template or the name of the template you wish to use to the `chat_template` argument. Available template names are ['BG', 'CS', 'DA', 'DE', 'EL', 'EN', 'ES', 'ET', 'FI', 'FR', 'GA', 'HR', 'HU', 'IT', 'LT', 'LV', 'MT', 'NL', 'PL', 'PT', 'RO', 'SK', 'SL', 'SV']

You can not choose the template names. You can only specify a complete chat template as alternative.

A more generic chat template as used by Mistral & Co would solve this.

May approach:

{%- if messages[0]["role"] == "system" %}
{{- messages[0]['role']|capitalize + ': ' + messages[0]['content'] + '\\n' }}
{%- set loop_messages = messages[1:] %}
{%- else %}
System: Ein Gespräch zwischen einem Menschen und einem Assistenten mit künstlicher Intelligenz. Der Assistent gibt hilfreiche und höfliche Antworten auf die Fragen des Menschen.{{- '\\n'}}
{%- set loop_messages = messages %}
{%- endif %}
{%- for message in loop_messages %}
{%- if (message['role']|lower == 'user') != (loop.index0 % 2 == 0) %}
{{- raise_exception('Roles must alternate User/Assistant/User/Assistant/...') }}
{%- endif %}
{%- if message['role']|lower == 'user' %}
{{- message['role']|capitalize + ': ' + message['content'] + '\\n' }}
{%- elif message['role']|lower == 'assistant' %}
{{- message['role']|capitalize + ': ' + message['content'] + eos_token + '\\n' }}
{%- else %}
{{- raise_exception('Only user and assistant roles are supported!') }}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- 'Assistant: '}}
{%- endif %}

This worked for me. As far as I know the default fixed System prompt I added as fallback is not standard in those other templates.

Kind regards,

Christian Stelter