system prompt template
Hey all, I run into this error when I try to build a prompt using the tokenizer.apply_chat_template(messages, return_tensors="pt"). The problem is with the system message inside the messages there. When I remove the system message I won't get the jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/... error anymore..
that is because according to the tokenizer config here, the chat template does not support system messages. So I'm wondering how the model was trained for supporting system messages and how should we use it?
Hope this is helpful
How we do it
We use vLLM to provide an openai compatible chat endpoint that can use system messages. Together with langchain we are able to accomplish this.
Alternate solutions
For TS Junkies here is a solution.
If you don't have the hardware to run the model, here is a Mistral API solution using langchain in Python (Mistral API is in beta)
Here is a post with a python solution that may be more relevant to your error message. This one uses the user, assistant, user framework that the error mentions.
Example
inference_server_url = "http://localhost:8000/v1"
chat = ChatOpenAI(
model="mistralai/Mistral-7B-Instruct-v0.2",
openai_api_key="EMPTY",
openai_api_base=inference_server_url,
max_tokens=250,
temperature=0,
)
messages = [
SystemMessage(
content="You are a helpful assistant that translates English to Italian."
),
HumanMessage(
content="Translate the following sentence from English to Italian: I love programming."
),
]
chat(messages)
AIMessage(content=' Io amo programmare', additional_kwargs={}, example=False)
If this doesn't help, then it's at least a handy reference.