mistralai/Mistral-7B-Instruct-v0.2 · Model doesn't stop generation after answering the user question.

Apr 12

Model doesn't stop generation after answering the user question. It continues to take the role of user and ask new question and answers these new questions.
I am using the prompt as given below:
[INST] prompt [/INST]

alexrods

Apr 12

•

edited Apr 12

I got better results with the next code:

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda"
model_name = "mistralai/Mistral-7B-Instruct-v0.2"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

input_text = "ask to mistral"
input_ids = tokenizer.encode(input_text, return_tensors='pt')

model.to(device)
model_inputs = input_ids.to(device)

output_ids = model.generate(
        model_inputs
        )
output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)

dmsovetov

Apr 18

•

edited Apr 18

Stumbled upon your question while googling because of the same problem. As a workaround I used additional [INST] tags and seems that it works in my case.
So the final prompt is:

[INST] <I put a context here and instructions> [/INST][INST]Answer the following question: <The question goes here>[/INST]

Does anyone knows about other options?