Is instruction format necessary
#142
by
supercharge19
- opened
In .md it says that following format must be complied with for good generation [INST] Instruction [/INST] Model answer [INST] Follow-up instruction [/INST]
however in code example below it was not followed
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
messages = [
{"role": "user", "content": "What is your favourite condiment?"},
{"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
{"role": "user", "content": "Do you have mayonnaise recipes?"}
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
outputs = model.generate(inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))```
Does this mean "apply_chat_template" takes care of this and the format is embedded in some json files? Or is the example incorrect and should be changed to end confusion?
The role of Apply_Chat_Template is exactly to apply the correct prompt template instruction so it works. So yeah, you dont need to use it if you are using the chat template, it handles it for u.
@pandora-s
what about quantized versions of the model, does a quantized version already include correct format or do I have to use [INST] I put a space after s so that line is not striked through </ s> [/INST] ?