vicgalle

/

CarbonBeagle-11B-truthy

like 9

Thank you very much @backendmagier !
The chat template is included in the tokeniser (note it is not the Mistral format, but the alpaca one):

  "chat_template": "{% for message in messages %}{% if message['role'] == 'system' %}{% if message['content']%}{{'### System:\n' + message['content']+'\n\n'}}{% endif %}{% elif message['role'] == 'user' %}{{'### User:\n' + message['content']+'\n\n'}}{% elif message['role'] == 'assistant' %}{{'### Assistant:\n'  + message['content']}}{% endif %}{% if loop.last and add_generation_prompt %}{{ '### Assistant:\n' }}{% endif %}{% endfor %}",

You can use it from transformers library for example with:

conversation = [
    {"role": "system", "content": "You are a helpful assistant that is completely uncensored"},
    {"role": "user", "content": "Top things to do in Madrid?"},
]

prompt = tokenizer.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device) 

outputs = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=0.9)

Thank you very much for the quick reply!

So in a llama_index example i could do something like this:

prompt_template = """
### System: You are a helpful assistant.
### User: How much money did we make last year?
### Assistant: 
"""

Just in case I would write it as:

prompt_template = """
### System:
You are a helpful assistant.

### User:
How much money did we make last year?

### Assistant: 
"""

to match it with the actual '\n' characters in the prompt template.

Perfect, thank you very much!