meta-llama

/

Meta-Llama-3-8B-Instruct

like 2.74k

I'm facing the same problem

Yes, For optimum performance we need to apply chat template provided by Meta
https://github.com/meta-llama/llama3/blob/main/llama/tokenizer.py#L202
[INST] [/INST], this template is for mistral or similar models, llama chat template is different

Here is the instruction template https://github.com/meta-llama/llama3?tab=readme-ov-file#instruction-tuned-models
Quoting below

The prompt begins with a <|begin_of_text|> special token, after which one or more messages follow. Each message starts with the <|start_header_id|> tag, the role system, user or assistant, and the <|end_header_id|> tag. After a double newline \n\n the contents of the message follow. The end of each message is marked by the <|eot_id|> token.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{{ system_prompt }}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{ user_msg_1 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{ model_answer_1 }}<|eot_id|>

I'm facing the same problem

on config.json change eos_token_id'to "eos_token_id": 128009

Anyone tried this with TGI? I used the following with for chatPromptTemplate but I get parseing errors (error: Parse error on line 7:...ssistant}}<|eot_id|>:

"chatPromptTemplate" : "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\r\n\r\n{{ You are a friendly assistant }}<|eot_id|><|start_header_id|>user<|end_header_id|>\r\n\r\n{{#each messages}}{{#ifUser}}{{content}}{{/ifUser}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\r\n\r\n{{#each messages}}{{#ifAssistant}}{{ content }}{{/ifAssistant}}<|eot_id|>",

Sorry - meant HuggingFace Chat UI not TGI - TGI working. When I use the old style Llama 2 prompt template (in HF Chat UI against TGI), the model returns garbage (expected as prompt not correct).

thanks

I'm facing the same problem

on config.json change eos_token_id'to "eos_token_id": 128009

Tried to do this but not working. Using it through vllm. Anyone has an idea?

Hello, I am using tgi version text-generation-inference:1.0.2 and hugging face chat-ui. I got to run Meta-Llama-3-8B with the following configuration:
Not sure if userMessageEndToken is needed, I did many testing and this is how it works, maybe in future will review the elements that are actually needed for this to work.

MODELS=[ { "name": "meta-llama/Meta-Llama-3-8B", "multimodal" : false, "description": "Llama-3-8B", "websiteUrl": "https://ai.meta.com/resources/models-and-libraries/llama-downloads/", "userMessageToken":"", "userMessageEndToken": "<|eot_id|>", "chatPromptTemplate":"<|begin_of_text|><|start_header_id|>system<|end_header_id|>{{@root.preprompt}}<|eot_id|>{{#each messages}}{{#ifUser}}<|start_header_id|>user<|end_header_id|>{{content}}<|eot_id|>{{/ifUser}}{{#ifAssistant}}<|start_header_id|>assistant<|end_header_id|>{{content}}<|eot_id|>{{/ifAssistant}}{{/each}}<|start_header_id|>assistant<|end_header_id|>", "parameters": { "temperature": 0.2, "top_p": 0.95, "repetition penalty": 1.2, "top_k": 50, "truncate": 1000, "max_new_tokens": 1024, "stop": [ "<|start_header_id|>", "<|end_header_id|>", "<|eot_id|>" ] }, "endpoints": [{ "type" : "tgi", "url": "http://127.0.0.1:8080" }] } ]

hello, for a conversational dataset intended for fine-tuning llama 3, at the assistant level, should we finally add a \n<|end_of_text|> token or not ?

for example : |start_header_id|>assistant<|end_header_id|>\n\n{assistant}<|eot_id|>\n<|end_of_text|>

King regards