MaziyarPanahi

/

Meta-Llama-3-70B-Instruct-GGUF

like 134

Text-generation-webui hasn't added instruct template for llama-3 yet. I've written my own that I think is correct, but not sure:
{%- set ns = namespace(found=false) -%}
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
{%- set ns.found = true -%}
{%- endif -%}
{%- endfor -%}
{%- if not ns.found -%}
{{- '<|start_header_id|>system<|end_header_id|>\n\nAnswer the questions.<|eot_id|>' -}}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{- '<|start_header_id|>system<|end_header_id|>\n\n' + message['content'] + '<|eot_id|>' -}}
{%- else -%}
{%- if message['role'] == 'user' -%}
{{-'<|start_header_id|>user<|end_header_id|>\n\n' + message['content'] + '<|eot_id|>'-}}
{%- else -%}
{{-'<|start_header_id|>assistant<|end_header_id|>\n\n' + message['content'] + '<|eot_id|>'-}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{-''-}}
{%- endif -%}

Awesome! Here's mine, see if we need to improve it:

instruction_template: |-
  {{-'<|begin_of_text|>'}}  
  {%- for message in messages %}
      {%- if message['role'] == 'system' -%}
          {{- '<|start_header_id|>system<|end_header_id|>\n\n' + message['content'].rstrip() + '<|eot_id|>' -}}
      {%- else -%}
          {%- if message['role'] == 'user' -%}
              {{-'<|start_header_id|>user<|end_header_id|>\n\n' + message['content'].rstrip() + '<|eot_id|>'-}}
          {%- else -%}
              {{-'<|start_header_id|>assistant<|end_header_id|>\n\n' + message['content'].rstrip() + '<|eot_id|>' -}}
          {%- endif -%}
      {%- endif -%}
  {%- endfor -%}
  {%- if add_generation_prompt -%}
      {{-'<|start_header_id|>assistant<|end_header_id|>\n\n'-}}
  {%- endif -%}

‌‌Firstly, I would like to thank the Owner for your outstanding work. However, I am unable to run this model on Ollama. To be more specific, I managed to import it into Ollama through the Modelfile, but the generated responses could not be stopped. Therefore, I drew on the valuable experience shared by several people above and modified the Modelfile to the following form:

FROM ./Meta-Llama-3-70B-Instruct.IQ2_XS.gguf

TEMPLATE """{{ if .System }<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER stop "<|reserved_special_token"

But the output still cannot be stopped. Can anyone help me?

But the output still cannot be stopped. Can anyone help me?

Has been resolved. For reference, I used the following Modefile

FROM ./Meta-Llama-3-70B-Instruct.IQ2_XS.gguf

PARAMETER temperature 1
PARAMETER num_ctx 1024

SYSTEM """You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability."""

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>
{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>
{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>
{{ .Response }}<|eot_id|>"""

PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER stop "<|reserved_special_token"

Thanks @SuperMaxine for sharing this template. Very useful, this is how TGI and LM Studio also use stop string(s) to simply stop the generation.