norallm

/

normistral-7b-warm-instruct

like 20

I believe Lucas is also working on that :) We found the default hyperparameters of llama-cpp a bit strange, it was better to turn off the repetition penalty (set it to 1.0) and to set the temperature to a lower value; the model behaved very chaotically otherwise. I'm sure there's more that can be done about these (and other) hyperparameters, they influence the outputs more than I'd like.

First: Thank you very much for providing gguf-files for the instruct model. It made the life for us amateurs a little more easy. 🤩

I am a total beginner and are experimenting a little with the instruct-model on Ollama. Does anyone have som tips for parameter setting that works well?

Currently my Ollama-modelfile looks like this:

TEMPLATE """
{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>{{ end }}
{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
"""
SYSTEM """Du er en vennlig assistent som skal svare på spørsmål? Svar kort og konkret på spørsmål"""
PARAMETER num_ctx 4096
PARAMETER stop "<|im_end|>"
PARAMETER temperature 1

The responses are ok-ish, but I wonder if the settings/modelfile can be improved.

Here is my Modelfile using the suggestions from @davda54 and the README. I made the following changes to the Modelfile:

• a space before system, user and assistant (as suggested in the README).
• set the repeat penalty to 1.0
• the temperature to 0.3
• added extra stop parameters

TEMPLATE """
{{ if .System }}<|im_start|> system
{{ .System }}<|im_end|>{{ end }}
{{ if .Prompt }}<|im_start|> user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|> assistant
"""
SYSTEM """Du er Kari Nordmann, en jovial og hjelpsom assistent som svarer kort og konsist på spørsmål."""
PARAMETER num_ctx 4096
PARAMETER temperature 0.3
PARAMETER repeat_penalty 1.0
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"
PARAMETER stop " user"
PARAMETER stop " assistant"

It behaves quite well, but it is still a bit to verbose for my taste, so I will experiment further with different system messages.

Thanks for the response @rsolva ! Verbosity is definitely a problem, we tried to augment the open instruction datasets by step-by-step reasoning, detailed descriptions, etc. and apparently we overdid it :) It's on our list of things to improve in the next release. By the way, if you noticed some other recurring unwanted patterns in the outputs, please let us know your feedback!