jsgreenawalt

/

gemma-2-9B-it-advanced-v2.1

like 21

Hey daswd,

I haven't tried it for coding beyond a couple of very small python scripts. I'm using Top P 0.90, Top K 40, I'm not setting Min P exclusively but I assume that's defaulting to zero (ollama)

I'm mostly using the model conversationally and for a series of roleplay and story generation prompt templates, Temperature between 0 and 0.7 mostly. For code generation or editing you probably want a very low temperature. It's possible though that this model just isn't great for that use-case since it hasn't been explicitly fine-tuned for coding, or it could be an issue with the training data sets. I've had the same issue with both GPT-4 and Claude in the past RE not generating full code, placeholders everywhere, even when instructed not to. Claude is much-improved on that front in the last few versions. You're also constrained to fairly short pieces of code with the context window of 8192 tokens, and the model may be dropping things or ignoring that instruction due to a response length bias in the training data

Thank you.
Regardless of it's lack of targeted training in programming, I've used plenty of LLM's that are supposedly trained for it and are useless, whereas I get good results from this.
With your permission I might give this AI some targeted training/merges.

Yeah, by all means have at it!

Are there prompts / parameters i can use to get longer responses from this model? i often ask it many questions within a single prompt and it usually answers only few of them before, i guess, running out of the predetermined preffered response length

I haven't tried multiple questions, but I do have a lot of general and specific organized instructions in some of my writing prompts. Usually they're geared at only producing one or two paragraphs at a time. I use a 'verbosity' phrase in my prompts, "Write one long paragraph", "Write ten sentences" or "Write three paragraphs" -- that's worked pretty well for me. I tried this in ollama:
"Tell me about World War Two. Write at least ten paragraphs of at least four long sentences each."
I get 8 or 9 paragraphs out of it. I tried telling it to write twenty paragraphs, but I still got only nine. However, when I told it to number the paragraphs, I got exactly twenty paragraphs out of it "Write twenty numbered paragrahs with at least six sentences each." You might try numbering your questions and telling it explicitly to number the responses, if you're not already doing that. I've also had good luck with this approach:

1. Describe the general task in a single sentence ex "You are tasked with answering the following questions"
2. Describe the information that will be presented "You will be presented with a list of {n} questions, followed by a list of detailed instructions. Read each question carefully. Follow all instructions"
3. List of questions
4. Specific instructions at the end, ex "- Number your responses according to the question numbers. - Work through the questions, one-by-one. - etc"

Providing more structure seems to really help in some cases, YMMV

Here's an example of a prompt that I'm using:

        il = "\n".join(instructionList)
        message = f"""
You are tasked with portraying the character {self.name} in a collaborative story. Specifically, you are to write {self.name}'s next turn from a first-person perspective. You will be given the following:
    - Task Information
        - The most recent line of the story
        - The full story
        - {self.name}'s character description
        - {self.name}'s current thoughts
        - Additional notes (if any)
    - Specific Instructions

Contextualize the most recent line of the story in the context of the full story. Be sure you understand the flow of knowledge, which character said what and when, and how the last line relates to everything that came before. Note that {self.name} is not aware of the private thoughts of others unless they're voiced by being enclosed in quotes. Utilize all information given when crafting your response, subject to the preceding caveat and specific instructions which will be supplied later.

## Task Information

- This is the most recent line of the story:
'''
{publicDialog[-1]}
'''

- This is the full story:
'''
{pdString}
'''

- {self.name}'s description:
'''
{self.characterDescription}
'''

- {self.name}'s current thoughts (private to {self.name} and not to be explicitly quoted or referenced):
'''
{self.thoughts}
'''

- Additional notes:
'''
{self.note}
'''

## Specific instructions:

Sanity Offset (if any): {self.sanityOffset}
Aggression Offset (if any): {self.aggressionOffset}
Verbosity: {verbosity}

{il}

Begin immediately below with {self.name}'s response. Do not add any tags or commentary. Don't quote your entire response, only the "spoken" parts of it (if the entire response is spoken, then you *should* quote it).
""".strip()

the 'instructionList' {il} looks like this:

        instructionList =\
        [
            f"- Write in first-person as {self.name}",
            #f"- Pay attention to any notes or instructions from the narrator, regardless of where they appear in the story.",
            #f"- Examine the most recent line of the story",
            #f"- Examine {self.name}'s description very carefully",
            #f"- Examine the Full Story",
            #f"- Pay attention to any notes or on-the-fly instructions from the narrator",
            #f"- IMPORTANT: Do *NOT*:",
            #f"  - repeat",
            #f"  - reiterate",
            #f"  - rephrase",
            #f"  any part of {self.name}'s prior responses.",
            f"- Do *NOT* suggest activities, especially not more than once",
            f"- Do *NOT* carry trivialities forward from {self.name}'s prior conversational turn, ex, talking about specific foods, alcohols, or other activities or consumables, especially not when there's an important conversation happening.",
            #f"- Text that is enclosed inside quotes represents spoken dialogue that is audible (visible) to all characters including {self.name}.",
            #f"- Only text enclosed in quotes was spoken aloud. {self.name} should *not* be aware of the private thoughts of others.",
            #f"- When crafting your response as {self.name}, you must not write any reactions or references to the the private non-quoted thoughts of others.",
            #f"- Don't respond to your own private thoughts, it creates a non-sequitur.",
            #f"- Keep an eye out for any abrupt changes or turns in the conversation, respond appropriately in context when something unexpected happens or is said.",
            #f"- Keep track of where {self.name} and other characters are physically located.",
            #
            #f"- IMPORTANT: When crafting your response, take action yourself in a natural way, instead of asking someone to do something for you. Banned phrases include:",
            #f"  - Hand me that...",
            #f"  - Pass me that...",
            #f"  - Now, Pass me that...",
            #f"  - Fetch that...",
            #f"  etc.",
            #f"- Be sure to address new developments or statement that have happened since {self.name}'s prior turn. {self.name}'s attention should follow the tail of the story closely (the newest line at the end).",
            f"- Focus heavily and primarily on {self.name}'s immediate responses to the *most recent* dialogue and narration, while considering the entire story context. Your response must take into account the most recent line of the story.",
            #f"- Don't talk about things that you're going to do, actually *do* them in the context of the story instead, 'show, don't tell'.",

            #f"- Pay careful attention to pronoun agreement in "spoken" (quoted) text as you write; be sure to maintain a clear grammatical and logical separation between your "spoken" (quoted) words and your narrated thoughts and actions.",
        ]
        il = "\n".join(instructionList)

Most of those are commented out, since they were put in for specific models, it's messy but you get the idea

thank you for such informative suggestions!