Repository: promptfoo/promptfoo
Issue: Conv history documentation does not work
Root cause: Documentation Deficiency

New to the tool... I tried to follow the conversation history fixture in the opening documentation but the example just doesn't work. I have tried many tweaks but the prompt.json never does what it says in the example. 
My use case is i have a really big sys prompt which is followed by one assist prompt then I want to try multiple user prompts on the same large prompt... con history should do it but it does not work.. any ideas?

Hi @SeanyCochrane, thanks for flagging this.  You're right that there's a problem with the example in the docs - mainly that it used an inconsistent variable name.  

I've corrected this and also added a full example [here](https://github.com/promptfoo/promptfoo/tree/main/examples/openai-chat-history) that you can reference.  Let me know if you encounter any issues!

Great thanks it's working now... one additional question. My default tests system prompt is large... how would I sub in a file to the yml... e.g. 
Messages:
    - system: [file with large prompt in it]

As of version 0.23.1 it is now possible to load a `vars` value directly from file using the `file://` prefix.  Here's an example:

```yaml
  - vars:
      system_message: file://path/to/system_message.yaml
      foo: some other var...
```

You may find it helpful to set the `system_message` var in `defaultTest`.  This will make the var available in every test.  Something like this:

```yaml
defaultTest:
  vars:
    system_message: file://path/to/system_message.yaml

tests:
  - vars:
      previous_messages:
        # ...
```

I've updated the example to show an example of this: https://github.com/promptfoo/promptfoo/blob/main/examples/openai-chat-history/promptfooconfig.yaml

There is more documentation on how to load tests and vars from file here: https://www.promptfoo.dev/docs/configuration/guide#import-tests-from-separate-files.  Hope this helps!

Thanks typo. I upgraded to 23.1 and it loads the prompt file... however my sys prompt is quite large and complex so now I have 'bad indentation' errors. I think we'll need something to process text prompts.

Aha... if I put the prompt as single line enclosed in double quotes it works... but would be nice to be able to auto strip line feeds.

Hi @typpo   - just to let you know... in order to load a multiline sys prompt I did a rough hack in the test case.js... 

vars[key] = fs.readFileSync(value.slice('file://'.length), 'utf-8');
 // Sort out quotes and new lines
vars[key] = "'" + vars[key].replace(/(\r\n|\n|\r)/gm, "").replace(/\"/gm,"\\\"").replace(/'/gm,"") + "'";

This now allows me to easily edit a large prompt avoiding single quotes. Not perfect but worth considering for future releases... 

@SeanyCochrane thanks, I see what you're saying here.  This change 64da64d4470a6a3169a5dd3c93f4961efa2b98ec strips newlines from imported vars and I've updated the [separate test configs](https://github.com/promptfoo/promptfoo/tree/main/examples/separate-test-configs) example to include a system message import.

There was a separate change a couple days ago that stopped files from being read as yaml, which should have solved the other indentation issue you mentioned.

I'm not sure the quote escaping is necessary, though.  Single and double quotes in vars are escaped if they are being included _within strings_ in a JSON prompt.  So maybe there is something else about your setup that we need to troubleshoot?

Ok just tried it... I no longer need enclosing single quote or remove single quote but I still need to escape the double quotes.... i.e.
vars[key] = vars[key].replace(/(\r\n|\n|\r)/gm, "").replace(/"/gm,"\"")...

Incidentally the progress bar when I use this method seems to get replace with " system_mes ... looked everywhere trying to figure out why... might be to do with the test heading which is a print out of prompt json

Agh... the replace is being replaced in GitHub... double quotes replaced with backslash double quote...

Would you be able to show me how you're including the system message var in your main prompt?

(config yaml)

prompts: [prompts2.json]

defaultTest:

  vars:

      system_message: file://mshare.txt

      previous_messages:



Prompts2.json

[

    {

      "role": "system",

      "content": "{{ system_message }}"

    },

    {% for message in previous_messages %}

      {% for role, content in message %}

        {

          "role": "{{ role }}",

          "content": "{{ content }}"

        },

      {% endfor %}

    {% endfor %}

    {

      "role": "user",

      "content": "{{ question }}"

    }

  ]



On Wed, 20 Sept 2023, 15:46 Ian Webster, ***@***.***> wrote:

> Would you be able to show me how you're including the system message var
> in your main prompt?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/136#issuecomment-1727878842>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A7K7EGGF3BP65WSXJ7TTEM3X3L6TLANCNFSM6AAAAAA4YDXV54>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Conversation history support has improved, feel free to reopen/open a new issue if you're still running into problems!