Repository: promptfoo/promptfoo
Issue: Is it possible to run a chat conversation and run tests for each model answer?
Root cause: Documentation Deficiency

I want to run a chat conversation with an LLM model and have some tests for the answer I'm going to get from each message.
Let's say that I have the following promptfooconfig.yaml file:

```yaml
providers:
   - openai:gpt-3.5-turbo:
       id: openai-gpt-3.5-lowtemp
       prompts: chat_prompt
       config:
         temperature: 0.2
         max_tokens: 128
defaultTest:
    options:
      provider: openai:gpt-3.5-turbo-0613
tests:
  - vars:
      message: What is Facebook?
    assert:
      - type: llm-rubric
        value: says that Facebook is a social network
  - vars:
      message: Who founded it?
    assert:
      - type: llm-rubric
        value: says that Facebook was created by Mark Zuckerberg
      - type: llm-rubric
        value: says that Mark Zuckberg did not found the company alone, but did it with other partners
  - vars:
      message: Did he also found another company?
```

Obviously, via average `promptfoo eval` this doesn't work since every query runs independently from the previous one:

![image](https://github.com/promptfoo/promptfoo/assets/13112588/e2f1cb8b-f61b-48d4-9fe3-8362db6135ff)

Is there any way for me to run these test cases with the information from previous messages being used by the next answers?


Hi @ayrtondenner,

Currently you can do this in two ways:

1. Use a [custom provider](https://promptfoo.dev/docs/providers/custom-api) that is stateful, and remembers the previous responses and adds them as chat messages, OR
2. Mock out the previous chat messages and include them in your test cases using nunjucks templating:

```
[
  {% for message in previous_messages %}
    {% for role, content in message %}
      {
        "role": "{{role}}",
        "content": "{{content}}"
      },
    {% endfor %}
  {% endfor %}
  {
    "role": "system",
    "content": "..."
  },
  ...
```

And then include it in your test cases like so:

```yaml
tests:
  - vars:
      previous_messages:
        user: Who founded Facebook?
        assistant: Mark Zuckerberg
      message: Did he also found another company?
```

Hope this helps!

The nunjuck file should be a txt or json file? I wrote the following `prompts/prompt.txt` file:

```
[
  {% for message in previous_messages %}
    {% for role, content in message %}
      {
        "role": "{{role}}",
        "content": "{{content}}"
      },
    {% endfor %}
  {% endfor %}
]
```

But when I try the following `promptfooconfig.yaml` file:

```yaml
prompts: ["prompts/prompt.txt"]

providers:
  - openai:gpt-3.5-turbo:
      id: openai:gpt-3.5-turbo
      config:
        temperature: 0

defaultTest:
    options:
      provider: openai:gpt-3.5-turbo
      assert:
        - type: llm-rubric
          value: does not describe self as an AI, model, or chatbot

tests:
  - vars:
      previous_messages:
        user: Who founded Facebook?
        assistant: Mark Zuckerberg
```

I get the following error:

![image](https://github.com/promptfoo/promptfoo/assets/13112588/0074d784-c5bd-4194-b8bd-69ce68d5e12c)

Hi @ayrtondenner, my apologies I made a typo in my last post.  The example template I provided assumes `previous_messages` is an _array_ of objects.

promptfooconfig.yaml:
```yaml
prompts: ["prompt.json"]

providers:
  - openai:gpt-3.5-turbo:
      id: openai:gpt-3.5-turbo
      config:
        temperature: 0

defaultTest:
    options:
      provider: openai:gpt-3.5-turbo
      assert:
        - type: llm-rubric
          value: does not describe self as an AI, model, or chatbot

tests:
  - vars:
      previous_messages:
        - user: Who founded Facebook?
          assistant: Mark Zuckerberg
```

Note that you also have to exclude the trailing `,` for valid JSON:

prompt.json:
```nunjucks
[
  {% for message in previous_messages %}
    {% for role, content in message %}
      {
        "role": "{{role}}",
        "content": "{{content}}"
      } {% if not loop.last %},{% endif %}
    {% endfor %}
  {% endfor %}
]
```

The example you updated worked, but I'm trying to mimic an entire conversation, and it seems that I cannot have more than one message per role:

```yaml
tests:
  - vars:
      previous_messages:
        - user: Who founded Facebook?
  - vars:
      previous_messages:
        - user: Who founded Facebook?
          assistant: Mark Zuckerberg
          user: Did he create any other companies?
```

It doesn't even send the prompt to GPT:

```cmd
YAMLException: duplicated mapping key (29:11)

 26 |       previous_messages:
 27 |         - user: Who founded Facebook?
 28 |           assistant: Mark Zuckerberg
 29 |           user: Did he create any other c ...
----------------^
  reason: 'duplicated mapping key'
```

Let's try this approach instead:

promptfooconfig.yaml:
```yaml
prompts: ["prompt.json"]

providers:
  - openai:gpt-3.5-turbo:
      id: openai:gpt-3.5-turbo
      config:
        temperature: 0

defaultTest:
    options:
      provider: openai:gpt-3.5-turbo
      assert:
        - type: llm-rubric
          value: does not describe self as an AI, model, or chatbot

tests:
  - vars:
      previous_messages:
        - role: system
          content: Respond as a pirate
        - role: user
          content: Who founded Facebook?
        - role: assistant
          content: Mark Zuckerberg
        - role: user
          content: Did he found any other companies?
```
prompt.json:
```nunjucks
{{ previous_messages | dump }}
```

> Let's try this approach instead:
> 
> promptfooconfig.yaml:
> 
> ```yaml
> prompts: ["prompt.json"]
> 
> providers:
>   - openai:gpt-3.5-turbo:
>       id: openai:gpt-3.5-turbo
>       config:
>         temperature: 0
> 
> defaultTest:
>     options:
>       provider: openai:gpt-3.5-turbo
>       assert:
>         - type: llm-rubric
>           value: does not describe self as an AI, model, or chatbot
> 
> tests:
>   - vars:
>       previous_messages:
>         - role: system
>           content: Respond as a pirate
>         - role: user
>           content: Who founded Facebook?
>         - role: assistant
>           content: Mark Zuckerberg
>         - role: user
>           content: Did he found any other companies?
> ```
> 
> prompt.json:
> 
> ```
> {{ previous_messages | dump }}
> ```

I tried this example in the past and it did worked, but I think that there should be a less verbose way of doing it, instead of having to type "role": "user" and "content": "..." every single time.

I was thinking about this approach here to solve the "duplicate key" problem:

```yaml
tests:
  - vars:
      previous_messages:
        - user: Who founded Facebook?
  - vars:
      previous_messages:
        - user: Who founded Facebook?
          assistant: Mark Zuckerberg
        - user: Did he create any other companies?
  - vars:
      previous_messages:
        - user: Who founded Facebook?
          assistant: Mark Zuckerberg
        - user: Did he create any other companies?
          assistant: "Yes, Mark Zuckerberg has been involved in the creation of other companies. Some notable ones include: The Chan Zuckerberg Initiative, Fwd.us, Internet.org, Oculus VR"
        - user: What is Mark Zuckerberg role at Internet.org company?
```

```json
[
  {% for message in previous_messages %}
    {% for role, content in message %}
      {
        "role": "{{role}}",
        "content": "{{content}}"
      } {% if not loop.last %},{% endif %}
    {% endfor %}
  {% endfor %}
]
```

But it seems that at every hyphen, nunjucks restarts the for loop, removing the comma in the wrong places:

```json
[
      {
        "role": "user",
        "content": "Who founded Facebook?"
      } ,
      {
        "role": "assistant",
        "content": "Mark Zuckerberg"
      } 
      {
        "role": "user",
        "content": "Did he create any other companies?"
      } ,
      {
        "role": "assistant",
        "content": "Yes, Mark Zuckerberg has been involved in the creation of other companies."
      } 
      {
        "role": "user",
        "content": "What is Mark Zuckerberg role at Internet.org company?"
      } 
]
```
![image](https://github.com/promptfoo/promptfoo/assets/13112588/13d14452-5d47-4386-9f65-363fee529ff8)


Ok, I guess I was able to do it. The key was to not use `{% if not loop.last %},{% endif %}`, but instead remove that and use a different notation (which was defined as `message`) to identify the last user message, and then remove the comma only in that case.

- promptfooconfig.yaml
```yaml
prompts: ["prompt.json"]

providers:
  - openai:gpt-3.5-turbo:
      id: openai:gpt-3.5-turbo
      config:
        temperature: 0

defaultTest:
    options:
      provider: openai:gpt-3.5-turbo
      assert:
        - type: llm-rubric
          value: does not describe self as an AI, model, or chatbot

tests:
  - vars:
      previous_messages:
        - message: Who founded Facebook?
  - vars:
      previous_messages:
        - user: Who founded Facebook?
          assistant: Mark Zuckerberg
          message: Did he create any other companies?
  - vars:
      previous_messages:
        - user: Who founded Facebook?
          assistant: Mark Zuckerberg
        - user: Did he create any other companies?
          assistant: "Yes, Mark Zuckerberg has been involved in the creation of other companies. Some notable examples include:\\n\\n1. The Chan Zuckerberg Initiative: Founded in 2015 by Mark Zuckerberg and his wife Priscilla Chan, this philanthropic organization aims to promote equality in areas such as education, science, and justice.\\n\\n2. FWD.us: Launched in 2013, FWD.us is an advocacy group co-founded by Zuckerberg that focuses on immigration and education reform in the United States.\\n\\n3. Internet.org: In collaboration with various technology companies, Zuckerberg launched Internet.org in 2013. The initiative aimed to provide internet access to underserved areas around the world.\\n\\n4. Oculus VR: In 2014, Facebook acquired Oculus VR, a virtual reality technology company. While not directly founded by Zuckerberg, this acquisition was a significant move by Facebook into the virtual reality space.\\n\\nThese are just a few examples of the companies and initiatives associated with Mark Zuckerberg."
          message: What is Mark Zuckerberg role at Internet.org company?
```

- prompt.json
```json
[
  {% for previous_message in previous_messages %}
    {% for role, content in previous_message %}
      {% if role != "message" %}
        {
          "role": "{{role}}",
          "content": "{{content}}"
        },
      {% else %}
        {
          "role": "user",
          "content": "{{content}}"
        }
      {% endif %}
    {% endfor %}
  {% endfor %}
]
```

- Result
![image](https://github.com/promptfoo/promptfoo/assets/13112588/a45ed4cb-c274-45a9-94e6-62f67051514c)


Here's another way to do it.  Not that pretty but saves you from having to key the last message differently:

promptfooconfig.yaml:
```yaml
prompts: ["prompt.json"]

providers:
  - openai:gpt-3.5-turbo:
      id: openai:gpt-3.5-turbo
      config:
        temperature: 0

defaultTest:
    options:
      provider: openai:gpt-3.5-turbo
      assert:
        - type: llm-rubric
          value: does not describe self as an AI, model, or chatbot

tests:
  - vars:
      previous_messages:
        - user: Who founded Facebook?
          assistant: Mark Zuckerberg
        - user: Did he create any other companies?
          assistant: "Yes, Mark Zuckerberg has been involved in the creation of other companies. Some notable ones include: The Chan Zuckerberg Initiative, Fwd.us, Internet.org, Oculus VR"
        - user: What is Mark Zuckerberg role at Internet.org company?
```
prompt.json:
```nunjucks
[
{% for message in previous_messages %}
  {% set outer_loop = loop %}
  {% for role, content in message %}
  {
    "role": "{{ role }}",
    "content": "{{ content }}"
  }{% if not (loop.last and outer_loop.last) %},{% endif %}
  {% endfor %}
{% endfor %}
]
```

This also worked, thanks!