Repository: promptfoo/promptfoo
Issue: Add support for multi-turn conversations with separate test configs
Root cause: Documentation Deficiency

**Describe the bug**
I'm trying to run multiple test configs, and each of these test configs should be conversational. However, I am unable to run anything after the first message is sent because I receive a JSON error.

**To Reproduce**
Here is my .yaml file:

```
prompts:
  - file://prompt.json

tests:
  - file://testcases/case1.yaml
  - file://testcases/case2.yaml
  ```
  
  Here is one test:
  ```
- vars:
    question: "I'd like to reschedule my appointment"

- vars:
    question: "Can I move it to Friday?"

- vars:
    question: "What time slots are available?"
```
Here is my prompt.json, as seen from the promptfoo docs about the [_conversation variable](https://www.promptfoo.dev/docs/configuration/chat/#using-the-conversation-variable):
```
[
  {% for completion in _conversation %}
    {
      "role": "user",
      "content": "{{ completion.input }}"
    },
    {
      "role": "assistant",
      "content": "{{ completion.output }}"
    },
  {% endfor %}
  {
    "role": "user",
    "content": "{{ question }}"
  }
]
```
However, I keep receiving the error. The first message is always sent successfully, but anything after is not:
`"Error: Chat Completion prompt is not a valid JSON string: SyntaxError: Unexpected token ↵ in JSON at position 165↵↵[↵  ↵    {↵      \"role\"…"`

and I'm not sure how to format my JSON differently. 

Here is the prompt I see while debugging in my console:
![Screenshot 2024-10-25 at 11 47 29 AM](https://github.com/user-attachments/assets/3fd5a027-cdb0-4f57-9361-a9ef7afb170e)

I even tried using a prompt.yaml but that made my tests treat each question as its own chat and lose the conversation aspect.
Please let me know if you have any ideas to fix my prompt.json/test configuration!


This prompt.yaml ended up working:
```
{% for completion in _conversation %}
- role: user
  content: "{{ completion.input }}"

- role: assistant
  content: "{{ completion.output }}"
{% endfor %}

- role: user
  content: "{{ question }}"
  ```

Did something change? The exact prompt.yaml from above is not working anymore:
```
reason: 'missed comma between flow collection entries',
  mark: {
    name: null,
    buffer: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json\n' +
      '\n' +
      '{% for completion in _conversation %}\n' +
      '- role: user\n' +
      '  content: "{{ completion.input }}"\n' +
      '\n' +
      '- role: assistant\n' +
      '  content: "{{ completion.output }}"\n' +
      '{% endfor %}\n' +
      '\n' +
      '- role: user\n' +
      '  content: "{{ question }}"\n',
    position: 75,
    line: 2,
    column: 1,
    snippet: ' 1 | # yaml-language-server: $schema=h ...\n' +
      ' 2 | \n' +
      ' 3 | {% for completion in _conversation %}\n' +
      '------^\n' +
      ' 4 | - role: user\n' +
      ' 5 |   content: "{{ completion.input }}"'
  }
}

Node.js v18.20.4
```

resolved with https://www.promptfoo.dev/docs/configuration/chat/#separating-chat-conversations