Repository: promptfoo/promptfoo
Issue: Question: how to assert on 2+ chat conversation?

I am trying to test the ability to have a conversation with 2+ chats in the conversation.

Something like "he said, she said, he said, she said" (and assert on the last she said).

Does promptfoo support this?  And if yes, can you link any docs or examples of this?

Yes, have a look at the documentation on Chat Conversations! https://promptfoo.dev/docs/providers/openai#chat-conversations

This is in the OpenAI section as chat formatting is specific to the API (and as you know there are many APIs that implement OpenAI's format)

Ha nice, and wow!

Just a follow up question, where are the actual assertions there?

I am trying to figure out how to do a `similar` assertion on the last `assistant` response in a chat.

Everything under `vars.messages` is part of the prompt.  Any `assert` is run on the LLM output, i.e. the assistant response.

```yaml
prompts: [prompt.json]

providers: [openai:gpt-3.5-turbo]

tests:
  - vars:
      messages:
        - role: system
          content: Respond as a pirate
        - role: user
          content: Who founded Facebook?
        - role: assistant
          content: Mark Zuckerberg
        - role: user
          content: Did he found any other companies?
    assert:
      # Check that assistant response is "hello world"
      - type: equals
        value: hello world
```

Okay I am beginning to follow. I think in your example above, the `assert` will fail, because the `assistant` is going to not answer with "hello world".

Let's do a more complex example. Let's say in the `messages` there are 2+ `assistant` responses. Then the `assert`s provided below it are like global to all 2+ `assistant` responses?

@jamesbraza as far as I understand, the `messages` are passed into the LLM, the LLM will respond something and that response is asserted against. 

The assistant content as defined in `messages` is not used for assertions. Only the model's response to the given messages.

Okay I see @Peetee06, thanks for clarifying assertions are only done on the last response, that "made it click" for me.

So basically if one wants to assert on separate messages in the chat, one needs entirely separate `promptfooconfig.yaml`'s, one per target message. To give an example, to assert on the 1st AI response:

```yaml
# promptfooconfig_firstAIresponse.yaml

tests:
  - vars:
      messages:
        - role: system
          content: You are a helpful AI assistant.
        - role: user
          content: Who founded Facebook?
    assert:
      - type: icontains
        value: mark zuckerberg
```

Then to assert on the 2nd AI response:

```yaml
# promptfooconfig_secondAIresponse.yaml

tests:
  - vars:
      messages:
        - role: system
          content: You are a helpful AI assistant.
        - role: user
          content: Who founded Facebook?
        - role: assistant
          content: Mark Zuckerberg
        - role: user
          content: Did he found any other companies?
    assert:
      - type: icontains
        value: no
```

Closing this out

@jamesbraza a more convenient approach would be the following. There you only have one `promptfooconfig.yaml` and only need to run `promptfoo eval` once. It runs both tests against the same given prompt(s):

```yaml
tests:
  - vars:
      messages:
        - role: system
          content: You are a helpful AI assistant.
        - role: user
          content: Who founded Facebook?
    assert:
      - type: icontains
        value: mark zuckerberg
  - vars:
      messages:
        - role: system
          content: You are a helpful AI assistant.
        - role: user
          content: Who founded Facebook?
        - role: assistant
          content: Mark Zuckerberg
        - role: user
          content: Did he found any other companies?
    assert:
      - type: icontains
        value: no
```

Oh yeah that's a lot better, thank you for sharing that!

Hi @Peetee06, I am just getting started with promptfoo. I want to test conversational LLMs. It looks like the messages have to be hardcoded. does promptfoo allow generation of those by the LLM i want to test and a second LLM that behaves as a user?

@teezeit not sure about that, you might find more help in the promptfoo docs. I've just started using promptfoo myself and don't have too much experience yet.