Repository: promptfoo/promptfoo
Issue: Bug: Simulated User Provider does not use Assistant Prompt
Root cause: Validation Gap

**Describe the bug**
Following the [Simulated User example from the docs](https://www.promptfoo.dev/docs/providers/simulated-user/) step-by-step, the Assistant does not incorporate the given prompt to generate new messages.

![Image](https://github.com/user-attachments/assets/d6b80a29-656c-4fd3-b3ae-5431b5daf95e)

I looked into why this happens, and I suspect the problem lies in this line:
https://github.com/promptfoo/promptfoo/blob/9f8ab34645394767710ce59748ab5a1c1e207808/src/providers/simulatedUser.ts#L93C7-L93C84

The `prompt` parameter of the `callAPI` is not passed to the `AgentProvider`, which makes it so that the agent does not use the given prompt to generate its following messages.

I managed to fix this by changing the line of the simulatedUser file to:

```ts
const response = await targetProvider.callApi(
            JSON.stringify([{ role: "system", content: prompt }, ...messages])
        );
```

However, I know that the `role: "system"` is not a general fix for this problem.

![Image](https://github.com/user-attachments/assets/2242d98c-259b-4d04-8fce-0e2d22d74274)

**To Reproduce**
1. Follow the [tau-simulated-user example](https://github.com/promptfoo/promptfoo/tree/9f8ab34645394767710ce59748ab5a1c1e207808/examples/tau-simulated-user)
2. Run promptfoo eval
3. See error

**Expected behavior**
The Assistant agent should use the given prompt to generate its next message. Something like this:

![Image](https://github.com/user-attachments/assets/243f2327-976a-4208-bcbe-6d55aef694ae)


**System information:**
Promptfoo version: v0.105.0


+1 for fixing this I also found that system prompt does not apply, resulting in the default OpenAI behavior.

Is there a known workaround until this is fixed? I saw the suggested code change in the above post, but is there any solution that doesn't require modifying the source code directly?!

Any guidance would be appreciated!

Thanks for flagging.  #3463 should fix this and a couple related issues.

@typpo This problem is happening again: Simulated User Provider does not use the Assistant Prompt in the version 0.117.8 of Promptfoo (I haven't tested version 0.117.10, but expect the same). It seems as if the simulatedUser provider has been rewritten and that this problem is reintroduced because of that rewrite.

I have fixed it in my local copy of the promptfoo library by doing the following:

1. open the promptfoo node module
2.open `dist/src/providers/simulatedUser.js`
3. change the agentResponse line to the following:

```
const prompt = context.prompt;
const agentVars = context.vars;
const agentInstructions = (0,
templates_1.getNunjucksEngine)().renderString(
    prompt.raw,
    agentVars
);

agentResponse = await this.sendMessageToAgent(
    [
        {
            role: "assistant",
            content: agentInstructions,
        },
        ...messagesToUser,
    ],
    context.originalProvider,
    context
);
```

I believe a real fix would do a very similar thing in `simulatedUser.ts` in [this line](https://github.com/promptfoo/promptfoo/blob/feaa4a8f9ab6f529bd0dcdcadc89ba968a34f00f/src/providers/simulatedUser.ts#L141)

Hello,

I'm experiencing the same issue.

When I use promptfoo:simulated-user as the provider, the system prompt isn't passed to the conversation.
The fix proposed by @kanarian is a good starting point, but I can't modify the Promptfoo source code every time I reinstall my dependencies.

Could you please consider reopening and fixing this issue, @typpo ?

Thank you!

I believe this was fixed by https://github.com/promptfoo/promptfoo/pull/6020 and it in the latest release. @maximepichou can you please pull the latest version and give it a try? Feel free to throw some time on my calendar if you would like to debug: https://cal.com/michael-dangelo/30min