Repository: promptfoo/promptfoo
Issue: Support for projects with "Multiple prompt styles"
Root cause: Unimplemented Feature Gap

Hi, I spent today messing with Promptfoo, it's a cool project. 

I see that the design is essentially meant to do "horizontal" comparisons of prompts intended for the same use case for comparative performance evaluation. Are there plans to support multiple of these sorts of prompts in one config?

For example, in a single project, I might have a bunch of prompts that are used for different things. It'd be nice to be able to get everything working in the same config. It doesn't seem to work well currently as prompts for different use cases would have different variables entirely.

Thanks for comment.  This is similar in spirit to #57, where the solution is to make additional prompt configs but that is tedious (and does not lend itself well to reusability).

One option would be to let developers set up multiple test suites in a file.  Another option would be to let you pass multiple test suite configs to the `promptfoo eval` runner..

Thanks for the quick reply. It is pretty similar, but I think 1 difference would be that in the use case I'm considering, each prompt could have very different variables. I actually don't hate multiple prompt configs, since they really are different, but it's a bit clunky, you're right.

I think allowing multiple test suites in a file makes sense to me, everything would work the same as now, except there could be multiple test blocks pointing to a prompt file and a bunch of tests?

We also have a slightly different use case, where we have more complicated chains of prompts on service side. We don't really want to test the individual prompts, but instead the final output of these chains (basically simulate what a user would see at the end -> have promptfoo call our API with certain options).

For this we implemented a custom provider and apply a patch (via `patch-package`) on promptfoo which simply passes down the `vars` object to the providers, in addition to the `prompt` string.

For us, the `prompt` string given to promptfoo is just `Run #1` etc and instead of the prompt, we use the vars which are set on the individual tests and send them to our API. The result string is then returned to promptfoo as usual.

We trigger `promptfoo.evaluate` inside of a custom script and we can specify a suite of individual tests to run (ie `yarn test:ai foo` or `yarn test:ai bar` will each only pass their respecitve test cases to evaluate).

The `TestCase` object in our case gets extended with specific additional (typed) keys in the `vars` object, where we can use our existing types for this API request (so it's all nicely typed across the board) - it would be nice if `TestCase` would accept generics for its vars 🤔

We might create some PRs with suggestions / additions here for promptfoo, but since I'm going to be on vacation the next 2 weeks it could take some time 😅   

I'm having similar cases. I need to run multiple totally different prompts against the same set of LLMs. Each prompt has different variables and assertions. The only way I can see now is to write different config for each prompt which is ok as I can manually combine the results. But it would be great to be able to see the results in the same report.

Hi @zhlmmc,

I think [Scenarios](https://promptfoo.dev/docs/configuration/scenarios) solve your problem.  It's a way to include multiple sets of vars and tests on the same set of LLMs.

@flipace I'm late to your feedback, but https://github.com/promptfoo/promptfoo/commit/d9623cbd2814919a9046b97f59673bb8f31d7804 makes `TestCase` generic and also passes `context: {vars}` as a second argument to `callApi`.  It will be in the next release :)

> Hi @zhlmmc,
> 
> I think [Scenarios](https://promptfoo.dev/docs/configuration/scenarios) solve your problem. It's a way to include multiple sets of vars and tests on the same set of LLMs.
> 
> @flipace I'm late to your feedback, but [d9623cb](https://github.com/promptfoo/promptfoo/commit/d9623cbd2814919a9046b97f59673bb8f31d7804) makes `TestCase` generic and also passes `context: {vars}` as a second argument to `callApi`. It will be in the next release :)

Hi,

I have tried to play with the Scenarios, but it seems not working in my case. What I need is actually "prompt-test" combination not "data-test" combination. In my case, all prompts are fixed prompts that do not need variables to input at runtime. The ideal output looks like this:
|   Prompt   |   GPT-3.5   |   GPT-4   |   Llama2   |   ChatGLM   |   PaLM   |
|------------|-------------|-----------|------------|-------------|----------|
|     1      |     score        |    score       |     score       |    score         |    score      |
|     2      |             |           |            |             |          |
|     3      |             |           |            |             |          |
|     4      |             |           |            |             |          |
|     5      |             |           |            |             |          |

Not sure if there is a way to achieve this, thanks!

To get this sort of layout, I would suggest the following:

prompt.txt simply outputs the `prompt` var:
```nunjucks
{{ prompt }}
```

promptfooconfig.yaml sets multiple providers, with each unique prompt as a var:
```yaml
prompts: [prompt.txt]
providers: [openai:gpt-3.5-turbo, openai:gpt-4, llama2, chatglm, palm]
tests:
  - vars:
      prompt: <insert prompt 1 here>
  - vars:
      prompt: <insert prompt 2 here>
  - vars:
      prompt: <insert prompt 3 here>
  # ...
```

This will produce a table similar to what you have outlined above.