Repository: promptfoo/promptfoo
Issue: Question: How to test multiple plugins / run multiple suites at once?
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
We are using Semantic Kernel, which allows us to define multiple plugins. Each plugin has one prompt template and one config object, which includes OpenAI parameters like `max_tokens` and `temperature`. We have multiple plugins, and as such, would like to test all plugins with one command. Currently, we have to run `promptfoo evaluate` on each plugin, as each has its own provider and set of tests. 

**Describe the solution you'd like**
Ideally, we'd be able to run `promptfoo evaluate` on an array of `EvaluateTestSuite` objects and get one clean report and pass/fail result at the end of the command.

After running, we'd love to see an output.json that's an array of results, and a `promptfoo view` screen that's an array of tables with merged metrics (total cost, etc.) at the top.

**Describe alternatives you've considered**
Currently we are working on a custom script that calls `promptfoo evaluate` multiple times, and we are forgoing the (very nice!) database view provided by `promptfoo view` in favor of a single command that gives us a clear pass/fail result on all our plugins at once.

**Additional context**
To clarify, we have multiple providers because we provide different OpenAI parameters to each one. Each provider-promptTemplate-testBatch triplet is unique: we don't want provider 1 to run anything from testBatch 2, nor do we want testBatch 2 to be ran against a prompt from promptTemplate 3, if that makes sense.

For example, imagine:
Suite 1:
 - provider: StoryWriter, max_tokens: `10_000`
 - promptTemplate: "Write me a story about {{ input }}"
 - tests: Assert that `input` is present in the output as least 3 times, and the output is at least 5_000 tokens

Suite 2:
- provider: Brainstormer, max_tokens: `1_000`
- promptTemplate: "Give me some ideas for {{ input }}"
- tests: Assert that the output is a set of 3 bullet points

Clearly, we don't want to mix these, but we would like to be able to test both in one command! Apologies if I'm missing something!

Hi @mwiemer-microsoft,  you might want to check out this example for running several multi-turn conversations in one evaluation: [Multiple-Turn Conversation Example](https://github.com/promptfoo/promptfoo/tree/main/examples/multiple-turn-conversation).  

I’d need a bit more context about your configuration to make specific recommendations. If you could include a minimal example to reproduce the issue, that would be super helpful. Alternatively, feel free to email me at michael@promptfoo.dev, or book a slot on my calendar: https://cal.com/michael-dangelo/15min 

Thanks @mldangelo, and sorry for the delay. I think this is a good repro, adapted from https://github.com/promptfoo/promptfoo/blob/main/examples/azure-openai/promptfooconfig.yaml.

Suppose I have two promptfooconfig files:

```yml
# promptfooconfig-facts
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - 'Generate one very interesting fact about {{topic}}'

providers:
  - id: azure:chat:gpt-35-turbo-deployment1
    config:
      apiHost: 'your-org.openai.azure.com'

defaultTest:
  assert:
    - type: latency
      threshold: 3000
    # more assertions here specific to the fact: less than 3 sentences, family-friendly, etc.

tests:
  - vars:
      topic: monkeys
  - vars:
      topic: bananas
    assert:
      - type: similar
        value: Bananas are naturally radioactive.
        provider:
          id: azure:embeddings:ada-deployment1
          config:
            apiHost: 'your-org.openai.azure.com'
```

```yml
# promptfooconfig-math
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - 'Write a story for how to solve this complex math problem: {{problem}}'

providers:
  - id: azure:chat:gpt-4o-deployment1
    config:
      apiHost: 'your-org.openai.azure.com'

defaultTest:
  assert:
    - type: latency
      threshold: 3000
    # more assertions here specific to the story: length, correctness, etc.

tests:
  - vars:
      problem: How can I prove the Pythagorean Theorem geometrically?
  - vars:
      problem: Explain Andrew Wiles' solution to Fermat's Last Theorem
```

As you can see, I need to run these separately, as I don't want GPT-3.5 to try to understand Wiles' solution, nor do I want to ping the costly GPT-4o (or whatever "insert most expensive model is here") just to give me a monkey fact.

However, I'm not sure how to run both these suites without simply `promptfoo eval promptfooconfig-facts && promptfoo eval promptfooconfig-math` (or whatever the syntax would be)

Again, in a perfect world, I'd get a simple command to run both suites, and I'd be able to see a merged dashboard of total cost, cost by suite, cost by test, etc. For now, I'm just using a custom TypeScript config (closed-source, sorry) that basically does the above without the dashboard or insights.

It seems like this use case can be emulated using [scenarios](https://www.promptfoo.dev/docs/configuration/scenarios/) + [model specfic prompts](https://www.promptfoo.dev/docs/configuration/prompts/#model-specific-prompts) to essentially separate one promptfoo config file into 2 completely separate scenarios with their own providers, but that is a lot of additional configuration