Repository: promptfoo/promptfoo
Issue: Isolated test-prompt combinations per config
Root cause: Unimplemented Feature Gap

**Description:** When executing `promptfoo eval -o report/report.html -c usecases/*-config.yaml`, all `prompts` and `tests` from all configuration files are combined. This means that each test is asserted against each prompt, regardless of the configuration file they are defined in.

However, there is currently no option to have isolated test-prompt combinations per configuration file. This could be useful in scenarios where specific tests should only be run against prompts defined in the same configuration file.

**Proposed Solution:** Introduce a new command-line flag or configuration option that allows users to specify whether `test`-`prompt` combinations should be isolated per configuration file or combined across all configurations.

For example, a new flag `--isolated` could be added to the promptfoo eval command:

`promptfoo eval -o report/report.html -c usecases/*-config.yaml --isolated`

When the `--isolated` flag is present, the evaluation would run each test only against the prompts defined in the same configuration file.

Alternatively, aquivalent to `scenarios` consisting of `var`-`test` pairs, `prompts` can have a `tests` property or a new property containing `prompts` and `tests` can be introduced.

**Benefits:**

Increased flexibility in running tests against prompts
Better organization and separation of concerns for different use cases or configurations
Easier debugging and troubleshooting when tests fail, as the scope is limited to a specific configuration file

Thanks for the suggestion!  I want to make sure I am understanding correctly.  It sounds like essentially you could just run the `promptfoo eval` command several times.  Is the catch that you want to view the results in a single view?

Yes, absolutely, you could execute `promptfoo eval` for each YAML file separately, but cannot take advantage of an easy asterix command like `usecases/*-config.yaml`, have to maintain or add another script for evaluating all YAML configurations and the output on the console and within the report are not aggregated in a single view.

+1, we ([in Chromium]) would like a way to tie prompts to specific tests instead of running every test against every prompt, collected across all configs. Right now, we can work around this by either:
1. Running each config in a separate `promptfoo eval` invocation
1. Have a single placeholder under `prompts` that simply subsitutes the actual prompt from a `TestCase`'s `vars`

... but these are both kind of ugly. In (1), we need to aggregate the test results (e.g., pass rate) ourselves. (2) kind of abuses the config semantics.

[in Chromium]: https://source.chromium.org/chromium/chromium/src/+/main:agents/testing/?q=f:agents%2Ftesting

Thanks @rgevrek for the detailed feature request and @jonathan-j-lee for the additional context from the Chromium team!

After investigating this, I found that **there's a solid workaround using the existing [dynamic test generation](https://www.promptfoo.dev/docs/configuration/test-cases/#dynamic-test-generation) feature** that achieves isolated test-prompt combinations without requiring changes to promptfoo.

## The Solution: JavaScript Test Generator

The key insight is: instead of trying to isolate configs at eval time, we can write a test generator that loads your separate config files and injects each config's prompt into its own tests.

### Setup

**1. Main config file (`promptfooconfig.yaml`):**

```yaml
description: Aggregated use cases with isolation

prompts:
  - '{{__prompt}}'

providers:
  - openai:gpt-4

tests: file://load_configs.js
```

**2. Install dependencies** (if not already in your project):

```bash
npm install js-yaml glob
```

**3. The test generator (`load_configs.js`):**

```javascript
import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
import yaml from 'js-yaml';
import { globSync } from 'glob';

// Get the directory where this script is located
const __dirname = path.dirname(fileURLToPath(import.meta.url));

export default function () {
  const testCases = [];

  // Glob pattern relative to this script's location
  const configFiles = globSync(path.join(__dirname, 'usecases/*-config.yaml'));

  for (const configFile of configFiles) {
    const configDir = path.dirname(configFile);
    const config = yaml.load(fs.readFileSync(configFile, 'utf8'));
    const useCaseName = path.basename(configFile, '-config.yaml');

    // Get prompt(s) from this config
    const prompts = Array.isArray(config.prompts) ? config.prompts : [config.prompts || ''];

    for (const prompt of prompts) {
      // Resolve file:// references if needed
      let promptText = typeof prompt === 'string' ? prompt : (prompt.raw || prompt.id || '');
      if (promptText.startsWith('file://')) {
        const promptPath = path.resolve(configDir, promptText.slice(7));
        promptText = fs.readFileSync(promptPath, 'utf8');
      }

      // Each test from THIS config gets paired with THIS config's prompt only
      for (const test of config.tests || []) {
        testCases.push({
          description: `[${useCaseName}] ${test.description || 'test'}`,
          vars: {
            ...test.vars,
            __prompt: promptText,
          },
          assert: test.assert || [],
          metadata: {
            ...test.metadata,
            useCase: useCaseName,
            sourceConfig: configFile,
          },
        });
      }
    }
  }

  console.log(`Loaded ${testCases.length} tests from ${configFiles.length} configs`);
  return testCases;
}
```

**4. Your use-case configs stay completely normal:**

```yaml
# usecases/travel-agent-config.yaml
prompts:
  - |
    You are a travel agent specializing in budget trips.
    User: {{query}}

providers:
  - openai:gpt-4

tests:
  - description: Flight booking
    vars:
      query: Find me a cheap flight to Paris
    assert:
      - type: llm-rubric
        value: Provides flight options
```

```yaml
# usecases/coding-assistant-config.yaml
prompts:
  - |
    You are a helpful coding assistant.
    Question: {{question}}

tests:
  - description: Hello world
    vars:
      question: Write hello world in Python
    assert:
      - type: contains
        value: print
```

### Directory structure

```
project/
├── promptfooconfig.yaml      # Main aggregator config
├── load_configs.js           # Test generator
└── usecases/
    ├── travel-agent-config.yaml
    ├── coding-assistant-config.yaml
    └── customer-support-config.yaml
```

### Running

```bash
# Run ALL configs aggregated with isolation
promptfoo eval

# Or run a single config standalone (still works!)
promptfoo eval -c usecases/travel-agent-config.yaml
```

## Why This Works Well

| Benefit | Description |
|---------|-------------|
| **Zero migration** | Your existing configs don't need restructuring |
| **True glob patterns** | Use any pattern in `globSync()` |
| **Dual-use configs** | Each config works standalone AND aggregated |
| **Aggregated results** | Single eval, single view in the UI |
| **Metadata tracking** | Filter results by `useCase` |

## Alternative: Scenarios (YAML-only)

If you prefer a pure YAML solution and don't mind restructuring, you can use [scenarios](https://www.promptfoo.dev/docs/configuration/scenarios/) with external files. Each scenario file defines its prompt as a variable:

```yaml
# promptfooconfig.yaml
prompts:
  - '{{systemPrompt}}\n\nUser: {{query}}'
providers:
  - openai:gpt-4
scenarios:
  - file://usecases/travel.yaml
  - file://usecases/coding.yaml
```

```yaml
# usecases/travel.yaml
description: Travel Agent
config:
  - vars:
      systemPrompt: You are a travel agent specializing in budget trips.
tests:
  - vars:
      query: Find me a cheap flight to Paris
    assert:
      - type: llm-rubric
        value: Provides flight options
```

---

I'm closing this as the test generator approach provides a solid workaround that:
- Supports glob patterns (`usecases/*-config.yaml`)
- Keeps tests isolated to their source config's prompts
- Shows aggregated results in a single view
- Doesn't require restructuring existing configs

If the community finds this insufficient, feel free to reopen. We could revisit a native `--isolated` flag if there's strong demand.