Repository: promptfoo/promptfoo
Issue: Is there a way to Separate Tests by Prompt?
Root cause: Unimplemented Feature Gap

Assuming that I have a long list of tests, I want to run a set number of tests only for a specific prompt, or in other words I want to separate tests by prompt. Let's say I want to run the first five tests on `prompt1.txt`, while the next three tests should run on `prompt2.txt`. How can I achieve this scenario? 

So far what I've thought of is to have separate config files for each prompt but I'm wondering if it's achievable using a single config file.

I have the same question, @aliasghar98 did you manage to find a way to do this?

@robodasha For the time being, I have split my promptfoo configuration files to correspond to one prompt each. This results in a large number of configuration files (since I have a large number of prompts) and a lot of `promptfoo eval` runs, but I was able to find a way around that too using a python script.

@aliasghar98  can you please share your approach ? 

@dmockingbird551 Like I mentioned previously, I have separate config files corresponding to each prompt. Each config file contains a list of assertions, and I've configured the provider separately in these too (I needed to test different LLMs in my case). Another point to note here is that I am treating a _single config file_ as _one test case_. 

The next problem which I had was that I needed to run `promptfoo eval` on each config file separately. To automate this, I used a python script in which I used the subprocess library to spawn a new process for each `promptfoo eval`. I accumulated the results from the console's output in each case (`stdout`). There could be better approaches than this, but this is the one I had thought of. Let me know if you require further clarity regarding this.

I hope this helps you achieve what you're trying to do. 😄 

I came across the same issue and implemented a similar solution as suggested in the comments above. Here is the Python script I used to handle multiple configuration files. Note that all configuration files must start with config_ to be detected by this script:

"""
import glob
import subprocess
from pathlib import Path
from loguru import logger


class PromptfooEvaluator:
    """Class to handle promptfoo evaluation for YAML config files."""

    def __init__(self):
        """Initialize the evaluator."""
        self.config_files = "path_to_your_config_files"

    def get_config_files(self, pattern: str) -> list[str]:
        """Retrieve a list of configuration files matching the given pattern.

        Args:
            pattern (str): The glob pattern to match configuration files.

        Returns:
            list[str]: A list of matched configuration file paths.
        """
        config_files = glob.glob(pattern)
        logger.info(
            f"Found these {len(config_files)} configuration files matching pattern: {config_files}"
        )
        return config_files

    def extract_base_name(self, config_file: str) -> str:
        """Extract the base name from the configuration file path.

        Args:
            config_file (str): The path to the configuration file.

        Returns:
            str: The base name extracted from the configuration file path.
        """
        base_name = Path(config_file).stem.replace("config_", "")
        return base_name

    def generate_output_paths(self, base_name: str) -> tuple[str, str]:
        """Generate HTML and JSON output paths.

        Args:
            base_name (str): The base name for the output files.

        Returns:
            tuple[str, str]: The paths for the HTML and JSON output files.
        """
        html_output = Path(f"{base_name}.html")
        json_output = Path(f"{base_name}.json")
        return str(html_output), str(json_output)

    def run_subprocess(self, command: list[str]) -> None:
        """Run a subprocess command.

        Args:
            command (list[str]): The command to run as a subprocess.
        """
        subprocess.run(command, check=True)

    def run_evaluation(self, config_file: str) -> None:
        """Run the promptfoo evaluation for a given configuration file.

        Args:
            config_file (str): The path to the configuration file.
        """
        self.run_subprocess(
            [
                "python",
                "-m",
                "promptfoo",
                "eval",
                "--config",
                config_file,
            ]
        )

    def run(self) -> None:
        """Run the full promptfoo evaluation process."""
        config_files = self.get_config_files(f"{self.config_files}/config_*.yaml")
        for config_file in config_files:
            base_name = self.extract_base_name(config_file)
            logger.info(f"Processing configuration file: {base_name}")

            html_output, json_output = self.generate_output_paths(base_name)
            self.run_evaluation(config_file)
            logger.info(f"Evaluation completed for: {base_name}")

        logger.info("Completed the promptfoo evaluation process")


if __name__ == "__main__":
    evaluator = PromptfooEvaluator()
    evaluator.run()
    """

Also running into this problem - the solution I'm running with is just a makefile and separate calls to promptfoo, but it would be nice for some support out of the box for this type of project/test structure.

+1

+1

Another option is to use a base prompt that will load in the actual prompt as a variable.

So you have a single config like this:

yaml config
```
prompts:
  - file://prompts/base_prompt.txt
providers:
  - id: openai:gpt-4o-mini
tests: tests/*.yml
```

baseprompt.txt
```
{{actual_prompt}}
```

test/example.yml
```
assert:
- type: javascript
  value: file://assert.js
vars:
  actual_prompt: file://prompt1.txt
```
I don't know if there is a way to load assertions from a yaml file as that would be pretty ideal. You can use a csv file to load some assertions though.
Is there any other methods of doing this? I would prefer to aggregate all the result on my test runs into the promptfoo viewer

Also adding to this since I ran into the same problem.

My solution was to define a dummy variable like:
```
vars:
  prompt_num: 1
```

and then use a python script to define the prompt based on this variable.

For example define in `example_prompt.py`
```
def my_prompt_function(context: dict) -> str:

    variables: dict = context['vars'] # access the test case variables

    match variables["prompt_num"]:
        case 1:
            prompt = "Insert prompt 1"
        case 2:
            prompt = "insert prompt 2"
        
    return prompt
```

and in `promptfooconfig.yaml`:
```
prompts:
  - file://example_prompt.py:my_prompt_function

tests:

  - description: Test 1
    vars:
      prompt_num: 1
  - description: Test 2
    vars:
      prompt_num: 2
```

Like this Test 1 will run with prompt 1, and Test 2 with prompt 2.

Same problem - this seems like a blindspot in the API