Repository: promptfoo/promptfoo
Issue: Launch promptfoo including several directories and configuration
Root cause: Algorithmic Error

Hello 👋,

Thank you for this great library—it's been instrumental in LLM testing.

I'm working on a use case requiring promptfoo execution across multiple directories, each containing several configuration files. The goal is to generate a comprehensive report aggregating all directories' results.

Here's an example of the directory structure:
```
root/env/dev/tests/llm1
├── promptfooconfig_1.yaml
└── promptfooconfig_2.yaml
root/env/dev/tests/llm2
├── promptfooconfig_1.yaml
└── promptfooconfig_2.yaml
```
Ideally, I'd like to run a command like `promptfoo eval --config <regex>` to evaluate all configurations matching the specified regex pattern.

Currently, I'm resorting to a loop to execute promptfoo for each file individually. However, this approach only allows me to consolidate the results into a single, unified report.

It would be beneficial if promptfoo could support:
- Recursive discovery of configuration files based on a regex pattern.
- Execution of tests for all discovered configurations.
- Generation of a global report summarizing the outcomes from all the tests.

This feature would streamline the testing process for projects with complex directory structures and numerous test configurations.

Thank you for considering this 🙏 

Hi Florian,

Since https://github.com/promptfoo/promptfoo/pull/285 landed a few days ago, something similar to this is possible.  You can use [glob syntax](https://github.com/isaacs/node-glob#glob-primer) to match multiple files, which are combined into a single eval.

For example:
```
promptfoo eval -c tests/**/*config_2.yaml
```

This produces a single report that combines the vars, prompts, providers, and test cases for each discovered configuration.

Let me know if this works for your use case!

Thanks @typpo 
It is the right option for my setup.
However, I encountered an issue on  `>0.27` versions with CSV files in the tests section.
I think it started to iterate over the CSV file content!

Tests cases like this:
`tests: tests.csv
`

Can you elaborate?  I'm able to run this one for example just fine: https://github.com/promptfoo/promptfoo/blob/main/examples/assistant-cli/promptfooconfig.yaml

Sure!

This is the command line:
`promptfoo eval --config **/tests/promptfooconfig_*.yaml
`

This is the configuration:
```
description: description
providers: openai:gpt-4-1106-preview
prompts: prompt.py
tests: tests.csv
```

Error:
```
node:fs:453
    return binding.readFileUtf8(path, stringToFlags(options.flag));
                   ^

Error: ENOENT: no such file or directory, open '/**/tests/q'
    at Object.readFileSync (node:fs:453:20)
    at readTest (/opt/homebrew/lib/node_modules/promptfoo/dist/src/testCases.js:114:55)
    at loadTestsFromGlob (/opt/homebrew/lib/node_modules/promptfoo/dist/src/testCases.js:136:32)
    at readTests (/opt/homebrew/lib/node_modules/promptfoo/dist/src/testCases.js:160:36)
    at Command.<anonymous> (/opt/homebrew/lib/node_modules/promptfoo/dist/src/main.js:260:61)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '**/tests/q'
}
```




https://github.com/promptfoo/promptfoo/pull/291 should fix this.  Will land shortly