Repository: promptfoo/promptfoo
Issue: Support evaluating multiple promptfoo files at once
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Currently promptfoo is more like a one-off research tool, instead of an ongoing continuous testing framework, due to the limitation of having only one evaluation file run when you run `promptfoo eval`.

**Describe the solution you'd like**
A solution that is focused on have many evaluation files in a project, something more similar to a regular testing framework like Vitest or Pytest.
I would suggest running any file that ends with `.promptfoo.ts`/`.promptfoo.js`/`.promptfoo.yaml`
For example, given the following project:
```
project-root/
├── README.md
├── package.json
├── tsconfig.json
├── src/
│   ├── translate-to-english.promptfoo.ts
│   ├── translate-to-spanish.promptfoo.ts
```
When I run `promptfoo eval-all`, I would expect it to run both `translate-to-english.promptfoo.ts` and `translate-to-spanish.promptfoo.ts`

Also, later on, a VSCode/JetBrains IDEs can have first-class citizen support to run multiple promptfoo files.

WDYT? Any plans to support it?

Currently I implemented a workaround that works for me, but isn't ideal as it lacks multiple features due to not being support built-in promptfoo.

```ts
const files = await findFiles('.', /\.promptfoo\.ts$/);
const formattedFiles = files.map((file) => file.replace('src', '.'));

for (const file of formattedFiles) {
  let loadedFile;

  try {
    loadedFile = await import(file);
  } catch {
    throw new Error(`Failed to load file '${file}'`);
  }

  if (
    !isPlainObject(loadedFile) ||
    !hasKey(loadedFile, 'default') ||
    typeof loadedFile['default'] !== 'function' ||
    loadedFile['default'].constructor.name !== 'AsyncFunction'
  ) {
    throw new TypeError(
      `File '${file}' does not export a default async function`,
    );
  }

  try {
    logger.info(`Evaluating file '${file}'`);
    await loadedFile['default']();
    logger.info(`File '${file}' evaluated successfully`);
  } catch (error) {
    throw new Error(`Failed to evaluate file '${file}'`, { cause: error });
  }
}
```

Hi @MatanYadaev,

Thank you for trying out promptfoo and taking the time to report this issue. I believe the functionality you're looking for is already supported. You can run multiple `promptfoo` files with commands like:

```bash
NODE_OPTIONS='--import @swc-node/register/esm-register' npx promptfoo eval -c examples/ts-config/promptfooconfig.ts examples/ts-config/promptfooconfig2.ts
```

Or by using a pattern:

```bash
NODE_OPTIONS='--import @swc-node/register/esm-register' npx promptfoo eval -c examples/ts-config/promptfooconfig*.ts
```

I realize this might not be immediately clear, so I'll work on improving the documentation. Thanks again for your feedback. Let me know if this resolves your issue or if there's anything else we can assist with.

Closing. Please feel free to leave another comment if this doesn't resolve your issue! 

@mldangelo Thanks for the help. I'll test it soon. Anyway, I think the GitHub Action doesn't support receiving multiple promptfoo configs.

@mldangelo It doesn't work for me, first, I wasn't able to make it work with TS, and second, exporting promptfoo config loses some of the capabilities the Node package has, like being able to pass functions to the `prompts`, `providers`, and `assert` properties ([examples here](https://www.promptfoo.dev/docs/usage/node-package/#example)).

I was looking into something similar: run any files in the tests folder and do not combine them. This approach is similar to how Jest or other testing frameworks execute each test file in isolation.

Here’s the script I ended up using in the package.json file:
```json
"test:promptfoo": "for config_file in tests/promptfoo/*.yaml; do npx promptfoo@latest eval -c \"$config_file\"; done",
```