Repository: promptfoo/promptfoo
Issue: What is the correct way to import tests from external files?
Root cause: Validation Gap

[According to the docs, this is supported:
](https://www.promptfoo.dev/docs/configuration/guide#import-tests-from-separate-files:~:text=%23%20Load%20%26%20runs%20all%20test%20cases%20matching%20these%20filepaths%0Atests%3A%0A%20%20%23%20You%20can%20supply%20an%20exact%20filepath%0A%20%20%2D%20tests/tests2.yaml)
```
# Load & runs all test cases matching these filepaths
tests:
  # You can supply an exact filepath
  - tests/tests2.yaml
```

I've attempted that:

```
tests:
  - ../SomeDir/mytest.yaml
```

And `mytest.yaml` in the format:
```
vars:
  foo: bar
```

I get this error message:
```
/Users/natanyellin/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/testCases.js:173
                for (const testCase of testCases) {
                                       ^

TypeError: testCases is not iterable
    at loadTestsFromGlob (/Users/natanyellin/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/testCases.js:173:40)
    at readTests (/Users/natanyellin/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/testCases.js:194:36)
    at resolveConfigs (/Users/natanyellin/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/main.js:129:57)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async runEvaluation (/Users/natanyellin/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/main.js:521:48)
    at async Command.<anonymous> (/Users/natanyellin/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/main.js:680:9)

Node.js v20.14.0
```

Hey @aantn, it expects the test file to be a _list of tests_, not just a single test. Try this:

```yaml
- vars:
    foo: bar
```

Thanks, that works. On a related topic, it would be great to get a warning
if the file doesn't exist. Right now it fails silently.

On Fri, Jun 21, 2024 at 11:32 PM Ian Webster ***@***.***>
wrote:

> Hey @aantn <https://github.com/aantn>, it expects the test file to be a *list
> of tests*, not just a single test. Try this:
>
> - vars:
>     foo: bar
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/982#issuecomment-2183415050>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADYUBYAOFU7GEUWWJ7D57TZISEUTAVCNFSM6AAAAABJWSU5LOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBTGQYTKMBVGA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Ah thanks for flagging.  Warning added in e34933ad76e290a04fa75fc082833845e7d75161

Wonderful, thank you as always!

On Sat, Jun 22, 2024 at 9:28 AM Ian Webster ***@***.***>
wrote:

> Ah thanks for flagging. Warning added in e34933a
> <https://github.com/promptfoo/promptfoo/commit/e34933ad76e290a04fa75fc082833845e7d75161>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/982#issuecomment-2183853274>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADYUB7GJZ5RMZYUGXGJLO3ZIUKQZAVCNFSM6AAAAABJWSU5LOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBTHA2TGMRXGQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>