Repository: promptfoo/promptfoo
Issue: Getting test files using custom logic
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
I want to manage test yaml file in other repository. However, in [this official docs](https://www.promptfoo.dev/docs/configuration/parameters/#tests-file), it seems that files in the identical repository are allowed for test files.

For example, suppose my `promptfooconfig.yaml` file in `repository A` is

```yaml
tests:
  - normal_test.yaml
  - special_test.yaml
  - path/to/more_tests/*.yaml
```

Above yaml file assumes that `normal_test.yaml`, `special_test.yaml` should be same directory as `promptfooconfig.yaml`. However, I want to get these yaml test files from other repository, e.g., `repository B`.

**Describe the solution you'd like**
It would be great if custom function can be used in test files similar to [prompt case](https://www.promptfoo.dev/docs/configuration/parameters/#prompt-functions).

**Describe alternatives you've considered**
I have not considered alternatives yet.

**Additional context**
NA


Hi @bohyunshin,

Thanks for bringing this up! You can reference test files in other repositories (or folders) by using relative or absolute paths with the `file://` prefix in your `promptfooconfig.yaml`. Here's an example:

```yaml
tests:
  - file://../repoB/normal_test.yaml
  - file://../repoB/special_test.yaml
  - file:///home/user/repoB/more_tests/*.yaml
```

This approach should allow you to manage your test files across different repositories. I'll update the documentation to recommend the file prefix. Let me know if this works for you, or if you need any further help!

Hi @mldangelo ,
thanks for your quick answer! It seems that we can do similar things in `test files` as we do in `prompt`.
One more question, can I use `functinos` in test files similar to [prompt functions](https://www.promptfoo.dev/docs/configuration/parameters/#prompt-functions)?
For example, when I write `get_test_cases.py` as below,

```python
def my_func(context):
  test_val_1 = other_func_getting_test_cases(context)
  return test_val_1
```

and if `test_val_1` is something like

```yaml
- vars:
    user_query: "hi"
  assert:
    - type: llm-rubric
      value: some rubric statement
```

is it possible to write `promptfooconfig.yaml` as 

```yaml
tests:
  - get_test_cases.py:my_func
```

Hi @bohyunshin, Thanks for your follow-up! While you can't use functions directly in the `tests` section, you can achieve similar functionality by using custom Python functions in the `assert` section.

For example:

```yaml
tests:
  - vars:
      input: 'sample input'
      expected_output: 'sample output'

defaultTest:
  assert:
    - type: python
      value: file://path/to/your_script.py:func_name
```

The `your_script.py` file would contain the logic for validating the output based on the test variables.

For a concrete example of how to set this up, you can check out this example on GitHub: https://github.com/promptfoo/promptfoo/blob/feat/ollama/examples/python-assert-external

Additionally, our documentation provides more details: https://www.promptfoo.dev/docs/configuration/expected-outputs/python/

Let me know if this helps, or if you have any further questions!

Hi, @mldangelo 
thanks for your answer. I will try it out.