Repository: promptfoo/promptfoo
Issue: Support for include var value from external files
Root cause: Unimplemented Feature Gap

Consider the following config:
```YAML
prompts: [prompt.txt]
providers: 
  - id: openai:gpt-3.5-turbo-0613
    config:
      apiKey: 
  - id: openai:gpt-4-0613
    config:
      apiKey: 
tests:
  - description: Requirement Check
    threshold: 0.6
    vars: 
      prompt: |
        You are an AI software requirement analyst. Please make a judgement if the following requirements needs persistent storage. YES or NO, no explanations.
        Requirements:
        I want to hold a party tomorrow night. Please send invitation emails to my friends. My friends emails: Ella: ella@gmail.com, jack: jack@gmail.com. In the email, give them a link that shows a web page, ask them what kind of alchol they prefer, whisky, wine or beer. I also need a web page to see their choices.
    assert:
      - type: equals
        value: "YES"
        weight: 2
      - type: starts-with
        value: "YES"
        weight: 1  
  - description: Code Completion
    threshold: 0.8
    vars:
      prompt: |
        You are an AI software developer. You will be provided with the TypeScript code that the user is working on, delimited by triple hashtags. The user is currently stopped at "<cursor_is_here>". Please complete this line of code. Code only, no explanations.

        ###
        import Config from "#config";
        import { Configuration, OpenAIApi } from "openai";

        export default async function (history:[], instruction: string) {
            const configuration = new Configuration({
                apiKey: Config.OPENAI_API_KEY,
            });
            const openai = new OpenAIApi(configuration);

            const messages = await build<cursor_is_here>

        }

        /*
        * Build messages for OpenAI based on instructions.
        */
        async function buildMessages (history: [], instruction: string) {
            switch (instruction) {
                case "chat":
                    history.unshift({
                        "role": "system",
                        "content": "You are a helpful assistant."
                    });
                    break;
                case "summarize":
                    history.unshift({
                        "role": "system",
                        "content": "Summarize user's message within 20 words."
                    });
                    break;
                case "complete":
                    history.unshift({
                        "role": "system",
                        "content": "Complete the following text within 60 words."
                    });
                    break
                case "code":
                    history.unshift({
                        "role": "system",
                        "content": "Complete the following code."
                    });
                    break
                default:
                //do nothing
            }
            return history;
        }
        ###
    assert:
      - type: equals
        value: "Messages(history, instruction);"
        weight: 1
      - type: contains-any
        value: 
          - "buildMessages(history, instruction);"
          - "Messages(history, instruction);"
        weight: 4    
```
I wish to include the value of each var "prompt" from external files, the expected config looks like this:
```YAML
prompts: [prompt.txt]
providers: 
  - id: openai:gpt-3.5-turbo-0613
    config:
      apiKey: 
  - id: openai:gpt-4-0613
    config:
      apiKey: 
tests:
  - description: Requirement Check
    threshold: 0.6
    vars: 
      prompt: prompts/requirement_check.txt
    assert:
      - type: equals
        value: "YES"
        weight: 2
      - type: starts-with
        value: "YES"
        weight: 1  
  - description: Code Completion
    threshold: 0.8
    vars:
      prompt: prompts/code_completion.txt
    assert:
      - type: equals
        value: "Messages(history, instruction);"
        weight: 1
      - type: contains-any
        value: 
          - "buildMessages(history, instruction);"
          - "Messages(history, instruction);"
        weight: 4
```    
The prompt.txt used in above cases contains only one line "{{prompt}}".

I have tried to mimic the usage of https://github.com/promptfoo/promptfoo/blob/main/examples/separate-test-configs/promptfooconfig.yaml but I can't get it work. Not sure my case is supported. 



Also, is there a document that I can see all the properties of all the entities? It confuses me a lot for example the "tests" entity. The value of "tests" can be a map or an array. And what kind of properties that I can use under "tests".

Hi @zhlmmc,

You're right, `vars` values do not currently support loading from file in exactly the way you specified.  Instead, `vars` itself can be a filepath that contains a yaml map of variable names (in this case, just `prompt: ...`).  This behavior can be tweaked fairly easily, though.

There is a new [Reference](https://promptfoo.dev/docs/configuration/reference) page on the documentation that outlines the configuration data structure.  Let me know if anything is unclear!

As of version 0.23.0 it is now possible to load a `vars` value directly from file using the `file://` prefix.  Here's an example:

```yaml
  - vars:
      topic: file://vars/var_topic.yaml
      content_type: file://vars/var_content_type.yaml
```

> As of version 0.23.0 it is now possible to load a `vars` value directly from file using the `file://` prefix. Here's an example:
> 
> ```yaml
>   - vars:
>       topic: file://vars/var_topic.yaml
>       content_type: file://vars/var_content_type.yaml
> ```

I tested 0.23.1 and find the following issue: https://github.com/promptfoo/promptfoo/commit/5300c58061292e13f4fa903994d7f24797ffbc00#r127552650

Thanks for the catch @zhlmmc, this should be resolved by 92beb749526b4ea44cec7f9c2bcec0faa041bc73

Edit: published in 0.24.1