Repository: promptfoo/promptfoo
Issue: Separate YAML test files not working starting from v0.104.1
Root cause: Algorithmic Error

**Describe the bug**
Separate YAML test files not working after promptfoo is upgraded to v0.104.1 and later (tested v0.104.2 as well). Is there any changes to the format or is this a bug? I checked the documentation (https://www.promptfoo.dev/docs/configuration/guide/#import-tests-from-separate-files and https://github.com/promptfoo/promptfoo/blob/main/examples/separate-test-configs/tests/tests1.yaml) but there isn't any changes.


**To Reproduce**
Please refers to the code sample [promptfootest.zip](https://github.com/user-attachments/files/18792428/promptfootest.zip)

Below is the `promptfoo debug` output from v0.104.2, the whole YAML test seems is now placed under `tests\var`, is there any changes to the def mapping?
```
Warning: Found 'assert' key in vars. This is likely a mistake in your configuration.

'assert' should be *unindented* so it is under the test itself, not vars. For example:

tests:
  - vars:
      foo: bar
    assert:
      - type: contains
        value: "bar"

To disable this message, set the environment variable PROMPTFOO_NO_TESTCASE_ASSERT_WARNING=1.
========================================================================================================================
Promptfoo Debug Information
========================================================================================================================
{
  "version": "0.104.2",
  "platform": {
    "os": "win32",
    "release": "10.0.19045",
    "arch": "x64",
    "nodeVersion": "v22.14.0"
  },
  "env": {},
  "configInfo": {
    "defaultConfigPath": "C:\\repos\\promptfootest\\promptfooconfig.yaml",
    "configExists": true,
    "configContent": {
      "config": {
        "tags": {},
        "description": "My eval",
        "prompts": [
          "file://C:\\repos\\promptfootest\\prompts\\prompts.yaml"
        ],
        "providers": [
          {
            "id": "azure:chat:gpt-4o",
            "config": {
              "apiHost": "remove.openai.azure.com",
              "temperature": 0.7,
              "top_p": 0.95,
              "max_tokens": 4000,
              "apiKey": "remove"
            }
          }
        ],
        "tests": [
          "file://tests/tests1.yaml",
          "file://tests/tests2.yaml"
        ],
        "scenarios": [],
        "env": {},
        "sharing": true,
        "defaultTest": {
          "vars": {},
          "assert": [],
          "options": {},
          "metadata": {}
        },
        "outputPath": [],
        "extensions": [],
        "metadata": {}
      },
      "testSuite": {
        "description": "My eval",
        "tags": {},
        "prompts": [
          {
            "raw": "[{\"role\":\"system\",\"content\":\"You are a highly knowledgeable assistant specializing in API documentation and OpenAPI v2 (Swagger) schema generation. Your role is to help developers create accurate OpenAPI v2 schemas for any existing API. When assisting:\\n- Ensure the schema adheres strictly to the OpenAPI v2 specification.\\n- Do not include any secrets or keys.\\n- Do not provide more than 10 operations.\\n- Include securityDefinitions if the API requires it, but do not include redundant parameters which are already defined in securityDefinitions.\\n- Define schema for the responses.\\n- Each path must contain summary, description and operationId.\\n- Ensure that the result is valid JSON.\\n- You should return just the JSON swagger content without any code blocks or any other formatting.\\n\"},{\"role\":\"user\",\"content\":\"{{ userprompt }}\"}]",
            "label": "C:\\repos\\promptfootest\\prompts\\prompts.yaml: [{\"role\":\"system\",\"content\":\"You are a highly knowledgeable assistant specializi"
          }
        ],
        "providers": [
          {
            "initializationPromise": {},
            "env": {},
            "deploymentName": "gpt-4o",
            "apiHost": "remove.openai.azure.com",
            "config": {
              "apiHost": "remove.openai.azure.com",
              "temperature": 0.7,
              "top_p": 0.95,
              "max_tokens": 4000,
              "apiKey": "remove",
              "basePath": "C:\\repos\\promptfootest"
            },
            "label": "",
            "authHeaders": {
              "api-key": "remove"
            }
          }
        ],
        "providerPromptMap": {
          "azure:chat:gpt-4o": [
            "C:\\repos\\promptfootest\\prompts\\prompts.yaml: [{\"role\":\"system\",\"content\":\"You are a highly knowledgeable assistant specializi"
          ]
        },
        "tests": [
          {
            "vars": {
              "vars": {
                "userprompt": "A Sharepoint API, that can retrieve list name"
              },
              "assert": [
                {
                  "type": "contains-json"
                },
                {
                  "type": "icontains",
                  "value": "sharepoint"
                }
              ]
            },
            "assert": [],
            "options": {},
            "description": "Row #1"
          },
          {
            "vars": {
              "vars": {
                "userprompt": "A Sharepoint API, that can retrieve list name."
              },
              "options": {
                "transform": "JSON.parse(output)"
              },
              "assert": [
                {
                  "type": "is-json"
                },
                {
                  "type": "icontains",
                  "value": "sharepoint",
                  "transform": "output.info.title"
                },
                {
                  "type": "icontains",
                  "value": "sharepoint.com",
                  "transform": "output.host"
                }
              ]
            },
            "assert": [],
            "options": {},
            "description": "Row #1"
          }
        ],
        "scenarios": [],
        "defaultTest": {
          "metadata": {},
          "options": {},
          "vars": {},
          "assert": []
        },
        "nunjucksFilters": {},
        "extensions": []
      },
      "basePath": "C:\\repos\\promptfootest"
    }
  }
}
========================================================================================================================
Please include this output when reporting issues on GitHub: https://github.com/promptfoo/promptfoo/issues
```


**Expected behavior**
Separate YAML test files are still supported.
Below is the `promptfoo debug` output from v0.104.0
```
========================================================================================================================
⚠️ The current version of promptfoo 0.104.0 is lower than the latest available version 0.104.2.

Please run npx promptfoo@latest or npm install -g promptfoo@latest to update.
========================================================================================================================

========================================================================================================================
Promptfoo Debug Information
========================================================================================================================
{
  "version": "0.104.0",
  "platform": {
    "os": "win32",
    "release": "10.0.19045",
    "arch": "x64",
    "nodeVersion": "v22.14.0"
  },
  "env": {},
  "configInfo": {
    "defaultConfigPath": "C:\\repos\\promptfootest\\promptfooconfig.yaml",
    "configExists": true,
    "configContent": {
      "config": {
        "tags": {},
        "description": "My eval",
        "prompts": [
          "file://C:\\repos\\promptfootest\\prompts\\prompts.yaml"
        ],
        "providers": [
          {
            "id": "azure:chat:gpt-4o",
            "config": {
              "apiHost": "remove.openai.azure.com",
              "temperature": 0.7,
              "top_p": 0.95,
              "max_tokens": 4000,
              "apiKey": "remove"
            }
          }
        ],
        "tests": [
          "file://tests/tests1.yaml",
          "file://tests/tests2.yaml"
        ],
        "scenarios": [],
        "env": {},
        "sharing": true,
        "defaultTest": {
          "vars": {},
          "assert": [],
          "options": {},
          "metadata": {}
        },
        "outputPath": [],
        "extensions": [],
        "metadata": {}
      },
      "testSuite": {
        "description": "My eval",
        "tags": {},
        "prompts": [
          {
            "raw": "[{\"role\":\"system\",\"content\":\"You are a highly knowledgeable assistant specializing in API documentation and OpenAPI v2 (Swagger) schema generation. Your role is to help developers create accurate OpenAPI v2 schemas for any existing API. When assisting:\\n- Ensure the schema adheres strictly to the OpenAPI v2 specification.\\n- Do not include any secrets or keys.\\n- Do not provide more than 10 operations.\\n- Include securityDefinitions if the API requires it, but do not include redundant parameters which are already defined in securityDefinitions.\\n- Define schema for the responses.\\n- Each path must contain summary, description and operationId.\\n- Ensure that the result is valid JSON.\\n- You should return just the JSON swagger content without any code blocks or any other formatting.\\n\"},{\"role\":\"user\",\"content\":\"{{ userprompt }}\"}]",
            "label": "C:\\repos\\promptfootest\\prompts\\prompts.yaml: [{\"role\":\"system\",\"content\":\"You are a highly knowledgeable assistant specializi"
          }
        ],
        "providers": [
          {
            "initializationPromise": {},
            "env": {},
            "deploymentName": "gpt-4o",
            "apiHost": "remove.openai.azure.com",
            "config": {
              "apiHost": "remove.openai.azure.com",
              "temperature": 0.7,
              "top_p": 0.95,
              "max_tokens": 4000,
              "apiKey": "remove",
              "basePath": "C:\\repos\\promptfootest"
            },
            "label": "",
            "authHeaders": {
              "api-key": "remove"
            }
          }
        ],
        "providerPromptMap": {
          "azure:chat:gpt-4o": [
            "C:\\repos\\promptfootest\\prompts\\prompts.yaml: [{\"role\":\"system\",\"content\":\"You are a highly knowledgeable assistant specializi"
          ]
        },
        "tests": [
          {
            "vars": {
              "userprompt": "A Sharepoint API, that can retrieve list name"
            },
            "assert": [
              {
                "type": "contains-json"
              },
              {
                "type": "icontains",
                "value": "sharepoint"
              }
            ]
          },
          {
            "vars": {
              "userprompt": "A Sharepoint API, that can retrieve list name."
            },
            "options": {
              "transform": "JSON.parse(output)"
            },
            "assert": [
              {
                "type": "is-json"
              },
              {
                "type": "icontains",
                "value": "sharepoint",
                "transform": "output.info.title"
              },
              {
                "type": "icontains",
                "value": "sharepoint.com",
                "transform": "output.host"
              }
            ]
          }
        ],
        "scenarios": [],
        "defaultTest": {
          "metadata": {},
          "options": {},
          "vars": {},
          "assert": []
        },
        "nunjucksFilters": {},
        "extensions": []
      },
      "basePath": "C:\\repos\\promptfootest"
    }
  }
}
========================================================================================================================
Please include this output when reporting issues on GitHub: https://github.com/promptfoo/promptfoo/issues
```

**Screenshots**
Below are the sample `promptfoo debug` screenshot, showing the tests mapping seems to go wrong
![Image](https://github.com/user-attachments/assets/846a27df-f1f7-4757-823e-de7d7f6394ea)


**Additional context**
Add any other context about the problem here.


Hey @siemenstan,

Thanks for reporting this! You’re right. This was an unintended regression introduced in [#2993](https://github.com/promptfoo/promptfoo/pull/2993) which added support for dynamically loading test cases from python scripts. We just issued a fix in [#3082](https://github.com/promptfoo/promptfoo/pull/3082) and released the fix in [0.104.3](https://github.com/promptfoo/promptfoo/releases/tag/0.104.3). Really appreciate your patience and for catching this so quickly. Let us know if you run into any further issues!

Great @mldangelo , thank you for swift response and fix, will wait for the fix in new release

It's available on npm now! https://www.npmjs.com/package/promptfoo/v/0.104.3 Closing the issue. Please feel free to re-open or file a new issue if you run into any other problems. Thanks so much and sorry for breaking Promptfoo.