Repository: promptfoo/promptfoo
Issue: Allow disabling of defaultTests for particular test instances
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Maybe I missed this in the docs but there doesn't seem to be a way to tell a test NOT to use the default test assertions. I have such a case where the tests expects an empty result from my custom provider but my defaultTest assertions fail if the result is empty. For that single test I'd like to state exactly which assertions to run and ignore defaultTest.

Is that possible?

**Describe the solution you'd like**
Maybe an option on a test case that disables defaultTests just for this test

**Describe alternatives you've considered**
I could create a separate yaml but that is really just a workaround



<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1360/allow-disabling-of-defaulttests-for-particular-test-instances">ENG-1360</a></p>

Hey @Wyverex42, there is currently no built-in way to disable `defaultTest` assertions for a specific test case. The default assertions always gets [merged in](https://github.com/promptfoo/promptfoo/blob/main/src/evaluator.ts#L1046-L1049).

That said, you can work around this today using a test case generation script. Instead of using `defaultTest`, you control the assertion composition per-test in the generator:

```yaml
# promptfooconfig.yaml
prompts:
  - "Process: {{input}}"

providers:
  - openai:gpt-5.2

# No defaultTest - generator handles assertion composition
tests: file://generate_tests.py:generate_tests  # or file://generate_tests.js
```

```python
# generate_tests.py
def generate_tests():
    # Define your "default" assertions once
    default_asserts = [
        {"type": "contains", "value": "expected output"},
    ]

    test_cases = [
        # Normal tests - include defaults
        {"vars": {"input": "normal query"}},
        {"vars": {"input": "another query"}},

        # Special case - skip defaults
        {
            "description": "Empty result test",
            "vars": {"input": "special trigger"},
            "assert": [{"type": "equals", "value": ""}],
            "skip_defaults": True,
        },
    ]

    tests = []
    for case in test_cases:
        skip = case.pop("skip_defaults", False)
        case_asserts = case.get("assert", [])

        if skip:
            case["assert"] = case_asserts
        else:
            case["assert"] = default_asserts + case_asserts

        tests.append(case)

    return tests
```

```javascript
// generate_tests.js
module.exports = function () {
  const defaultAsserts = [
    { type: 'contains', value: 'expected output' },
  ];

  const testCases = [
    // Normal tests - include defaults
    { vars: { input: 'normal query' } },
    { vars: { input: 'another query' } },

    // Special case - skip defaults
    {
      description: 'Empty result test',
      vars: { input: 'special trigger' },
      assert: [{ type: 'equals', value: '' }],
      skipDefaults: true,
    },
  ];

  return testCases.map((test) => {
    const { skipDefaults, ...rest } = test;
    const caseAsserts = test.assert || [];

    return {
      ...rest,
      assert: skipDefaults ? caseAsserts : [...defaultAsserts, ...caseAsserts],
    };
  });
};
```

This gives you full control over which tests get which assertions. The `skip_defaults` flag is just a convention in your data - you could also use metadata, separate lists, or any logic that fits your use case.

Let me know if this works for you. Would be happy to consider design proposals for native support. 


Hi @mldangelo, thanks for the detailed reply!

I wasn't aware of the case generation script, so indeed this could work, and I can see this as a very useful feature if you have more complex requirements for test case generation. However, for this simple requirement it adds too much complexity IMHO. You have to write a separate file, you have to write code, when reading the file you have additional cognitive load, and the test cases are now separate from the overall test definition. 

Wouldn't it be nicer if you could state something like this? The option makes it clear that the defaultTest assertions are not applied, so you only have the assertions you're writing yourself. And it keeps the configuration together.
```
tests:
  - vars:
      a: 42
      b: 123
    config:
      skipDefaultTest: true
    assert:
      ...
```

I'm not familiar with your codebase, but from the code snippet you linked above it looks like it would be a fairly simple thing, especially since you have access to a `config` object in that function (although I don't know what kind of config that refers to).