Repository: promptfoo/promptfoo
Issue: Feature Request: Define prompt:assertion pairs rather than only being able to define assertions in test objects
Root cause: Unimplemented Feature Gap

Especially for text classification use cases, it would be very useful to be able to define test assertions in a prompt-aware manner.

Often, assertions on prompt completions necessarily have tight coupling with the input prompt. This especially applies to assertion types like `contains` and `regex` in the context of binary classification prompts, as they are essentially binary parsers for the prompt completion, whose format is heavily influenced by the specific input prompt.

One approach would be to define the assertion value alongside each prompt. Thus `prompts.txt` could become `prompts.json` and look something like:
```json
[{
    "prompt": "Is the following a violation of U.S. Patent Law?\n\n{{legal_case_text}}\n\nPlease ONLY give a Yes or No answer.",
    "assertion_value": "Yes"
},
{
    "prompt": "Is the following likely a violation of U.S. Patent Law?\n\n{{legal_case_text}}\n\nPlease give a brief explanation along with your answer.",
    "assertion_value": "It is likely"
}]
```
Note that different test cases would then use either `contains` or `not-contains` in order to differentiate between positive and negative test cases.

I realize binary classification is only one particular use case for LLM prompts, but I do think it's an important one that is currently a bit difficult to work with in promptfoo.

I'm also quite new to promptfoo so if there are other ways to achieve this, please educate me :)

Hi @zeldrinn,

Would the output depend on the value of `legal_case_text`?  If so, can you structure it like this:

```yaml
# ...
tests:
  - vars:
      legal_case_text: 'first case text....'
    assert:
      - type: equals
        value: "Yes"
  - vars:
        legal_case_text: 'second case text.....'
    assert:
      - type: equals
        value: "It is likely"
```

thanks for the quick reply @typpo! that's the approach i initially took, but it doesn't fully address the fact that the output _also_ depends on the prompt, not just on the value of the vars (in this case `legal_case_text`). in other words, a common pattern when trying to optimize prompts is to run all the same test cases (input vars) on all of the different prompt variations (i.e. define your test cases via the cartesian product of `[[var1a, var1b, ...], [var2a, var2b, ...], ...]` and `[prompt_variant_1, prompt_variant_2, ...]`).

concretely, in the example we're discussing, "It is likely" is specifically a pattern only found in responses from prompt 2, so it wouldn't make sense to use it for tests run against prompt 1. you could alternatively create separate test config files, each of which references a different set of prompts, but that mostly defeats the purpose of being able to compare the performance of different prompt variations. ideally, the full set of tests (the cartesian product) in this example would be:
```
[{
    "prompt": "Is the following a violation of U.S. Patent Law?\n\n{{legal_case_text}}\n\nPlease ONLY give a Yes or No answer.",
    "legal_case_text": "first case text",
    "assertion_type": "contains",
    "assertion_value": "Yes"
},
{
    "prompt": "Is the following a violation of U.S. Patent Law?\n\n{{legal_case_text}}\n\nPlease ONLY give a Yes or No answer.",
    "legal_case_text": "second case text",
    "assertion_type": "contains",
    "assertion_value": "Yes"
},
{
    "prompt": "Is the following likely a violation of U.S. Patent Law?\n\n{{legal_case_text}}\n\nPlease give a brief explanation along with your answer.",
    "legal_case_text": "first case text",
    "assertion_type": "contains",
    "assertion_value": "It is likely"
},
{
    "prompt": "Is the following likely a violation of U.S. Patent Law?\n\n{{legal_case_text}}\n\nPlease give a brief explanation along with your answer.",
    "legal_case_text": "second case text",
    "assertion_type": "contains",
    "assertion_value": "It is likely"
}]
```

Thanks for explaining - let me give this some thought and get back to you with a suggestion!

I could implement something like this, essentially what you asked for, an `assert` list associated with each prompt that will get merged into each individual test cases.

```yaml
prompts:
  - prompt1.txt:
      assert:
        - type: contains
          value: Yes
  - prompt2.txt:
      assert:
        - type: contains
          value: It is likely
providers: ['anthropic:completion']
tests:
  - vars:
      legal_case_text: 'first case text.....'
  - vars:
      legal_case_text: 'second case text.....'
    assert:
      - type: contains
        value: Other
```

Based on my understanding I think this will work for you, but just to be explicit, this would test the following:
- anthropic, prompt1, first case text, contains "Yes"
- anthropic, prompt1, second case text, contains "Yes" and "Other"
- anthropic, prompt2, first case text, contains "It is likely"
- anthropic, prompt2, second case text, contains "It is likely" and "Other"

this would _almost_ work. only issue is that this wouldn't permit toggling the assertion from "positive" to "negative" (in the case of binary classification tasks), because the `type` is tied to the prompt. if we just put the `value` in the `prompt > assertion` and put the `type` in the `tests` section then i believe it would be resolved.

better yet, perhaps both `type` and `value` are optional in the `prompts` section, and it uses them if defined there. otherwise, for each one, it will fall back to the `type` and `value` defined in the `tests` section. perhaps this is what you were implying? e.g.:
```
prompts:
  - prompt1.txt:
      assert:
        - type: icontains
          value: Yes
  - prompt2.txt:
      assert:
        - value: It is likely
providers: ['anthropic:completion']
tests:
  - vars:
      legal_case_text: 'first case text.....'
     assert:
       - type: contains
         value: it is likely
  - vars:
      legal_case_text: 'second case text.....'
     assert:
       - type: contains
         value: it is likely
  - vars:
      legal_case_text: 'third case text.....'
    assert:
      - type: contains
        value: Other
```
this would effectively let users define both the `type` and `value` alongside each prompt as an override of the default that is defined in the test cases. we're probably still missing something fundamental in how binary classification test cases should be modeled, but this would probably be good enough for now!

Hey guys!

I have a very similar problem. Basically I want to assert that my LLM is not hallucinating. In order to do that, I have a prompt that I send to the LLM and the answer from the LLM should only contain links that are in the prompt, which means that I need to have access to both the prompt sent to the LLM and the output generated by the LLM in my assertion. Is it possible?

> I could implement something like this, essentially what you asked for, an `assert` list associated with each prompt that will get merged into each individual test cases.
> 
> 
> 
> ```yaml
> 
> prompts:
> 
>   - prompt1.txt:
> 
>       assert:
> 
>         - type: contains
> 
>           value: Yes
> 
>   - prompt2.txt:
> 
>       assert:
> 
>         - type: contains
> 
>           value: It is likely
> 
> providers: ['anthropic:completion']
> 
> tests:
> 
>   - vars:
> 
>       legal_case_text: 'first case text.....'
> 
>   - vars:
> 
>       legal_case_text: 'second case text.....'
> 
>     assert:
> 
>       - type: contains
> 
>         value: Other
> 
> ```
> 
> 
> 
> Based on my understanding I think this will work for you, but just to be explicit, this would test the following:
> 
> - anthropic, prompt1, first case text, contains "Yes"
> 
> - anthropic, prompt1, second case text, contains "Yes" and "Other"
> 
> - anthropic, prompt2, first case text, contains "It is likely"
> 
> - anthropic, prompt2, second case text, contains "It is likely" and "Other"

Is this feature online?

Hi all,

Circling back on this long-standing feature request. Shortly after this issue was opened, we added `context.prompt` to the assertion context ([35d4262](https://github.com/promptfoo/promptfoo/commit/35d4262)), which enables prompt-aware assertions via JavaScript (or Python).

For the binary classification use case described in the original request, you can now do something like:

```yaml
prompts:
  - file://prompt1.txt  # "Please ONLY give a Yes or No answer"
  - file://prompt2.txt  # "Please give a brief explanation..."

providers:
  - openai:gpt-4o

tests:
  - vars:
      legal_case_text: 'first case text (positive)'
    assert:
      - type: javascript
        value: |
          // Determine expected pattern based on which prompt was used
          if (context.prompt.includes('ONLY give a Yes or No')) {
            return output.includes('Yes');
          } else {
            return output.includes('It is likely');
          }
  - vars:
      legal_case_text: 'second case text (negative)'
    assert:
      - type: javascript
        value: |
          if (context.prompt.includes('ONLY give a Yes or No')) {
            return output.includes('No');
          } else {
            return output.includes('It is unlikely');
          }
```

Or for cleaner organization, you could use variables and an external JS function:

```yaml
tests:
  - vars:
      legal_case_text: 'first case text'
      expected_short: 'Yes'
      expected_verbose: 'It is likely'
    assert:
      - type: javascript
        value: file://assert-classification.js
```

```js
// assert-classification.js
module.exports = (output, context) => {
  const isShortFormat = context.prompt.includes('ONLY give a Yes or No');
  const expected = isShortFormat
    ? context.vars.expected_short
    : context.vars.expected_verbose;
  return output.includes(expected);
};
```

For @thiagosalvatore's hallucination detection use case - `context.prompt` gives you full access to the rendered prompt, so you can extract any links/sources from it and validate them against the output.

We also added [Scenarios](https://www.promptfoo.dev/docs/configuration/scenarios) which help group test data with related assertions, useful for matrix-style testing.

While we never implemented the exact syntax proposed (assertions defined directly on prompts in the config), `context.prompt` + JavaScript assertions should handle most of these use cases. The original proposal would add complexity to the config schema for a feature that's achievable with the current approach.

I'm going to close this as the core need (prompt-aware assertions) is addressed. If there's still a strong desire for the declarative syntax, feel free to reopen with updated use cases that aren't well-served by the current approach.