Repository: promptfoo/promptfoo
Issue: PASS/FAIL rate on assertion level
Root cause: Unimplemented Feature Gap

Not sure if it is possible with the current functionality but I can't seem to find a a good way to solve the following problem.

## Problem

I have a prompt, which is designed to specify why sentences in a paragraph should be highlighted. The prompt returns an array of sentences. The sentences returned are then used to highlight in the paragraph.

Example where the prompt is supposed to return sentence 1 and 3 given some conditions defined in the prompt.

```
paragraph = "This is sentence 1. This is sentence 2. Finally, this is sentence 3"

output = LLM(paragraph)

output # ["this is sentence 1", "Finally, this is sentence 3"]
```

This is how a test would look like:

```yaml
- vars:
    text: "This is sentence 1. This is sentence 2. Finally, this is sentence 3"
  assert:
  - type: contains
    value: "This is sentence 1"
  - type: contains
    value: "Finally, this is sentence 3"
```

The problem is when I run the eval on this test case, it has to pass _all_ the asserts to count as a PASS. However, what if the prompt got half the asserts correct, like identifying "This is sentence 1" but not "Finally, this is sentence 3"? It makes it impossible to see performance on an 'assert' level rather than a 'test' level.

I don't want to define this as two separate test cases with the same vars, as it would have to run the exact same query twice. 

## Solution proposals

Not sure what the best solution is but here is a few suggestions.

### Caching flag

You could add a flag to specify that the cache should be used during execution, so if the same vars has already been run it should use that result. I could then make them separate test cases and just have one assert in each.
Probably not a good idea, since many of them could be executed in parallel, so it wont reach the cache before another one is executed.

### Add a flag or config to run in at 'assert' level
If flag is set, the PASS/FAIL rate would be calculated on assert level rather than test level.
This would require that a column was added to the UI with the 'assert' being tested for and then have duplicate rows in the 'variable' column. I think this would be pretty useful, as it is currently impossible to see from the UI _what_ the prompt actually PASSED. it could be 1 assert or it could be 10. It only shows up what assert it failed on and not what it passed.

## Summary of issue

The main issue is that true performance can be hidden. Imagine I have 500 asserts on 10 paragraphs. Each paragraph have 50 asserts, each one being a sentence which should be returned given the paragraph. This would result in a total of 10 test cases. Let's say I get 480 of the sentences right and 20 wrong. If those 20 happens to be spread out over the 10 paragraphs I get a pass/fail rate of 0/10 instead of 480/500, making it very hard to get the "real" performance of the system.

Hi @KasperDjernaes,

I believe this can be partially addressed by using the `weight` attribute on a test case.  This was not well-documented before, but I've just added a section on this to the docs:

https://promptfoo.dev/docs/configuration/expected-outputs/#weighted-assertions

This will compute a passing %, and you can focus on the percentage instead of the test count.

This change prevents the short-circuiting of test cases so that the % is accurate https://github.com/promptfoo/promptfoo/commit/785442413ad1598eb92230f32fbd5564d1f7f399

I'll keep this issue open until we've added assert counts in addition to test counts.

#100 has added an assertion count in the table header:

![assertion count](https://user-images.githubusercontent.com/310310/258942130-e76d458e-8ed1-47ba-8f64-0f1fe8b3449f.png)