Repository: promptfoo/promptfoo
Issue: PROMPTFOO_PASS_RATE_THRESHOLD but for individual repeated tests
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Agent evals are non-deterministic. I want to make sure that each of my tests pass - but I don't expect all of them to pass every time.
I can repeat tests using the cli flag `--repeat 10` and I want to assert that each test passes 8/10 of those repeats.
I want this 80% threshold to determine whether the CLI returns a healthy exit code.

**Describe the solution you'd like**
I would like an environmen variable similar to `PROMPTFOO_PASS_RATE_THRESHOLD`, except instead of asserting the threshold across ALL tests, I would like the threshold to apply across all repeats of individual tests.
E.g. `PROMPTFOO_TEST_REPEAT_PASS_RATE_THRESHOLD`

**Describe alternatives you've considered**
I have considered using `PROMPTFOO_PASS_RATE_THRESHOLD`. However, I want to assert that EVERY individual test is passing at least 80% of the time. I don't want to allow ANY tests to be failing 100% of the time. `PROMPTFOO_PASS_RATE_THRESHOLD` cannot assert that for me.

**Additional context**
Happy to contribute towards this solution if it is inline with the product & isn't already possible through another configuration.


Hi Denis, great suggestion here. We would gladly accept a contribution
along these lines.

On Tue, Oct 7, 2025, 9:22 AM Denis ***@***.***> wrote:

> *denis-snyk* created an issue (promptfoo/promptfoo#5847)
> <https://github.com/promptfoo/promptfoo/issues/5847>
>
> *Is your feature request related to a problem? Please describe.*
> Agent evals are non-deterministic. I want to make sure that each of my
> tests pass - but I don't expect all of them to pass every time.
> I can repeat tests using the cli flag --repeat 10 and I want to assert
> that each test passes 8/10 of those repeats.
> I want this 80% threshold to determine whether the CLI returns a healthy
> exit code.
>
> *Describe the solution you'd like*
> I would like an environmen variable similar to
> PROMPTFOO_PASS_RATE_THRESHOLD, except instead of asserting the threshold
> across ALL tests, I would like the threshold to apply across all repeats of
> individual tests.
> E.g. PROMPTFOO_TEST_REPEAT_PASS_RATE_THRESHOLD
>
> *Describe alternatives you've considered*
> I have considered using PROMPTFOO_PASS_RATE_THRESHOLD. However, I want to
> assert that EVERY individual test is passing at least 80% of the time. I
> don't want to allow ANY tests to be failing 100% of the time.
> PROMPTFOO_PASS_RATE_THRESHOLD cannot assert that for me.
>
> *Additional context*
> Happy to contribute towards this solution if it is inline with the product
> & isn't already possible through another configuration.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/5847>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJUDMK3M5AZWWMVCVBD3WO5C7AVCNFSM6AAAAACIQOBLBGVHI2DSMVQWIX3LMV43ASLTON2WKOZTGQ4TCNBYGMZDEMA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>