Repository: promptfoo/promptfoo
Issue: Report to compare current execution with one of the previous runs
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
We develop a solution, that has a few quite different usage scenarios. So there is no easy way to combine all on them into sort of one big regression suite within one promptfooconfig.yaml.
So we do have a set of test suites spread over several promptfoo configs and each time we need to perform "regression" it becomes a hell-ish manual routine to enter each run per config, browse through results.
And on top of that there is a need to compare with previous run, which is even more complicate than it should.

**Describe the solution you'd like**
1. Show the Pass/Fail rate and Score average in the "Eval-YYYY-MM-DD-THH-MM" drop-down list
2. Additional view to compare selected Evaluations (i.e. test runs). It is more common task to compare not two different providers, but Latest and one of the Previous evals to see the progress. Especially on the assertion level. Checks can be matched on the input data + assertions, since same inputs can be executed in different scenarios, although not very likely
3. Ability to combine several runs with different configs and tests into one.
4. Extremely useful would be the pass/fail+score trends over time, graph of the accuracy score for ground-truth checks and "heat-maps" with assertion results colour-coded for different runs.

**Describe alternatives you've considered**
- Alternative is to export the outcomes into a stadalone shared folder or into a DB, map Grafana to it and try implement the analytics on your own.
- WanDB has an ability to control what data gets, and which doesn't into the test report. While it is not as handy in terms of creating tests, it's visualisation capabilities can come in more handy.

**Additional context**
Promptfoo seems to be a great tool for QA to use on the ML/GenAI/LLM-projects to assert the solution behavior as a black-box. However to use it like that, it requires some tricks to fool it into interacting with LLM. While in fact it gets some file or just a file path an response code, that is replaced via content by custom provider. So asking it for a prompt to be mandatory - not really a usefull thing, that is just placed there as a dummy.
Of course it is also used for different scenarios like chats, summarization and actual work with prompts as well. But the possible applications possibilities are wider and it's not a very good idea to limit them


I support this feature request. It would greatly enhance our workflow as we also struggle with managing multiple configurations and comparing different versions manually. Combining runs and having visualized trends would save us significant time and reduce errors. 

Thank you for considering this improvement!

I understand this is a pain point :) Have you tried the "Compare with another eval" functionality under the "Eval Actions" menu?  This may partially address your need.

![image](https://github.com/user-attachments/assets/2ef78461-29b3-404d-9fbc-471237c2b9e4)



Thanks @typpo, that in fact makes it a lot easier for comparing same suites! And something I was in fact not aware of. 
Sadly there are a few drawbacks in this approach:

1. Hard to visually differentiate separate runs, especially if you add more than one. A thicker border between them can actually solve that.
2. If there are different suites, you will still have to go through each of them, find the previous run of that suite from a drop-down list and repeat that for each suite without mistake.
3. Not able to remove, if you have accidentally added the wrong evaluation (test suite) to comparison
4. Unfortunately, numbers 1, 3, and 4 from my original feature suggestion are not going to be covered by this even partially