Repository: promptfoo/promptfoo
Issue: Support setup/teardown commands for tests
Root cause: Unimplemented Feature Gap

When testing agents on complex situations, we need to setup the test first and then tear it down. We currently implement this with a custom provider that reads `setup` and `teardown` variables and interprets them as bash scripts.

In our case, we are testing an [AIOps agent](http://github.com/robusta-dev/holmesgpt/), so setup/teardown involves creating IT problems and opening Jira tickets for them.

However, using a custom provider to do this has a number of limitations:

* When running promptfoo with --repeat there isn't a great way to identify that a test was already setup and therefore doesn't need further setup. We've tried to work around this, but its ugly. In our case, setting up tests can take up to 90 seconds, so adding 90 second latency for each eval is painful.
* We have a similar problem with parallelism when using promptfoo Scenarios and multiple tests on the same test variables. Here too, we want to setup the test once, not one time per test.

For the above reasons, having first-class support for setup/teardown in promptfoo would let us solve this more elegantly than using a custom provider. I believe this type of requirement will become increasingly relevant for other promptfoo users as people test agents on more and more complex cases.

For our needs, having a setup/teardown setting in each test would be sufficient. It can be interpreted as a bash script which is templated with vars. There should also be a setting to control if the setup/teardown happen once or many times when the test is run multiple times (due to either --repeat or Scenarios)

One alternative to this that would work for us: add a callback to custom providers that is called once before all tests are run and once after all tests are run. If the provider receives a list of tests and their configurations that will be run, that will give us the capability to solve this on the provider side.

> One alternative to this that would work for us: add a callback to custom providers that is called once before all tests are run and once after all tests are run. If the provider receives a list of tests and their configurations that will be run, that will give us the capability to solve this on the provider side.



> One alternative to this that would work for us: add a callback to custom providers that is called once before all tests are run and once after all tests are run. If the provider receives a list of tests and their configurations that will be run, that will give us the capability to solve this on the provider side.

I came across same issue. Could you share how to add callback providers? Can show a simple demo?

Hey, to clarify promptfoo doesn't currently support a global initialization callback for providers like I described. I was just suggesting it as another way we could solve the same problem, if that was implemented.

What is your use case for it?

On Fri, Jun 28, 2024, 21:19 landhu ***@***.***> wrote:

> One alternative to this that would work for us: add a callback to custom
> providers that is called once before all tests are run and once after all
> tests are run. If the provider receives a list of tests and their
> configurations that will be run, that will give us the capability to solve
> this on the provider side.
>
> One alternative to this that would work for us: add a callback to custom
> providers that is called once before all tests are run and once after all
> tests are run. If the provider receives a list of tests and their
> configurations that will be run, that will give us the capability to solve
> this on the provider side.
>
> I came across same issue. Could you share how to add callback providers?
> Can show a simple demo?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/1022#issuecomment-2197425134>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADYUB6P4XQ37QEAJTXE4NDZJWSLXAVCNFSM6AAAAABJ7DJSQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJXGQZDKMJTGQ>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


Adding one more comment on this: it doesn't need to be a per-provider callback. I just need *any* extension point that is called with a list of tests that will be run. It can be a global extension hook as well. i.e. if I could easily provide a javascript function that is called before tests are actually run (and called again afterwards), that works. 

@typpo would you be open to accepting a PR that adds this, in the form of a global extension point called [around here](https://github.com/promptfoo/promptfoo/blob/50b943ffcb985531fcf5b28ae3b0e26d7060d9ed/src/evaluator.ts#L681)?

@aantn Thanks for the detailed explanation and suggestions. I agree that having first-class support for setup/teardown commands would be highly beneficial, especially for complex testing scenarios like yours.

I see two viable paths forward:

First-class support for setup/teardown: Implementing setup/teardown settings in each test, interpreted as bash scripts with templated variables, and controlling their execution frequency (once or multiple times).

Global initialization callback: Adding a global extension point or a callback to custom providers, called before and after all tests. This could involve passing a list of tests and their configurations, allowing for flexible initialization and teardown processes.

I believe the global callback is a simpler first step and can provide immediate relief for your use case. If you can work on a PR for this, we would be happy to review and merge it. Thanks for the suggestion and for offering to drive this.

Thanks, I've created a PR with the global callback. Please provide any feedback.

I've updated the PR with an example showing how it can be used to implement setup/teardown commands.