Repository: promptfoo/promptfoo
Issue: Feature Request: Testing Chains of Prompts
Root cause: Unimplemented Feature Gap

So I've been looking for a way to enforce a stricter evaluation metric for testing prompt-engineering and this looks like the best open source tool for the job. However, it seems like currently promptfoo is only supporting "single Q/A" prompts ie. prompts that only generate one answer. I am building a chatbot right now that will generate a chain of responses that feeds into the next prompt, based on a constrained multiple choice selection from the user on the answer from the previous prompt. So something like:

promptA => answer [1,2,3,4] -- [ choose 2. ] --> promptB => answer [1,2,3,4]  ---> [ choose 3.] ---> promptC .....

Is there a good way to modify the current code to support visualizing this type of chained prompt?

Hi @JohnPeng47,

Currently the way I'm testing prompt chains is step-by-step, with separate configs for each step.  I realize that is not ideal.  Do you have any thoughts on what your ideal integration would look like for testing chains of prompts?  There is complexity here because the output of intermediate prompts is often manipulated with custom logic before being fed into the next prompt.

Hmm. An ideal ideal integration imo would be at a LLM chaining framework level (ie. Langchain). Some advantages as far as I can see:
- you can get the direct prompt used for inference after any pre-processing step
- you can trigger promptfoo eval automatically as part of a chain inference flow, with no/minimal code

Not sure how much work it would be, altho I did poke around the langchain code a little bit, it shouldn't be too difficult to plug promptfoo into their LLMChains. Also not sure how the Python/Typescript interface would look like, you might have some better ideas for that


Hi @JohnPeng47,

Thanks for your thoughts on this.  At the moment, the best way to test a prompt chain is by using a custom provider: https://www.promptfoo.dev/docs/configuration/providers#custom-api-provider

Your custom provider should run the full chain and return the final result.  Right now you'd have to write some Javascript for this, but if your chain is not written in Javascript, you could use Javascript to [execute a shell command](https://stackoverflow.com/questions/1880198/how-to-execute-shell-command-in-javascript).

Ultimately I see two ways to support chains:
- Make it possible to write a third party provider that handles the chain.  This is what is currently supported, and there are some small improvements that can be made (e.g. to make it easier to execute a script, so you can be fully language-agnostic)
- Require the user to instrument their code with both a trigger and a completion call.  This requires a small SDK per language, or something like webhook support.

I've made it easier to write third-party providers and added some documentation for [end-to-end LLM chain testing](https://www.promptfoo.dev/docs/configuration/testing-llm-chains#end-to-end-testing-for-llm-chains).

Specifically, I've created [an example](https://github.com/promptfoo/promptfoo/tree/main/examples/langchain-python) that shows how to evaluate a Python LangChain implementation.

The example compares raw GPT-4 with LangChain's LLM-Math plugin by using the `exec` provider to run the LangChain script:
```yaml
# promptfooconfig.yaml
# ...
providers:
  - openai:chat:gpt-4-0613
  - exec:python langchain_example.py
# ...
```

The result is a side-by-side comparison of GPT-4 and LangChain doing math:

![langchain gpt-4 eval](https://www.promptfoo.dev/assets/images/langchain-eval-1c44c280e70607b6c1682aa03ee33074.png)

Hope this helps your use case.  If not, interested in learning more.  

I also have begun implementing a webhook provider, which allows you to instrument a webapp for evaluation with a trigger and completion call.

Ah this is interesting. Yes, this is very helpful, much appreciate it @typpo!

Sorry for reopening, I was just gonna say that you should provide some more flexibility in terms of the new script providers features, to allow post-processing of the prompt inputs, but then I realized that your current implementation allows for it already since it just reads the script output from stdout. Thanks again!