Repository: promptfoo/promptfoo
Issue: Test a component of a LangChain chain
Root cause: Documentation Deficiency

I have what I think is a simple question:

We are using the LangChain ConversationalRetrievalChain and I like the look of promptfoo. 

I followed the example in the documentation to make my Python executable as follows:
```python
import sys

from langchain import chains
from langchain.prompts import PromptTemplate

from hotodgqa.data import prompts
from hotdogqa.data.bot import preconfigured

INDEX_NAME = 'index'

chat_model = preconfigured.ModelProvider.get_chat_model()
embedding_model = preconfigured.ModelProvider.get_embedding_model()
vector_store = preconfigured.VectorStoreProvider.get_vectorstore(INDEX_NAME, embedding_model.embed_query)

QUESTION = "How do I prepare the perfect hotdog?"

qa_prompt_template = sys.argv[1]

qa_prompt = PromptTemplate(
    template=qa_prompt_template, input_variables=["context", "question"]
)

chain = chains.ConversationalRetrievalChain.from_llm(
            llm=chat_model,
            retriever=vector_store.as_retriever(),
            verbose=True,
            rephrase_question=True,
            condense_question_prompt=prompts.CONDENSE_QUESTION_PROMPT,
            combine_docs_chain_kwargs={"document_prompt": prompts.DOCUMENT_PROMPT, "prompt": qa_prompt},
            return_source_documents=False,
        )

chain.run({'question': QUESTION, 'chat_history': []})
```

Here I am fixing the input question to the chain and altering the question-answering prompt in the chain. I have a ```prompts.txt``` that looks like this:
```text
{{prompt}}

{context}

Question: {question}
Helpful Answer:
```

I expected that the ```{{prompt}}``` would get replaced by the contents of my ```promptfooconfig.yaml``` field and then a LangChain```PromptTemplate``` could be constructed where the program correctly identifies {context} and {question} as the input variables. However, I just keep getting errors no matter how I try to format things. 

Is there a proper way to do this? Also ideally, I would like to pass a list of initial questions to the chain and evaluate the final output with different Q&A templates. Is there some way to do this without isolating every component of the chain?


Hey @lambertsbennett, happy to help.  Would you be able to share the following to help further troubleshoot:
1. Relevant portions of your promptfooconfig.yaml
2. In the web viewer, click the magnifying glass (:mag:) icon that appears when you hover over an output open a Details modal and see the raw prompt that's sent to the script.  We can use this to check if the full prompt is being properly constructed.
3. More detail on the error that you're getting when you run the eval

1. Here's the ```promptfooconfig.yaml```:
```yaml
prompts: prompts.txt
providers:
  - exec:python chain_evaluation.py
tests:
  - vars:
      prompt: "Use the following sources to answer the question at the end. For each source used to answer the question include the 'Source Page Number' and 'Source Section' at the end of the answer in parentheses. Example: (Source Section 6.5, Source Page Number 18). Many sources refer to images that are critical to understanding. These image references are present in square brackets. When information from the source used to construct and answer contains an image reference also include that in the answer. Example: Source Text: 'When a hotdog cracks [E02220-11,  1]  it is ready to eat.'. Answer text: 'When a hotdog cracks [E02220-11,  1]  it is ready to eat.'. If you do not know the answer, just say that you do not know, do not try to make up an answer."
```
2. 
<img width="859" alt="image" src="https://github.com/promptfoo/promptfoo/assets/46295426/f2d45918-35f9-48a0-b82f-b33e9eed18a8">
3.  Here's the error trace:

Traceback (most recent call last):
File "C:\Users\lambertb\Documents\00_092_HOTDOGQA\modqa\evaluation\chain_evaluation.py", line 19, in
qa_prompt = PromptTemplate(
File "C:\Users\lambertb\Anaconda3\envs\00_092_HOTDOGQA\lib\site-packages\langchain\load\serializable.py", line 75, in __init__
super().__init__(**kwargs)
File "C:\Users\lambertb\Anaconda3\envs\00_092_HOTDOGQA\lib\site-packages\pydantic\v1\main.py", line 341, in __init__
raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for PromptTemplate
__root__
Invalid prompt schema; check for mismatched or missing input parameters. {'question', 'context'} (type=value_error)

at ChildProcess.exithandler (node:child_process:422:12)
at ChildProcess.emit (node:events:517:28)
at maybeClose (node:internal/child_process:1098:16)
at ChildProcess._handle.onexit (node:internal/child_process:303:5)


The problem seems to be that when the constructed prompt is passed to the ```PromptTemplate``` constructor that it doesn't recognize {context} and {question} as input variables. 

I'm not a langchain expert, but it seems like you may not be passing the template strings to `chain.run` correctly.  According to [this documentation](https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/prompts_pipelining#string-prompt-pipelining), it should look like:

```py
chain.run(question=question, context=context)
```

whereas you're passing a dict instead of kwargs, and you do not supply `context` anywhere in your script.

The problem here is that the ConversationalRetrievalChain is composed of several subchains. The top-level chain takes a single question and the chat history, passes this to another chain which, rephrases that to a standalone question. This standalone question is then embedded and documents are retrieved from a vector database. The standalone question generated by the first chain and the documents are then fed to the last chain which performs question answering. LangChain takes care that these intermediate steps work well together. What seems to be happening here is that when the prompt is created by promptfoo it can no longer be interpreted properly LangChain, such that {context} and {question} are not recognized. I think it likely has to do with how to prompt is formatted, but I still can't sort it out.

fwiw, I still don't see in your python code where `{context}` is provided to LangChain.  Is this just some magic keyword, or does it expect you to provide a `context` variable?

It does not expect a context variable at the start of the chain. The thing is that the document search returns documents that get formatted and then passed into the next step in the chain with the variable name ```context```. This is hidden somewhat deep inside the ConversationalRetrievalChain.