Repository: truera/trulens
Issue: Exception in LangChain causes TruLens evaluation to halt 
Root cause: External Dependency Breakage

First off, this may be a langChain issue and not a TruLens issue. Let me know if I should submit a ticket with them instead.  A similar evaluation using llamaIndex did not raise an unhandled exception on the same query.

The query:
```
What did the Manchu's make all the men in the city do?
```

I'm getting this exception/stack trace when trying to evaluate a langchain pipeline. I tried wrapping the code with a try/except block, but I still encounter the error.

I'm using AstraDB vector store and AzureOpenAI gpt-3.5-turbo chat LLM for inference and evaluation.

```
Traceback (most recent call last):
  File "/home/ec2-user/tru_langchain.py", line 53, in <module>
    pipeline.invoke(query)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/app.py", line 734, in __exit__
    raise exc_value
  File "/home/ec2-user/tru_langchain.py", line 53, in <module>
    pipeline.invoke(query)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/instruments.py", line 778, in tru_wrapper
    ctx.app._on_add_record(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/app.py", line 782, in _on_add_record
    raise error
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/instruments.py", line 729, in tru_wrapper
    rets, cost = Endpoint.track_all_costs_tally(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 459, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 408, in track_all_costs
    return Endpoint._track_costs(thunk, with_endpoints=endpoints)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 565, in _track_costs
    result: T = thunk()
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/instruments.py", line 730, in <lambda>
    lambda: func(*bindings.args, **bindings.kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 1514, in invoke
    input = step.invoke(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/instruments.py", line 790, in tru_wrapper
    raise error
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/instruments.py", line 729, in tru_wrapper
    rets, cost = Endpoint.track_all_costs_tally(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 459, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 408, in track_all_costs
    return Endpoint._track_costs(thunk, with_endpoints=endpoints)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 565, in _track_costs
    result: T = thunk()
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/instruments.py", line 730, in <lambda>
    lambda: func(*bindings.args, **bindings.kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 160, in invoke
    self.generate_prompt(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 491, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 378, in generate
    raise e
  File "/home/ec2-user/.local/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 368, in generate
    self._generate_with_cache(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 524, in _generate_with_cache
    return self._generate(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/langchain_community/chat_models/openai.py", line 435, in _generate
    response = self.completion_with_retry(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/langchain_community/chat_models/openai.py", line 352, in completion_with_retry
    return self.client.create(**kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 843, in wrapper
    response: Any = func(*args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/openai/_utils/_utils.py", line 303, in wrapper
    return func(*args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/openai/resources/chat/completions.py", line 604, in create
    return self._post(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/openai/_base_client.py", line 1088, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/home/ec2-user/.local/lib/python3.9/site-packages/openai/_base_client.py", line 853, in request
    return self._request(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/openai/_base_client.py", line 930, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400}}
```

my LCEL pipeline:

```
prompt_template = """
Answer the question based only on the supplied context. If you don't know the answer, say: "I don't know".
Context: {context}
Question: {question}
Your answer:
"""
prompt = ChatPromptTemplate.from_template(prompt_template)

pipeline = (
    {"context": vstore.as_retriever(), "question": RunnablePassthrough()}
    | prompt
    | chatModel
    | StrOutputParser()
)
```

my feedback functions:

```
def getFeedbackFunctions(pipeline, golden_set):
    # Initialize provider class
    azureOpenAI = AzureOpenAI(deployment_name="gpt-35-turbo")

    context = App.select_context(pipeline)

    # Define a groundedness feedback function
    grounded = Groundedness(groundedness_provider=azureOpenAI)
    f_groundedness = (
        Feedback(grounded.groundedness_measure_with_cot_reasons)
        .on(context.collect()).on_output()
        .aggregate(grounded.grounded_statements_aggregator)
    )

    # Question/answer relevance between overall question and answer.
    f_answer_relevance = (
        Feedback(azureOpenAI.relevance_with_cot_reasons)
        .on_input_output()
    )

    # Question/statement relevance between question and each context chunk.
    f_context_relevance = (
        Feedback(azureOpenAI.qs_relevance_with_cot_reasons)
        .on_input().on(context)
        .aggregate(np.mean)
    )

    # GroundTruth for comparing the Answer to the Ground-Truth Answer
    ground_truth_collection = GroundTruthAgreement(golden_set, provider=azureOpenAI)
    f_answer_correctness = (
        Feedback(ground_truth_collection.agreement_measure)
        .on_input_output()
    )
    return [f_answer_relevance, f_context_relevance, f_groundedness, f_answer_correctness]
```

eval execution:
```
feedbacks = getFeedbackFunctions(pipeline, golden_set)

for name in datasets:
    tru_recorder = TruChain(
        pipeline,
        app_id=name,
        feedbacks=feedbacks,
    )
    for query in datasets[name]:
        try:
            with tru_recorder as recording:
                pipeline.invoke(query)
        finally:
```



How different is the full prompt when using the Llama-Index pipeline? Is it possible there's some difference in the full prompts between the two that causes this issue?

Might be worth adjusting the content filtering also: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/content-filters

Do you get the same error when you just do:
`pipeline.invoke(query)` without TruLens?

good idea on trying `pipeline.invoke(query)` without TruLens. I do get the same error. 

I'm struggling to find the LlamaIndex prompt.

Either way, I think it is okay to close this here and instead make an issue at langchain.