Repository: truera/trulens
Issue: Getting While Running provider.summary_with_cot_reasons feedback with Bedrock as Provider
Root cause: Interface Contract Mismatch

Hello,

I am trying to evaluate Summary response using below code. Prompt is reduced to keep small intentionally to keep this description short. I tried getting into the Bedrock provider code and debug but I don't understand why it throws this error message from an internal assertion? 

Appreciate any help you can extend.

**Error:**

`_
> bedrock request failed <class 'AssertionError'>=Bedrock can only operate on 'prompt', not 'messages'.. Retries remaining=3.
> bedrock request failed <class 'AssertionError'>=Bedrock can only operate on 'prompt', not 'messages'.. Retries remaining=2.
> bedrock request failed <class 'AssertionError'>=Bedrock can only operate on 'prompt', not 'messages'.. Retries remaining=1.
> bedrock request failed <class 'AssertionError'>=Bedrock can only operate on 'prompt', not 'messages'.. Retries remaining=0.
> Feedback Function exception caught: Traceback (most recent call last):
>   File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 501, in run
>     result_and_meta, part_cost = Endpoint.track_all_costs_tally(
>   File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 447, in track_all_costs_tally
>     result, cbs = Endpoint.track_all_costs(
>   File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 396, in track_all_costs
>     return Endpoint._track_costs(thunk, with_endpoints=endpoints)
>   File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 553, in _track_costs
>     result: T = thunk()
>   File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 502, in <lambda>
>     lambda: self.imp(**ins)
>   File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py", line 1013, in summary_with_cot_reasons
>     return self._extract_score_and_reasons_from_response(system_prompt)
>   File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py", line 148, in _extract_score_and_reasons_from_response
>     response = self.endpoint.run_me(
>   File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 261, in run_me
>     raise RuntimeError(
> RuntimeError: API bedrock request failed 4 time(s).
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 506, in run
>     raise RuntimeError(
> RuntimeError: Evaluation of summary_with_cot_reasons failed on inputs: 
> {'source': 'CREATE DATE:2021-12-31 NOTE:Contacted Authority: Peaceful Grove '
>            'POLICE DEPARTMENT\n'
>            'CREAT
> API bedrock request failed 4 time(s)..

_`


**Code:**

`prompt_template = """Human: You are a helpful claims notes summarization assistant working on Auto Claims Insurance. The notes provided below are sorted by date and contain text associated with them. However, the notes may have duplicate facts and sentences.

{context}

INSTRUCTIONS:
1. Your task is to summarize the notes in less than 200 words.
2. When in doubt, prioritize the latest information and eliminate duplicate facts.
3. You must respond in clear and coherent english.
4. Keep your responses concise and avoid adding extra text.

If no information exists for a given section, respond with 'No Information Found.'

Assistant:
"""

PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context"]
)

from trulens_eval import Bedrock as TBedrock
from trulens_eval.feedback.provider.base import Provider, LLMProvider
from trulens_eval.feedback.provider.bedrock import Bedrock as fBedrock
from trulens_eval.tru_custom_app import instrument

import numpy as np

# Initialize provider class
tru = Tru()
#tru.reset_database()
fclaude = fBedrock(model_id = "anthropic.claude-instant-v2")

# Define a groundedness feedback function
f_summary_with_cot = (
    Feedback(fclaude.summary_with_cot_reasons, name = "SummaryCoT")
    .on_input_output()
)


 #print(prompt)
llm = langBedrock(model_id="anthropic.claude-instant-v1")

PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context"]
)

chain = LLMChain(
    llm=llm,
    prompt=PROMPT,
    verbose=True
)
    
tru_recorder = tru.Chain(
    chain,
    app_id='Claim_Summary_Prompt1',
    feedbacks=[f_summary_with_cot]
)

with tru_recorder as recording:
    completion = chain({"context": notes}, return_only_outputs=True)`

Hey @SanjuMLGeek ! This should be fixed by https://github.com/truera/trulens/pull/732

Please take a look - thanks!

Thanks @joshreini1 . I have been searching for a good example to implement Summary with CoT  but can't seem to find one. Would you be able to point me to an example notebook or document for the same?

Not sure if we have one right now but I can create one in the new year. If you have an example use case you’d like to see, I’d be happy to add the TruLens implementation

@joshreini1 I'm still getting same issue with 0.20.0 version