Repository: truera/trulens
Issue: question: Bedrock claude
Root cause: Interface Contract Mismatch

Hi 
The issue is from https://github.com/truera/trulens/issues/609

I'm using 0.20.1 and 0.20.2.

!pip install trulens-eval==0.20.2

Name: trulens-eval
Version: 0.20.2
Summary: Library to systematically track and evaluate LLM based applications.
image

I also tied !pip install trulens-eval==0.20.1
I still got same error
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: required key [max_tokens_to_sample] not found#: extraneous key [inputText] is not permitted#: extraneous key [textGenerationConfig] is not permitted, please reformat your input and try again.. Retries remaining=3.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: required key [max_tokens_to_sample] not found#: extraneous key [inputText] is not permitted#: extraneous key [textGenerationConfig] is not permitted, please reformat your input and try again.. Retries remaining=2.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: required key [max_tokens_to_sample] not found#: extraneous key [inputText] is not permitted#: extraneous key [textGenerationConfig] is not permitted, please reformat your input and try again.. Retries remaining=1.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: required key [max_tokens_to_sample] not found#: extraneous key [inputText] is not permitted#: extraneous key [textGenerationConfig] is not permitted, please reformat your input and try again.. Retries remaining=0.
Feedback Function exception caught: Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 501, in run
result_and_meta, part_cost = Endpoint.track_all_costs_tally(
File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 465, in track_all_costs_tally
with_hugs=with_hugs,
File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 414, in track_all_costs
@staticmethod
File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 571, in _track_costs
# return others.
File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 502, in
lambda: self.imp(**ins)
File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py", line 286, in relevance
"relevant".
File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 268, in run_me
RuntimeError: API bedrock request failed 4 time(s).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 506, in run
raise RuntimeError(
RuntimeError: Evaluation of relevance failed on inputs:
{'prompt': "What's the capital of the USA?",
'response': ' Washington D.C. is the capital of the United States of America. '

API bedrock request failed 4 time(s)..

Hey @HannaHUp - we've heard from some other folks that the Bedrock models can be inconsistent in their ability to perform these evals. We've specifically tested and confirmed the Titan models succeed with this task.

We'll continue to work on expanding this to ensure this works on a larger set of models from Bedrock, but there may be limitations in the models themselves.

Thanks for understanding