Repository: truera/trulens
Issue: question: Bedrock
Root cause: Validation Gap

How can i run Bedrock with the latest version of trulens? The jupyter notebook example does not work.

Code:
```
# Initialize Huggingface-based feedback function collection class:
bedrock = Bedrock(model_engine="Bedrock", model_id = "anthropic.claude-v2", region_name="us-west-2")

# Define a language match feedback function using HuggingFace.
f_relevance = Feedback(bedrock.relevance).on_input_output()
# By default this will check language match on the main app input and main app
# output.

tru_recorder = TruChain(chain,
    app_id='Chain1_ChatApplication',
    feedbacks=[f_relevance])


with tru_recorder as recording:
    llm_response = chain.run("What's the capital of the USA?")
display(llm_response)
```

Error:
```
OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
```


@piotrm0 - this looks like a similar errors being faced as: https://github.com/truera/trulens/issues/546

Can you take a look?

I believe recent fixes and additions to BedrockEndpoint should address this problem. @arm-diaz : can you try using the trulens from github and see if you are still seeing the issue? (a release with the fixes should be available on pip later this week).

Hey, how can I install the library without the release?

I usually execute:

pip install git+https://github.com/truera/trulens.git

But i am getting an error that says,` https://github.com/truera/trulens.git does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.`



@arm-diaz You'll need to clone trulens and install from the trulens/trulens_eval directory.

```
git clone https://github.com/truera/trulens.git
cd trulens/trulens_eval
pip install -e .
```

Hi,

Is there a good reason for the bedrock endpoint name to be setup as "bedrock"?
Check link below:
https://github.com/truera/trulens/blob/6a41f2a1d201ded7c5d8add31d7cade2c2d51ed8/trulens_eval/trulens_eval/feedback/provider/endpoint/bedrock.py#L139

if I change it to bedrock-runtime, it works fine. otherwise, it gives me the following error:

```
bedrock request failed <class 'AttributeError'>='NoneType' object has no attribute 'invoke_model'. Retries remaining=3.
bedrock request failed <class 'AttributeError'>='NoneType' object has no attribute 'invoke_model'. Retries remaining=2.
bedrock request failed <class 'AttributeError'>='NoneType' object has no attribute 'invoke_model'. Retries remaining=1.
bedrock request failed <class 'AttributeError'>='NoneType' object has no attribute 'invoke_model'. Retries remaining=0.
Feedback Function exception caught: Traceback (most recent call last):
  File "trulens/trulens_eval/trulens_eval/feedback/feedback.py", line 481, in run
    result_and_meta, part_cost = Endpoint.track_all_costs_tally(
  File "trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 438, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
  File /trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 388, in track_all_costs
    return Endpoint._track_costs(thunk, with_endpoints=endpoints)
  File "trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 544, in _track_costs
    result: T = thunk()
  File "trulens/trulens_eval/trulens_eval/feedback/feedback.py", line 482, in <lambda>
    lambda: self.imp(**ins)
  File "trulens/trulens_eval/trulens_eval/feedback/provider/base.py", line 286, in relevance
    self.endpoint.run_me(
  File "trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 263, in run_me
    raise RuntimeError(
RuntimeError: API bedrock request failed 4 time(s).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "trulens/trulens_eval/trulens_eval/feedback/feedback.py", line 486, in run
    raise RuntimeError(
RuntimeError: Evaluation of relevance failed on inputs: 
{'prompt': "What's the capital of the USA?",
 'response': '\n'
             'AI: The capital of the United States of America is 
API bedrock request failed 4 time(s)..
```

in addition, can you guys provide a good example with claude on bedrock?

model_id="anthropic.claude-v2"

I tried to use it. But it gives me the following error:

```
bedrock-runtime request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=3.
bedrock-runtime request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=2.
bedrock-runtime request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=1.
bedrock-runtime request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=0.
Feedback Function exception caught: Traceback (most recent call last):
  File "trulens/trulens_eval/trulens_eval/feedback/feedback.py", line 481, in run
    result_and_meta, part_cost = Endpoint.track_all_costs_tally(
  File "trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 438, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
  File "trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 388, in track_all_costs
    return Endpoint._track_costs(thunk, with_endpoints=endpoints)
  File "trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 544, in _track_costs
    result: T = thunk()
  File "trulens/trulens_eval/trulens_eval/feedback/feedback.py", line 482, in <lambda>
    lambda: self.imp(**ins)
  File "trulens/trulens_eval/trulens_eval/feedback/provider/base.py", line 286, in relevance
    self.endpoint.run_me(
  File "trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 263, in run_me
    raise RuntimeError(
RuntimeError: API bedrock-runtime request failed 4 time(s).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "trulens/trulens_eval/trulens_eval/feedback/feedback.py", line 486, in run
    raise RuntimeError(
RuntimeError: Evaluation of relevance failed on inputs: 
{'prompt': "What's the capital of the USA?",
 'response': ' Washington D.C. is the capital of the United States of America.'}
API bedrock-runtime request failed 4 time(s)..
```
The error only shows up when I changed the model to claude. it works okay with titan models.


Hey @arm-diaz !

I can put together an example with claude on bedrock soon. In the meantime, can you try upgrading trulens_eval to 0.18.3 to see if this fixes your issues?

Thanks!

> Hi,
> 
> Is there a good reason for the bedrock endpoint name to be setup as "bedrock"? Check link below:
> 
> https://github.com/truera/trulens/blob/6a41f2a1d201ded7c5d8add31d7cade2c2d51ed8/trulens_eval/trulens_eval/feedback/provider/endpoint/bedrock.py#L139
> 
> if I change it to bedrock-runtime, it works fine. otherwise, it gives me the following error:
> 
> ```
> bedrock request failed <class 'AttributeError'>='NoneType' object has no attribute 'invoke_model'. Retries remaining=3.
> bedrock request failed <class 'AttributeError'>='NoneType' object has no attribute 'invoke_model'. Retries remaining=2.
> bedrock request failed <class 'AttributeError'>='NoneType' object has no attribute 'invoke_model'. Retries remaining=1.
> bedrock request failed <class 'AttributeError'>='NoneType' object has no attribute 'invoke_model'. Retries remaining=0.
> Feedback Function exception caught: Traceback (most recent call last):
>   File "trulens/trulens_eval/trulens_eval/feedback/feedback.py", line 481, in run
>     result_and_meta, part_cost = Endpoint.track_all_costs_tally(
>   File "trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 438, in track_all_costs_tally
>     result, cbs = Endpoint.track_all_costs(
>   File /trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 388, in track_all_costs
>     return Endpoint._track_costs(thunk, with_endpoints=endpoints)
>   File "trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 544, in _track_costs
>     result: T = thunk()
>   File "trulens/trulens_eval/trulens_eval/feedback/feedback.py", line 482, in <lambda>
>     lambda: self.imp(**ins)
>   File "trulens/trulens_eval/trulens_eval/feedback/provider/base.py", line 286, in relevance
>     self.endpoint.run_me(
>   File "trulens/trulens_eval/trulens_eval/feedback/provider/endpoint/base.py", line 263, in run_me
>     raise RuntimeError(
> RuntimeError: API bedrock request failed 4 time(s).
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "trulens/trulens_eval/trulens_eval/feedback/feedback.py", line 486, in run
>     raise RuntimeError(
> RuntimeError: Evaluation of relevance failed on inputs: 
> {'prompt': "What's the capital of the USA?",
>  'response': '\n'
>              'AI: The capital of the United States of America is 
> API bedrock request failed 4 time(s)..
> ```

I still got same error even I set name="bedrock-runtime",

`client = boto3.client(service_name="bedrock-runtime", region_name="us-east-1")`

`from langchain.prompts.chat import (
    ChatPromptTemplate,
    SystemMessagePromptTemplate,
    AIMessagePromptTemplate,
    HumanMessagePromptTemplate,
)
from langchain import LLMChain
from langchain.llms.bedrock import Bedrock
bedrock_llm = Bedrock(
    model_id=model_id,
    client=client
)


template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
example_human = HumanMessagePromptTemplate.from_template("Hi")
example_ai = AIMessagePromptTemplate.from_template("Hi")
human_template = "{text}"


human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)

chat_prompt = ChatPromptTemplate.from_messages(
    [system_message_prompt, example_human, example_ai, human_message_prompt]
)
chain = LLMChain(llm=bedrock_llm, prompt=chat_prompt, verbose=True)

print(chain.run("What's the capital of the USA?"))

#Instrument chain for logging with TruLens
tru_recorder = TruChain(
    chain, 
    app_id='Chain1_ChatApplication',
    feedbacks=[f_relevance])

with tru_recorder as recording:
    result = chain.run("What's the capital of the USA?")  

display(result)
`

A new object of type <class 'langchain.chains.llm.LLMChain'> at 0x7f1bd7d59340 is calling an instrumented method <function Chain.__call__ at 0x7f1c5a0465f0>. The path of this call may be incorrect.
Guessing path of new object is app based on other object (0x7f1bdc284680) using this function.
A new object of type <class 'langchain.chains.llm.LLMChain'> at 0x7f1bd7d59340 is calling an instrumented method <function LLMChain._call at 0x7f1c5a0471c0>. The path of this call may be incorrect.
Guessing path of new object is app.combine_documents_chain.llm_chain based on other object (0x7f1bd7de7c80) using this function.


> Entering new LLMChain chain...
Prompt after formatting:
System: You are a helpful assistant.
Human: Hi
AI: Hi
Human: What's the capital of the USA?

> Finished chain.
' The capital of the United States of America is Washington, D.C.'
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=3.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=2.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=1.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=0.
Feedback Function exception caught: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 481, in run
    result_and_meta, part_cost = Endpoint.track_all_costs_tally(
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 438, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 388, in track_all_costs
    return Endpoint._track_costs(thunk, with_endpoints=endpoints)
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 544, in _track_costs
    result: T = thunk()
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 482, in <lambda>
    lambda: self.imp(**ins)
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py", line 286, in relevance
    self.endpoint.run_me(
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 263, in run_me
    raise RuntimeError(
RuntimeError: API bedrock request failed 4 time(s).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 486, in run
    raise RuntimeError(
RuntimeError: Evaluation of relevance failed on inputs: 
{'prompt': "What's the capital of the USA?",
 'response': ' The capital of the United States of America is Washington, D.C.'}
API bedrock request failed 4 time(s)..

Any help? Thank you

> 0.18.3

I'm using trulens-eval 0.18.3 langchain 0.0.305

The error I got keep saying that" bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=3."




I heard this is an issue with the prompt: https://github.com/langchain-ai/langchain/issues/11130
All bedrock prompts must be in the following format "\n\nHuman:
\n\nAssistant:"



Hey @HannaHUp ! Can you upgrade to trulens_eval==0.19.2? We did a dependency refactor among some other changes that should help with this.

Hey @joshreini1
Thank you. I have upgraded it and still got the same issue. 

```

A new object of type <class 'langchain.chains.llm.LLMChain'> at 0x7f79030daf00 is calling an instrumented method <function LLMChain._call at 0x7f795800acb0>. The path of this call may be incorrect.
Guessing path of new object is app.combine_documents_chain.llm_chain based on other object (0x7f7903278500) using this function.
' Washington D.C. is the capital of the United States of America. As for a company name suggestion, here are a few ideas:\n\n- Capital City Co. \n- Washington Works\n- D.C. Creations\n- American Artisans \n- Stars & Stripes Studio\n- Patriot Productions\n- Freedom Foundry\n\nThe name could highlight "Washington D.C." or "capital" if you want to emphasize the location/theme. Pick something catchy and descriptive. You\'ll also want to check that the name isn\'t already trademarked before deciding. But hopefully these spark some ideas for you!'
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=3.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=2.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=1.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=0.
Feedback Function exception caught: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 481, in run
    )
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 438, in track_all_costs_tally
    with_litellm: bool = True,
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 388, in track_all_costs
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 544, in _track_costs
    endpoints[callback_class] = []
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 482, in <lambda>
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py", line 286, in relevance
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 263, in run_me
    )
RuntimeError: API bedrock request failed 4 time(s).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 486, in run
    try:
RuntimeError: Evaluation of relevance failed on inputs: 
{'prompt': "What's the capital of the USA?",
 'response': ' Washington D.C. is the capital of the United States of America. '
  
API bedrock request failed 4 time(s)..

```

Here is my code:

`from langchain.chains import LLMChain
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate
import boto3
client = boto3.client(service_name="bedrock-runtime", region_name="us-east-1")
from langchain.llms.bedrock import Bedrock
bedrock_llm = Bedrock(
    model_id=model_id,
    client=client
)


prompt_template = "What is a good name for a company that makes {product}?"

llm_chain = LLMChain(llm=bedrock_llm, prompt=PromptTemplate.from_template(prompt_template))
llm_chain.run("colorful socks")

tru_recorder = TruChain(
    llm_chain, 
    app_id='Chain1_ChatApplication',
    feedbacks=[f_relevance])

with tru_recorder as recording:
    result = llm_chain.run("What's the capital of the USA?")  # it is not working too. same prompting error!

display(result)
`



Hi guys,
I'm having same issue with Claude v2 i.e.

> bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=3.
> bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=2.
> bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=1.

When I tried to use Titan it gave different error i.e. 

> 'Bedrock' object has no attribute 'relevance'

The issues listed above should be fixed in various changes included in 0.20.1 and 0.20.2. Please upgrade to 0.20.2 to get all changes and re-open if you continue to experience issues using the Bedrock provider.

Hi I'm using 0.20.1 and 0.20.2.But now I can't even import package?

!pip install trulens-eval==0.20.2

Name: trulens-eval
Version: 0.20.2
Summary: Library to systematically track and evaluate LLM based applications.
![image](https://github.com/truera/trulens/assets/43529795/8eb4344a-280e-4ae9-b89b-0150ae1452b7)


I also tied !pip install trulens-eval==0.20.1
I still got same error
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: required key [max_tokens_to_sample] not found#: extraneous key [inputText] is not permitted#: extraneous key [textGenerationConfig] is not permitted, please reformat your input and try again.. Retries remaining=3.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: required key [max_tokens_to_sample] not found#: extraneous key [inputText] is not permitted#: extraneous key [textGenerationConfig] is not permitted, please reformat your input and try again.. Retries remaining=2.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: required key [max_tokens_to_sample] not found#: extraneous key [inputText] is not permitted#: extraneous key [textGenerationConfig] is not permitted, please reformat your input and try again.. Retries remaining=1.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: required key [max_tokens_to_sample] not found#: extraneous key [inputText] is not permitted#: extraneous key [textGenerationConfig] is not permitted, please reformat your input and try again.. Retries remaining=0.
Feedback Function exception caught: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 501, in run
    result_and_meta, part_cost = Endpoint.track_all_costs_tally(
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 465, in track_all_costs_tally
    with_hugs=with_hugs,
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 414, in track_all_costs
    @staticmethod
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 571, in _track_costs
    # return others.
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 502, in <lambda>
    lambda: self.imp(**ins)
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/base.py", line 286, in relevance
    "relevant".
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 268, in run_me
RuntimeError: API bedrock request failed 4 time(s).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/feedback.py", line 506, in run
    raise RuntimeError(
RuntimeError: Evaluation of relevance failed on inputs: 
{'prompt': "What's the capital of the USA?",
 'response': ' Washington D.C. is the capital of the United States of America. '
  
API bedrock request failed 4 time(s)..