Repository: truera/trulens
Issue: Consistent format expectations for all Bedrock models
Root cause: Interface Contract Mismatch

bedrock request failed <class 'botocore.errorfactory.ResourceNotFoundException'>=An error occurred (ResourceNotFoundException) when calling the InvokeModel operation: Could not resolve the foundation model from the provided model identifier.. Retries remaining=3.
bedrock request failed <class 'botocore.errorfactory.ResourceNotFoundException'>=An error occurred (ResourceNotFoundException) when calling the InvokeModel operation: Could not resolve the foundation model from the provided model identifier.. Retries remaining=2.
bedrock request failed <class 'botocore.errorfactory.ResourceNotFoundException'>=An error occurred (ResourceNotFoundException) when calling the InvokeModel operation: Could not resolve the foundation model from the provided model identifier.. Retries remaining=1.
bedrock request failed <class 'botocore.errorfactory.ResourceNotFoundException'>=An error occurred (ResourceNotFoundException) when calling the InvokeModel operation: Could not resolve the foundation model from the provided model identifier.. Retries remaining=0.


I'm getting these issue while running the Feedback functions using bedrock

from trulens_eval import TruChain, Feedback, Bedrock, Tru
bedrock = Bedrock(model_id = "amazon.ai21.j2-ultra-v1", region_name="us-east-1")

f_qa_relevance = (
    Feedback(bedrock.relevance_with_cot_reasons, name = "Answer Relevance")
    .on(Select.RecordCalls.retrieve.args.query)
    .on_output())

 #Question/statement relevance between question and each context chunk.
f_context_relevance = (
    Feedback(bedrock.qs_relevance_with_cot_reasons, name = "Context Relevance")
    .on(Select.RecordCalls.retrieve.args.query)
    .on(Select.RecordCalls.retrieve.rets.collect())
    .aggregate(np.mean)














Hey @budhewarvijay0407 ! Looks like the issue you’re having is not passing the correct model_id to bedrock. This could be because you don’t have access to the model or you have incorrectly included the ID.

on first glance, it looks like it should be without the “Amazon prefix, like so: ai21.j2-ultra-v1

reference: https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids-arns.html

Hi @joshreini1 , Thanks for your reply , it was ny typo , However the error still persists , please have a look at this :

from trulens_eval import Feedback, Select
from trulens_eval.feedback import Groundedness

from trulens_eval import TruChain, Feedback, Bedrock, Tru
import numpy as np
bedrock = Bedrock(model_id = "ai21.j2-mid-v1", region_name="us-east-1")
grounded = Groundedness(groundedness_provider=bedrock)
f_relevance = Feedback(bedrock.relevance).on_input_output()
# Define a groundedness feedback function
f_groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)

When we run this with TruCustomApp , it throws this error : 
"bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=3."


This issue expects some input schema , which is mismatched 


Please upgrade your bedrock version and let me know if it still persists

I'm now operating on the latest release of boto3 and botocore 
boto3                         1.34.17
botocore                      1.34.17

When i ran the following code , it still generates the error


bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: 3 schema violations found, please reformat your input and try again.. Retries remaining=3.



reproduceable code :


from trulens_eval import Feedback, Select
from trulens_eval.feedback import Groundedness
from trulens_eval import TruChain, Feedback, Bedrock, Tru
bedrock_main = Bedrock(model_id = "ai21.j2-mid-v1", region_name="us-east-1")
from trulens_eval.feedback.provider.bedrock import Bedrock as fbedrock
bedrock = fbedrock(model_id = "ai21.j2-ultra-v1")
grounded = Groundedness(groundedness_provider=bedrock_main)
f_relevance = Feedback(bedrock_main.relevance).on_input_output()
# Define a groundedness feedback function
f_groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)

-----Works fine--------
tru_rag = TruCustomApp(rag,
    app_id = 'RAG_v_1',
    feedbacks = [f_groundedness])
-----Works fine--------
import pandas as pd
import time 
#df= pd.read_csv(r'C:\Users\Rideema Malji\OneDrive\Desktop\Others\Upwork\SGD-report analysis\SDG-LLMops\streamlit test\esg_analytics_new-main (1)\esg_analytics_new-main - LLMops_Trulens\ai_goals_summary.csv')
total_queries = ["Who is prime minister of india","What is the purpose of Surat diamond industry?","When was the inauguration of Diamond Bourse?"]
with tru_rag as recording:
    rag.query(f"{total_queries[1]}")


--Produces error---


Hey @budhewarvijay0407 think I found the issue.

In the below lines, you set the `groundedness_provider=bedrock_main`. Instead it should just be bedrock (instantiated in the prior line.
```
bedrock = fbedrock(model_id = "ai21.j2-ultra-v1")
grounded = Groundedness(groundedness_provider=bedrock_main)
```

Updated:
```
bedrock_provider = fbedrock(model_id = "ai21.j2-ultra-v1")
grounded = Groundedness(groundedness_provider=bedrock_provider)
```

Let me know if this solves the issue on your end. Thanks!


please refer to given code below :

from trulens_eval import Feedback, Select
from trulens_eval.feedback import Groundedness
from trulens_eval import TruChain, Feedback, Bedrock, Tru
from trulens_eval.feedback.provider.bedrock import Bedrock as fbedrock
bedrock = fbedrock(model_id = "ai21.j2-ultra-v1")
grounded = Groundedness(groundedness_provider=bedrock)
f_relevance = Feedback(bedrock.relevance).on_input_output()
# Define a groundedness feedback function
f_groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)
--------------------------------------------------------------------------------------------------------------------
from trulens_eval import TruCustomApp
tru_rag = TruCustomApp(rag,
    app_id = 'RAG_v_1',
    feedbacks = [f_groundedness])  #[f_groundedness, f_qa_relevance, f_context_relevance]
 
---------------------------------------------------------------------------------------------------------------------

import pandas as pd
import time 
total_queries = ["Q1?","Q2?"]
with tru_rag as recording:
    rag.query(f"{total_queries[1]}")
    print('Adding completion now')


rag is a Class with RAG from scratch is implemented 




The error is : 

bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: extraneous key [inputText] is not permitted#: extraneous key [textGenerationConfig] is not permitted, please reformat your input and try again.. Retries remaining=3.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: extraneous key [inputText] is not permitted#: extraneous key [textGenerationConfig] is not permitted, please reformat your input and try again.. Retries remaining=2.

@joshreini1 Are there any updates  on this issue ?, The issue still persists when Im using model_id = "ai21.j2-ultra-v1

However When i tried it using "amazon.titan-text-express-v1" its working fine , I think we need to accommodate the input format type for other aws bedrock model

@budhewarvijay0407 - Working on getting access on my side to ai21 models so I can test this. Agreed with your solution - if you're able to contribute it in the meantime, that would be great too

Sure I will try

Hey @budhewarvijay0407 - I have a fix with this PR: https://github.com/truera/trulens/pull/878

This will be available in the next release (0.23)