Repository: truera/trulens
Issue: [BUG] groundedness_measure_with_cot_reasons_consider_answerability raises when evaluating abstention
Root cause: Validation Gap

**Bug Description**
The metric `groundedness_measure_with_cot_reasons_consider_answerability` raises an error when evaluating abstention.

**To Reproduce**
```python
import boto3
from trulens.providers.bedrock import Bedrock

client = boto3.client(service_name="bedrock-runtime")
bedrock = Bedrock(
    model_id="anthropic.claude-3-haiku-20240307-v1:0",
    client=client,
)

bedrock.groundedness_measure_with_cot_reasons_consider_answerability(
    question="How many stomachs does a cow have?",
    source="A cow is a mammal. It lives on a farm and has 4 stomachs.",
    statement="The source contains now answer to the question."
)
```

**Expected behavior**
no error

**Relevant Logs/Tracebacks**
Rating must be in [0, 3].
Traceback (most recent call last):
  File "/home/user/projects/my_project/scripts/genai_metrics.py", line 23, in <module>
    bedrock.groundedness_measure_with_cot_reasons_consider_answerability(
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/feedback/llm_provider.py", line 1756, in groundedness_measure_with_cot_reasons_consider_answerability
    results.append(future.result())
                   ^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/core/utils/python.py", line 561, in _future_target_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/feedback/llm_provider.py", line 1727, in evaluate_hypothesis
    abstention_score = evaluate_abstention(hypothesis)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/feedback/llm_provider.py", line 1696, in evaluate_abstention
    score = self.generate_score(
            ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/providers/bedrock/provider.py", line 217, in generate_score
    return (re_configured_rating(response) - min_score_val) / (
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/feedback/generated.py", line 86, in re_configured_rating
    raise ParseError(f"{min_score_val}-{max_score_val} rating", s)
trulens.feedback.generated.ParseError: Tried to find 0-3 rating in
  10

**Environment:**

- OS: [e.g. MacOS, Windows]: Ubuntu 22.04
- Python Version: 3.12.5
- TruLens version: 1.0.4


__Update__: Other metrics have related issues:

```python
bedrock.coherence_with_cot_reasons(
    text="A cow has 4 stomachs and has a diet that relies primarily on grazing.",
)
```

raises

```
Rating must be in [0, 3].
Traceback (most recent call last):
  File "/home/user/projects/my_project/scripts/genai_metrics.py", line 29, in <module>
    answer: tuple[float, dict[str, str]] = bedrock.coherence_with_cot_reasons(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/feedback/llm_provider.py", line 908, in coherence_with_cot_reasons
    return self._langchain_evaluate_with_cot_reasons(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/feedback/llm_provider.py", line 782, in _langchain_evaluate_with_cot_reasons
    return self.generate_score_and_reasons(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/providers/bedrock/provider.py", line 265, in generate_score_and_reasons
    re_configured_rating(
  File "/home/user/miniconda3/envs/my-env/lib/python3.12/site-packages/trulens/feedback/generated.py", line 86, in re_configured_rating
    raise ParseError(f"{min_score_val}-{max_score_val} rating", s)
trulens.feedback.generated.ParseError: Tried to find 0-3 rating in
  Score: 10
```

further affected functions (I'll update the list if I find further ones):

* coherence
* harmfulness_with_cot_reasons
* harmfulness


Hello, 

Just wanted to add some other metrics and context. For my part I can very easily use the Honesty feedbacks but all other have some issues and return out of range values. I saw that using the param, max_val_score in an example setup works as fix:
`azopenai.sentiment_with_cot_reasons(
    text = "Germany is ouest-european country, located between France and Poland" , max_score_val= 10
) `
Returns `(1.0,
 {'reason': "Criteria: Accuracy of statement regarding Germany's location in Europe.\nSupporting Evidence: Germany is indeed a country located in Europe. It is situated in the western part of the continent and shares borders with several other European countries, including France and Poland. Therefore, the statement that Germany is a west-European country located between France and Poland is accurate.\n\nScore: 10"})` 

But when trying to reproduce this fix with Feedbacks of my app like this: 
`from trulens.core import Feedback`
`f_coherence = Feedback(
    azopenai.coherence_with_cot_reasons, name="Coherence", max_score_val = 20
).on_output()`
`f_input_sentiment = Feedback(
    azopenai.sentiment_with_cot_reasons, name="Input Sentiment", max_score_val = 20
).on_input()`
`f_output_sentiment = Feedback(
    azopenai.sentiment_with_cot_reasons, name="Output Sentiment", max_score_val = 20
).on_output()`


`helpful_feedbacks = [
    f_coherence,
    f_input_sentiment,
    f_output_sentiment
    ]`
`tru_query_engine_recorder = TruLlama(
    query_engine,
    app_name="Test_full_feedbacks",
    feedbacks= honesty_feedbacks + helpful_feedbacks
)`
`query = "What is the purpose of YC?"`
`with tru_query_engine_recorder as recording:`
    `answer = query_engine.query(query)`
    
`record = recording.get()`
`print("query was:", query)`
`print("answer was:", answer)`
`print(answer.get_formatted_sources())`
`for feedback, result in record.wait_for_feedback_results().items():`
    `print(feedback.name, `result.result)`

I end up with this print: 
query was: What is the purpose of YC?
answer was: The purpose of YC is to fund and help startups by providing seed funding, mentorship, and resources to young and technical founders. YC also aims to create a community of startups that can help and learn from each other. The batch model is used to fund a group of startups all at once, twice a year, and then to spend three months focusing intensively on trying to help them.
> Source (Doc id: 819f1509-24a9-4ce0-b702-3e2338d951b0): When I was dealing with some urgent problem during YC, there
was about a 60% chance it had to do ...

> Source (Doc id: 91d8923d-7907-4882-ae01-e3562de50dd3): So
we just made what seemed like the obvious choices, and some of the things we
did turned out to...

> Source (Doc id: daf70518-df23-4c52-a112-41859571035c): We invited about 20 of the 225 groups to interview in person, and from those
we picked 8 to fund....

> Source (Doc id: 5a2f2eeb-f4bf-4d3a-86ea-e2ce8198d6c3): So before we could launch
publicly we had to launch privately, in the sense of recruiting an init...

> Source (Doc id: 533c3848-107b-4f0d-b3ef-f7a06eefc688): I asked Jessica if she wanted to be president, but she didn't, so we decided
we'd try to recruit ...

> Source (Doc id: e7f4726f-0819-4001-b74c-a133d635f90f): I also worked on spam
filters, and did some more painting. I used to have dinners for a group of
...

> Source (Doc id: e6c6949a-872b-403f-826c-91907c7c2c14): [16] She reports that they were all different shapes and sizes, because there
was a run on air co...
WARNI [trulens.feedback.generated] Rating must be in [0, 3].
WARNI [trulens.feedback.generated] Rating must be in [0, 3].
WARNI [trulens.feedback.generated] Rating must be in [0, 3].
WARNI [trulens.feedback.generated] Rating must be in [0, 3].
Groundedness 1.0
Answer Relevance 1.0
Context Relevance 0.3333333333333333
cosine_distance 0.11731985395227429
Coherence None
Input Sentiment None
Output Sentiment None

We can see that Honesty Feedbacks get computed and returned but the Helpful ones got left out because of the error discussed here! 
I'm on trulens-apps-llamaindex  == 1.0.7 trulens == 1.0.7 and trulens-feedbacks== 1.0.7
Hope this can help you find the issue if there is one, thanks a lot for everything!

Thanks for reporting! We'll fix this asap

Hey @drooms-sandrus @ClementP2 - we've released a patch for this (1.0.8). Please upgrade your trulens packages and let us know if you have any persistent issues related. Thanks!

https://github.com/truera/trulens/releases/tag/trulens-1.0.8

Hello sorry for the delay, 
From my part I still have one feedback that prints the warning (sometimes two) and still can't compute some feedbacks, like input_sentiment, output_sentiment, Hate/Threatening, Violent....
I believe I'm using the appropriate code to wait for their computation:
`query = "Can you tell me about the Y Combinator?"
with tru_query_engine_recorder as recording:
    answer = query_engine.query(query)
    
record = recording.get()
print("query was:", query)
print("answer was:", answer)
print(answer.get_formatted_sources())
for feedback, result in record.wait_for_feedback_results().items():
    print(feedback.name, result.result)`

I'm using GPT-3.5-turbo, does it suits such feedbacks? 
Andcould you confirm that we can use sentiment feedback (and other feedbacks such as violent, hate...) on input only, we aim to better understand if a Violent response from our chatbot would come in response of a violent question in input! 

Also I saw a strange behavior on the maliciousness feedback, where even with the higher_is_better parameter to False, still outputs a 1 when finding not ill intent in the output. Others don't perform that way, here is the definition of the function:
`f_maliciousness = Feedback(
    azopenai.maliciousness_with_cot_reasons,
    name="Maliciousness",
    higher_is_better=False,
).on_output()`
![image](https://github.com/user-attachments/assets/b8760e8c-b04e-4f9a-a82b-424b246dce0c)


Can you switch to 'gpt-4o-mini' for providing feedbacks?

It didn't solve the issue : 
![image](https://github.com/user-attachments/assets/7903c4af-a2da-41ed-bc26-5cf5f9b60427)
****

On the pip side I'm now using: 
`trulens-core == 1.0.10,trulens-dashboard==1.0.10, trulens-providers-openai==1.0.10,trulens-feedback==1.0.10`

Hello @ClementP2 seems like for malicious content, gpt-4o models performs the best. Other models including gpt-4o-mini are not deterministic in my local tests for your inputs.  Can you try with gpt-4o once and let us know if everything works as expected?

@ClementP2 closing this for now, we will reopen if this issue still persists on your end.