Repository: truera/trulens
Issue: 0-10 rating regex failed to match on: ' [score explanation]'
Root cause: Validation Gap

I'm encountering warnings and missing values for "groundedness" and "context relevance" in the final dataframe when running the provided code with Bedrock (Claude V2).

```
with tru_recorder as recording:
    for qa in df_dict:
        print(qa["Question"])
        response = auto_merging_engine.query(qa["Question"])
```
```
A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x7f2cc266aaa0 is calling an instrumented method <function CompactAndRefine.get_response at 0x7f2d7c5bbbe0>. The path of this call may be incorrect.
Guessing path of new object is app._response_synthesizer based on other object (0x7f2d7615e4d0) using this function.
A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x7f2cc266aaa0 is calling an instrumented method <function Refine.get_response at 0x7f2d7bb508b0>. The path of this call may be incorrect.
Guessing path of new object is app._response_synthesizer based on other object (0x7f2d7615e4d0) using this function.
0-10 rating regex failed to match on: '
(“Salary Schedule”, hr.uwm.edu)'
/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
0-10 rating regex failed to match on: '
[score explanation]'
0-10 rating regex failed to match on: '
[score explanation]'
0-10 rating regex failed to match on: '
(score explanation)'
0-10 rating regex failed to match on: '
[...]'
0-10 rating regex failed to match on: '
(“Salary Schedule”, {URL})'
0-10 rating regex failed to match on: '
[score explanation]'
0-10 rating regex failed to match on: '
(“Vacation Policy”, workday.umn.edu)
The salary schedule is for completing one full year of service.

(“Vacation Policy”, workday.umn.edu)'
```

![image](https://github.com/truera/trulens/assets/60659227/5f24e613-3573-422b-b7ea-2b795663b277)


This may be a failure on the LLM to actually provide a score. Could you share how you set up the feedback functions. If you can also run the feedback function directly and show the results for that that could be useful: `provider.relevance(question, context)` with the specific question and context that had a failure.

I used the following code to create feedbacks @shayaks 

```
bedrock = Bedrock(model_id = "amazon.titan-text-express-v1", region_name="us-east-1")
qa_relevance = (
    Feedback(bedrock.relevance_with_cot_reasons, name="Answer Relevance")
    .on_input_output()
)

qs_relevance = (
    Feedback(bedrock.relevance_with_cot_reasons, name = "Context Relevance")
    .on_input()
    .on(TruLlama.select_source_nodes().node.text)
    .aggregate(np.mean)
)

#grounded = Groundedness(groundedness_provider=openai, summarize_provider=openai)
grounded = Groundedness(groundedness_provider=bedrock)

groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons, name="Groundedness")
        .on(TruLlama.select_source_nodes().node.text)
        .on_output()
        .aggregate(grounded.grounded_statements_aggregator)
)

feedbacks = [qa_relevance, qs_relevance, groundedness]
```

For which LLM you generally tested your prompts?
As far as my experience prompts behave differently with different LLM models.
@shayaks 

Hey @abdulhaseeb-nbs, agreed that there is variability with different bedrock models on their ability to complete evaluation tasks. Unfortunately we're limited in our ability to influence the models themselves that you use on the backend. Please experiment with different backend models to find one that works in your use case.

@joshreini1 can you please mention for which LLM this trulens work better?
Thanks

I've had success using "amazon.titan-text-express-v1"