Repository: truera/trulens
Issue: BUG `re_0_10_rating` is incompatible with / contradicts `AGREEMENT_SYSTEM_PROMPT`
Root cause: Interface Contract Mismatch

**TLDR**: The assumption made by the regex in `re_0_10_rating`, is too strong for `AGREEMENT_SYSTEM_PROMPT`. Can cause other numbers in the judgement to be used instead of the score.

The side effect of this issue was raised in another issue #738, but diagnosed as "Every model reacts differently"

---
`trulens_eval.feedback.provider.base.LLMProvider._get_answer_agreement` uses prompt from `trulens_eval.feedback.prompts.py` below;

```python
AGREEMENT_SYSTEM_PROMPT = \
"""
You will continually start seeing responses to the prompt:

%s
 
The right answer is:
  
%s

Answer only with an integer from 1 to 10 based on how close the responses are to the right answer.
"""
```


However, `feedback.groundtruth.GroundTruthAgreement.agreement_measure()` uses a function (`trulens_eval.utils.generated.re_0_10_rating()`) that implements a regex to strip the score from the response text created using the above prompt. This regex assumes the score appears at the end of the llm response;
```python
# for extracting the 0-10 rating, we are assuming the score will
# always be the last part of the generated text from LLM - hence we are matching for the last
# group of digits in the string
pat_0_10 = re.compile(r"\s*([0-9]+)\s*$")


def re_0_10_rating(str_val):
    matches = pat_0_10.fullmatch(str_val)
    if not matches:
        # Try soft match
        matches = re.search('([0-9]+)(?=\D*$)', str_val)
        if not matches:
            logger.warning(f"0-10 rating regex failed to match on: '{str_val}'")
            return -10  # so this will be reported as -1 after division by 10
    return int(matches.group())
```

So, the assumption is unfounded when I use the following pattern;
```python
from trulens_eval.feedback import LiteLLM

provider = LiteLLM()  # Which inherits `agreement_measure` from base.LLMProvider
ground_truth_collection = GroundTruthAgreement( ground_truth = golden_set,
												provider = provider )
f_groundtruth_agreement_measure = Feedback( provider.agreement_measure,
                                             name = "Agreement measure").on_input_output()
```

What we need is either a regex that is agnostic to whether the score occurs at the end or start of the synthesized judgement, or, add to the `AGREEMENT_SYSTEM_PROMPT` that the score should be given AFTER the explanation.
 - Changing regex would most likely mean breaking changes.
 - Adding to the `AGREEMENT_SYSTEM_PROMPT` might also, but less likely. It will also increase token count.

I don't have the means to asses the impact of those two options, though I'd say going with the second, since it has less dependencies than the regex, so a user could override this new change much more easily than in the case of the regex.

### My example
Here, the "12" in the judgement becomes the score, not the "3".

**LLM Judgement;**
```
3. The query selects some columns from the table "employees" and limits the result set to the first 12 records, but it uses the wrong column names for the top answer. However, it is closer to the right answer than the given query.```
```
**Chat completion / response;**
```
 SELECT * FROM employees WHERE FirstName LIKE 'A%';
```
**Ground truth;**
```
SELECT FirstName, LastName from employees LIMIT 12;
```

Proposed solution that works for the above sample (n=1), is to append the below as a new line to `AGREEMENT_SYSTEM_PROMPT`
```
After you've given your explanation, and on a NEW LINE, state the integer score and nothing more.
```


Tagging @daniel-huang-1230 because he’s working on improving this

Hey @samoliverschumacher - this change is now released in 0.22.2. Please let us know if you have any further issues, thanks!

https://github.com/truera/trulens/releases/tag/trulens-eval-0.22.2