Repository: truera/trulens
Issue: Can Groundedness Contextualize and Understand the Semantic Relationship Between User Questions and Algorithm Answer?
Root cause: Validation Gap

I uploaded a document that contains:

`Optimizer: AdamW(lr=0.002, momentum=0.9)`

And I asked the algorithm

`What learning parameter did it use?
`

The reply was

`The learning parameter used in the training process is a learning rate of 0.002`

Which is correct, the algorithm was able to understand that lr is the learning rate of 0.002


However Groundedness returned the following

```
Statement Sentence: The learning parameter used in the training process is a learning rate of,
Supporting Evidence: NOTHING FOUND
Score: 0
```

Is it because of the cutoff it had in the sentence? or does Groundedness does not understand that lr does relate directly to learning parameter in this case?


Another question I asked is:

`Does this learning rate like chicken?`

of which the algorithm said

`I'm sorry, but I couldn't find any information in the provided sources about whether the learning rate used in the training process is related to chicken or not.
`
Which is also correct, 
but Groundedness was zero
```
Statement Sentence: I'm sorry, but I couldn't find any information in the provided sources about whether the learning rate used in the training process is related to chicken or not.,
Supporting Evidence: NOTHING FOUND
```
In my opinion, shouldnt Groundedness understand that indeed the algorithm succesfully grounded its answer by saying their is no answer?


I am using:

```
grounded = Groundedness(groundedness_provider=openai)
grounded.groundedness_measure_with_cot_reasons(prompt, response)
```

Any guidenes will help, thank you

Hey @magedhelmy1 - good question. Groundedness, and by extension GPT-3.5 in this case, is able to semantically understand the context/answer when it makes a judgement.

Were you able to confirm that the source text you mentioned `Optimizer: AdamW(lr=0.002, momentum=0.9)` was successfully retrieved and used in the groundedness evaluation? This information can be seen the trulens dashboard on the evaluations page, after clicking on the record in question.

If the text is being used in the evaluation, I'd recommend testing a more powerful model for evaluation here. It may be the case that GPT-3.5 does not understand the relationship between lr=0.002 and the learning rate, while GPT-4 does.

Changing the algorithm somehow helps but the problem still presists.
I am using the GPT4 for the below.

Steps to reproduce:
1. Upload random document
2. Ask it, `what is the speed of light`?
3. The algorithm should reply something along the lines of 

`The provided sources do not contain any information specifically about the speed of light.` which is True.

Now please check Groundedness

```
Statement Sentence: The provided sources do not contain any information specifically about the speed of light.,
Supporting Evidence: NOTHING FOUND
Score: 0
```


My 2 cents,
The core of the bug is that 'Groundness' fails to recognize the significance of the algorithm not finding any relevant information as valid evidence supporting its response. The absence of information in the provided sources directly supports the algorithm's statement that there is no information about the speed of light.


Expected Behavior:
The correct evaluation should recognize that the algorithm's inability to find relevant information (evidenced by "NOTHING FOUND") directly supports its response. Therefore, instead of a score of 0, the 'Groundness' score should actually be 10, indicating perfect alignment between the lack of found information and the algorithm's statement.


Hey @magedhelmy1 - good point. This is an improvement we can make.

It also seems to be a distinct issue to the issue with the learning rate (where supporting evidence was available). Was that issue solved with GPT-4?

For the learning rate, I did the experiment 5 times, the algorithm failed 1 time to recognize the learning rate and lr are related, but in all instances, Groundedness failed 5 out of 5 times to understand lr and learning rate are related:

Context:
`Optimizer: AdamW(lr=0.002, momentum=0.9) with parameter groups 57 weight (decay=0.0), 64 weight(decay=0.0005), 63 bias(decay=0.0)Image sizes: 500 train, 25 val`

User: `What is the learning rate of the algorithm?`
Algorithm: 
`The learning rate of the algorithm is 0.002. This is specified in the optimizer section of the context you provided: "Optimizer: AdamW(lr=0.002, momentum=0.9)". The "lr" stands for learning rate.`


Truelens:
```
Statement Sentence: The "lr" stands for learning rate.
Supporting Evidence: NOTHING FOUND
Score: 0
```


Hey @magedhelmy1 - I think this TruLens evaluation is reasonable. There is nothing directly in the context that says lr stands for learning rate, which is exactly what the groundedness feedback function is looking for here.

If you'd like to tailor the feedback function for your use case, I'd recommend tweaking it using a custom function. Check out the [docs](https://www.trulens.org/trulens_eval/custom_feedback_functions/) on how to do that.