Repository: truera/trulens
Issue: [BUG] Inconsistent results between records variable values and dashboard/leaderboard values
Root cause: Resource Mishandling

**Bug Description**
I am building simple RAG application using few pages from wikipedia.
Values for Groundedness, Answer Relevance and Context's Relevance are not the same between ```records``` variable (output of ```tru.get_records_and_feedback(app_ids=[])``` code) and dashboard values.
Also, feedback variable does not contain Context's Relevance feedback function name.

**To Reproduce**
```
from trulens_eval import Tru
from trulens_eval.app import App
from trulens_eval import OpenAI as fOpenAi
from trulens_eval import Feedback
from trulens_eval import TruLlama

tru = Tru()
tru.reset_database()

provider = fOpenAi()

context = App.select_context(query_engine)

# Define a groundedness feedback function
f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons, name="Groundedness")
    .on(context.collect()) # collect context chunks into a list
    .on_output()
)

# Question/answer relevance between overall question and answer.
f_answer_relevance = (
    Feedback(provider.relevance, name="Answer Relevance")
    .on_input_output()
)
# Question/statement relevance between question and each context chunk.
f_context_relevance = (
    Feedback(provider.context_relevance_with_cot_reasons, name="Context's Relevance")
    .on_input()
    .on(context)
    .aggregate(np.mean)
)

tru_query_engine_recorder = TruLlama(query_engine,
    app_id='LlamaIndex_App1',
    feedbacks=[
        f_groundedness, 
        f_answer_relevance, 
        f_context_relevance])

questions = ["question 1", "question 2"]
for question in questions:
    with tru_query_engine_recorder as recording:
        query_engine.query(question)

records, feedback = tru.get_records_and_feedback(app_ids=[])

tru.run_dashboard()
```


**Expected behavior**
Values in leaderboard match values in ```record``` variable.
```feedback``` variable values match all defined feedback function names.

**Relevant Logs/Tracebacks**

Output when feedback functions are initialized:
✅ In Groundedness, input source will be set to __record__.app.query.rets.source_nodes[:].node.text.collect() .
✅ In Groundedness, input statement will be set to __record__.main_output or `Select.RecordOutput` .
✅ In Answer Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .
✅ In Answer Relevance, input response will be set to __record__.main_output or `Select.RecordOutput` .
✅ In Context's Relevance, input question will be set to __record__.main_input or `Select.RecordInput` .
✅ In Context's Relevance, input context will be set to __record__.app.query.rets.source_nodes[:].node.text .

```feedback``` variable does not contain Context's Relevance value:
```
pd.set_option("display.max_colwidth", None)
records[["input", "output"] + feedback]
```
<img width="678" alt="Screenshot_38" src="https://github.com/truera/trulens/assets/89461159/46826e96-3666-4b55-bb3d-42dcee1c1e73">

Values in dashboard/leaderboard are not the same as ```records```:
<img width="657" alt="Screenshot_39" src="https://github.com/truera/trulens/assets/89461159/1fcd9cd9-782b-4059-8820-15459bbdec87">


**Environment:**
 - OS: Windows 10
 - Python Version: 3.9.13
 - TruLens version: 0.29.0
 - llama-index version: 0.10.38

**Additional context**
/


<!-- Greeting -->
@marko-radovic96 Hello there! I'm Dosu, a bot here to help you with bug fixes, questions, and becoming a contributor. Let's tackle this issue together while we wait for a human maintainer.

<!-- Answer -->
The inconsistency in the values for Groundedness, Answer Relevance, and Context's Relevance between the `records` variable and the dashboard/leaderboard in your RAG application is likely due to the different feedback functions being used in the two setups. Specifically, the `feedback` variable is missing the Context's Relevance feedback function name in one of the setups.

In your provided code, the feedback functions are defined as follows:

```python
# Define a groundedness feedback function
f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons, name="Groundedness")
    .on(context.collect())  # collect context chunks into a list
    .on_output()
)
# Question/answer relevance between overall question and answer.
f_answer_relevance = (
    Feedback(provider.relevance, name="Answer Relevance")
    .on_input_output()
)
# Question/statement relevance between question and each context chunk.
f_context_relevance = (
    Feedback(provider.context_relevance_with_cot_reasons, name="Context's Relevance")
    .on_input()
    .on(context)
    .aggregate(np.mean)
)
```

The key difference is in the `f_answer_relevance` feedback function. In your setup, it uses `provider.relevance`, while in another setup, it might use `provider.relevance_with_cot_reasons`. This difference in the feedback function definitions can lead to inconsistent values between the `records` variable and the dashboard/leaderboard.

Additionally, the `feedback` variable is missing the Context's Relevance feedback function name, which can cause discrepancies in the displayed values. The `leaderboard` function in the `trulens_eval` module retrieves and displays feedback values, and any missing or inconsistent feedback function definitions can result in the observed inconsistencies.

To resolve the inconsistency, ensure that the feedback functions are defined consistently across both setups and that the `feedback` variable includes the Context's Relevance feedback function name. The `trulens_eval` module provides a method to create a triad of feedback functions for evaluating context retrieval generation steps, which can help ensure consistency:

```python
def rag_triad(
    provider: mod_base_provider.LLMProvider,
    question: Optional[mod_serial_utils.Lens] = None,
    answer: Optional[mod_serial_utils.Lens] = None,
    context: Optional[mod_serial_utils.Lens] = None
) -> Dict[str, Feedback]:
    """Create a triad of feedback functions for evaluating context retrieval
    generation steps.
    
    If a particular lens is not provided, the relevant selectors will be
    missing. These can be filled in later or the triad can be used for rails
    feedback actions whick fill in the selectors based on specification from
    within colang.

    Args:
        provider: The provider to use for implementing the feedback functions.
    
        question: Selector for the question part.

        answer: Selector for the answer part.

        context: Selector for the context part.
    """

    assert hasattr(
        provider, "relevance"
    ), "Need a provider with the `relevance` feedback function."
    assert hasattr(
        provider, "qs_relevance"
    ), "Need a provider with the `qs_relevance` feedback function."

    are_complete: bool = True

    ret = {}

    for f_imp, f_agg, arg1name, arg1lens, arg2name, arg2lens, f_name in [
        (provider.groundedness_measure_with_cot_reasons, np.mean, "source",
         context.collect(), "statement", answer, "Groundedness"),
        (provider.relevance_with_cot_reasons, np.mean, "prompt", question,
         "response", answer, "Answer Relevance"),
        (provider.context_relevance_with_cot_reasons, np.mean, "question",
         question, "context", context, "Context Relevance")
    ]:
        f = Feedback(f_imp, if_exists=context, name=f_name).aggregate(f_agg)
        if arg1lens is not None:
            f = f.on(**{arg1name: arg1lens})
        else:
            are_complete = False

        if arg2lens is not None:
            f = f.on(**{arg2name: arg2lens})
        else:
            are_complete = False

        ret[f.name] = f

    if not are_complete:
        logger.warning(
            "Some or all RAG triad feedback functions do not have all their selectors set. "
            "This may be ok if they are to be used for colang actions."
        )

    return ret
```

Ensure that the feedback functions are consistently defined and include all necessary selectors to avoid discrepancies <sup>[[1]](https://github.com/truera/trulens/blob/main/trulens_eval/generated_files/all_tools.py)</sup><sup>[[2]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/Leaderboard.py)</sup><sup>[[3]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/feedback.py)</sup>.


<!-- Dosu Comment Footer -->
<details open>
<summary>

#### Details
</summary>

**About Dosu**
This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
</details>

*To continue the conversation, mention @dosu.*


Tried to replace
```
f_answer_relevance = (
    Feedback(provider.relevance, name="Answer Relevance")
    .on_input_output()
)
```
with
```
f_answer_relevance = (
    Feedback(provider.relevance_with_cot_reasons, name="Answer Relevance")
    .on_input_output()
)
```
but still doesn't work.

<!-- Answer -->
To resolve the inconsistency between the `records` variable and the dashboard/leaderboard values for Groundedness, Answer Relevance, and Context's Relevance, and ensure that the `feedback` variable includes the Context's Relevance feedback function name, you need to ensure that the feedback functions are correctly defined and included in the `feedbacks` list when constructing the `TruLlama` instance.

Here is the relevant code snippet that shows how to set up the feedback functions and include them in the `TruLlama`:

```python
import numpy as np
from trulens_eval import Feedback, Select, TruLlama
from trulens_eval.feedback.provider.openai import OpenAI

provider = OpenAI()

# Define a groundedness feedback function
f_groundedness = (
    Feedback(
        provider.groundedness_measure_with_cot_reasons, name="Groundedness"
    ).on(Select.RecordCalls.retrieve.rets.collect()).on_output()
)

# Question/answer relevance between overall question and answer.
f_answer_relevance = (
    Feedback(provider.relevance_with_cot_reasons, name="Answer Relevance").on(
        Select.RecordCalls.retrieve.args.query
    ).on_output()
)

# Context relevance between question and each context chunk.
f_context_relevance = (
    Feedback(
        provider.context_relevance_with_cot_reasons, name="Context Relevance"
    ).on(Select.RecordCalls.retrieve.args.query).on(
        Select.RecordCalls.retrieve.rets
    ).aggregate(np.mean)  # choose a different aggregation method if you wish
)

# Construct the app
tru_query_engine_recorder = TruLlama(
    query_engine,
    app_id='LlamaIndex_App1',
    feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance]
)

# Run the app
questions = ["question 1", "question 2"]
for question in questions:
    with tru_query_engine_recorder as recording:
        query_engine.query(question)

# Retrieve records and feedback
records, feedback = tru.get_records_and_feedback(app_ids=["LlamaIndex_App1"])

# Display the leaderboard
tru.get_leaderboard(app_ids=["LlamaIndex_App1"])

# Run the dashboard
tru.run_dashboard()
```

This code ensures that the `feedback` variable includes the Context's Relevance feedback function name and that the feedback functions are correctly set up and included in the `TruLlama` instance. This should resolve any inconsistencies between the `records` variable and the dashboard/leaderboard values <sup>[[1]](https://github.com/truera/trulens/blob/main/trulens_eval/generated_files/all_tools.py)</sup><sup>[[2]](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/tests/test_cases.py)</sup>.


<!-- Dosu Comment Footer -->
<details>
<summary>

#### Details
</summary>

**About Dosu**
This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
</details>

*To continue the conversation, mention @dosu.*


It looks like the screenshots you have here were taken at different times and the feedback was not finished computing. By default, feedback results are computed asyncronously so as not to slow down your app and may not yet be available whenever you first view the results (via the notebook or dashboard).

If you want to ensure the feedback results are available when you check for them, you can block on their computation.

You can use `as_completed` to wait until they have finished evaluating or use the utility method:

```python
for feedback, feedback_result in rec.wait_for_feedback_results().items():
    print(feedback.name, feedback_result.result)

# See more about wait_for_feedback_results:
# help(rec.wait_for_feedback_results)
```

Hi @joshreini1 ,

Thank you for your response.
When I added `tru_query_engine_recorder.wait_for_feedback_results()` before `records, feedback = tru.get_records_and_feedback(app_ids=["LlamaIndex_App1"])` line, all feedbacks are present in `records` variable.

But, still some values are not the same in `records` variable and in dashboard.
<img width="718" alt="Screenshot_1" src="https://github.com/truera/trulens/assets/89461159/a3e5a46e-2e0f-4bca-b69a-81383417c382">

<img width="555" alt="Screenshot_2" src="https://github.com/truera/trulens/assets/89461159/59d00a01-2b7f-401a-bec1-0ef4e7dd2ff6">

Context relevance for question 5 is NaN in `records` variable, but it is 0.7 in dashboard. Why?
Also, what does that mean when value is NaN? 


@marko-radovic96 when the value is NaN, it means that the feedback has not computed yet (or failed to compute). Other than the discrepancy in NaNs (which I think should go away if this data is loaded from your trulens db (`default.sqlite` by default) at the same time.