Repository: truera/trulens
Issue: [BUG] GroundTruthAgreement is not working with Litellm
Root cause: Documentation Deficiency

**Bug Description**
I am using ground truth agreement feedback function, and the when I print the result for that feedback, it is none. all other feedbacks work okay though.
<img width="212" alt="image" src="https://github.com/user-attachments/assets/c472e77d-a78a-4d27-b8cc-7755316bdc2e">


**To Reproduce**
```py
    from trulens.providers.litellm import LiteLLM
    from trulens.core import TruSession, Feedback
    from trulens.feedback import GroundTruthAgreement
    from trulens.apps.llamaindex import TruLlama

    trulens_provider = Litellm(model)
    qa_dataset = [{"query": "Who are you?", "response": "I am an ai Chatbot"}]
    f_groundtruth = Feedback(
        GroundTruthAgreement(
            ground_truth=qa_dataset, 
            provider=trulens_provider
        ).agreement_measure, name="Answer Correctness"
    ).on_input_output()
   
    tru_recorder = TruLlama(
        query_engine,
        app_name="Naive RAG Eval Llama",
        app_version="1.0",
        feedbacks=[f_groundtruth],
    )

    with tru_recorder as recording:
            query_engine.query(qa_dataset[0]["query"]);
    
    rec = recording.get();
    for feedback, feedback_result in rec.wait_for_feedback_results().items():
            print(f"{feedback.name}: {feedback_result.result}")
```


**Expected behavior**
groundtruth feedback should provide the score for answer correctness in the result.


**Environment:**

- OS: MacOS
- Python Version: 3.12
- TruLens version: 1.2.7


Hi @Arslan-Soomro, 

could you provide the complete code snippet including the implementation of `query_engine` if possible? 

@sfc-gh-dhuang I have included a sample from the dataset and also the full code snippet. 
I have done nothing special with the query engine, it is the default provided by llama index.
I also tried doing ```f_groundtruth('Who is the captain of CSK team', 'MS Dhoni')```, I printed the response onto the console, and nothing was printed. so It returned nothing.

**Dataset**
```csv
query,response
What is Lineal consanguinity?,"Lineal consanguinity is that which subsists between two persons, one of whom is descended in a direct line from the other, as between a man and Ins father, grandfather and great-grandfather, and so upwards in the direct ascending line ; or between a man and his son, grandson, great-grandson and so downwards in the direct descending line.(2) Every generation constitutes a degree, either a sending or descending.(3) A persons father is related to him in the first degree, and so likewise is his son ; his grandfather and grandson in the second degree ; his great-grandfather and great-grandson in the third degree, and so on."
```

**Code Snippet**
```py
import litellm
from utils.llama_utils import init_settings
from utils.kb_llama_utils import load_knowledge_base, create_knowledge_base, is_knowledge_base
from trulens.providers.litellm import LiteLLM
from trulens.core import TruSession, Feedback
from trulens.feedback import GroundTruthAgreement
from trulens.apps.llamaindex import TruLlama
from trulens.dashboard import run_dashboard
from utils.tru_utils import delete_tru_app
import numpy as np
from utils.file_utils import load_csv
from utils.globals import APP_NAME

trulens_provider = LiteLLM(
    model_engine="claude-3-5-sonnet-20240620")

# litellm.set_verbose = True # Enable for logging LLM Calls

# KB Details
KB_ID = "succession_act" # Unique ID for the knowledge base
KB_SOURCE_PATH = "docs/The-Succession-Act-1925.pdf" # Path to the file that knowledge base is created from
KB_TESTSET_PATH = "datasets/Succession_Act_Dataset.csv" # Path to the dataset that is used for testing
KB_DB_PROVIDER = "faiss" # Database provider for the knowledge base

# Tru APP Details
APP_VERSION = "Llama Naive"

def main():
    qa_dataset = load_csv(KB_TESTSET_PATH)
    print(f"\nQA Pair Sample: {qa_dataset[0]}\n")

    init_settings()
    index = None

    if is_knowledge_base(KB_ID, KB_DB_PROVIDER):
        # Load the knowledge base
        print(f"📖 Knowledge base with ID {KB_ID} exists, Loadig...")
        index = load_knowledge_base(KB_ID, KB_DB_PROVIDER)
    else:
        # Create the knowledge base
        print("📖 Creating knowledge base...")
        index = create_knowledge_base(
            [KB_SOURCE_PATH], KB_DB_PROVIDER, KB_ID)

    query_engine = index.as_query_engine()

    # Trulens
    session = TruSession()
    delete_tru_app(session, APP_NAME, APP_VERSION);

    context = TruLlama.select_context(query_engine)

    # Feedback Functions

    # Is the answer relevant to the question?
    f_answer_relevance = (Feedback(
        trulens_provider.relevance_with_cot_reasons, name="Answer Relevance"
    ).on_input().on_output())

    # How good is the retrieval?
    f_context_relevance = (Feedback(
        trulens_provider.context_relevance_with_cot_reasons, name="Context Relevance"
    ).on_input().on(context).aggregate(np.mean))

    # Is the response supported by the context?
    f_groundedness = (Feedback(
        trulens_provider.groundedness_measure_with_cot_reasons, name="Groundedness"
        # .collect() puts all the context chunks into a list
    ).on(context.collect()).on_output())

    # how well does the response match the ground truth?
    # Requires QA Pairs, in the shape [{ query, expected_response }]
    # Ground truth doesn't work
    f_groundtruth = (Feedback((GroundTruthAgreement(qa_dataset,trulens_provider)).agreement_measure, name="Answer Correctness").on_input_output())
    
    # Recording feedback
    tru_recorder = TruLlama(
        query_engine,
        app_name=APP_NAME,
        app_version=APP_VERSION,
        feedbacks=[f_context_relevance, f_groundedness,
                   f_answer_relevance, f_groundtruth],
    )

    

    # Go through each question in the dataset and record the feedback
    for (i, question) in enumerate([row["query"] for row in qa_dataset]):
        with tru_recorder as recording:
            query_engine.query(question)
        rec = recording.get()
        print(f"\nRecord {i+1}: {question}")
        for feedback, feedback_result in rec.wait_for_feedback_results().items():
            print(f"{feedback.name}: {feedback_result.result}")

    print(f"\nTrulens Leaderboard: {session.get_leaderboard()}\n")

    run_dashboard(session, port=50570)


main()

```

@Arslan-Soomro 

I see. I think the issue lies in how the ground truth dataset `qa_dataset` is initialized. 

`GroundTruthAgreement` expects both `"query"` and `"expected_response`" as keys in each row / question-answer pair


Can you try rename / transform the column name from `response` to `expected_response` when loading the csv file?


For code reference, see https://www.trulens.org/getting_started/quickstarts/groundtruth_evals/#initialize-feedback-functions


lmk if you could confirm this is indeed the case. Regardlessly, we should update the doc to make sure it's clear the expected key names matter, and thanks for reporting the issue!


Another place to check more explicit error messages will be to go into the `trulens_feedbacks` table in the datastore backing TruLens. And since you seem to be using the default `session = TruSession()`, this table will very likely just be in the default sqlite file called `default.sqlite`



@sfc-gh-dhuang the documentation around ground truth is indeed confusing. I have tried both `response` and `expected_response`, the result remains same.

Hi @Arslan-Soomro,

My apology for the delayed response.

I ran the code you provided and was not able to reproduce the issue. The only thing I did differently was that I was using a local dictionary containing ground truth dataset, instead of reading from the CSV.

I notice you mentioned the sanity check you did - 
"""I also tried doing f_groundtruth('Who is the captain of CSK team', 'MS Dhoni'), I printed the response onto the console, and nothing was printed. so It returned nothing."""

so the minimal reproducible / debugging step will be 

1. create a gt dataset with `query` and `expected_response` keys

```
golden_set = [
    {
        "query": "who invented the lightbulb?",
        "expected_response": "Thomas Edison",
    },
    {
        "query": "Who is the captain of CSK team",
        "expected_response": "MS Dhoni",
    },
]
```

2. define feedback function with the gt dataset:
```
f_groundtruth = Feedback((GroundTruthAgreement(golden_set, OpenAI())).agreement_measure, name="Answer Correctness").on_input_output()
```

3. run below and you should get a response that looks like `(0.9 , {'ground_truth_response': 'MS Dhoni'})`

```
f_groundtruth('Who is the captain of CSK team', 'The captain's name is MS Dhoni') # this should give you a high score 
```
Could you help verify if the above minimal example works in your setting?