Repository: explodinggradients/ragas
Issue: When i use the context recall metric and other retrieval metrics on my own embedding model, should I index the docs and renew the context retrieved by my model?
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Context**
I will use my own retrieval and rerank model, such fine-tuned bge-embedding and bge-reranker. And I want to test it on "explodinggradients/fiqa - ragas_eval", maybe i will expand it on my own dataset too.

**Your Question**
1. When i use the context recall metric and other retrieval metrics on my own embedding model, should I index the docs and renew the context retrieved by my model?
2. In the dataset, is there any question's answer depends on the context? Then if I need to renew the context, should I renew the ground truth too?







<!-- Greeting -->
@gcy0926 Hello there! I'm a bot designed to help you with bugs, questions, and contributions. I'm here to assist you while you wait for a human maintainer.

<!-- Answer -->
1. Yes, you should index the documents and renew the context retrieved by your model when using the context recall metric and other retrieval metrics on your own embedding model <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md)</sup>.

2. Yes, there are questions in the "explodinggradients/fiqa - ragas_eval" dataset whose answers depend on the context. Therefore, if you renew the context, you should also renew the ground truth <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


> @gcy0926 Hello there! I'm a bot designed to help you with bugs, questions, and contributions. I'm here to assist you while you wait for a human maintainer.
> 
> 1. Yes, you should index the documents and renew the context retrieved by your model when using the context recall metric and other retrieval metrics on your own embedding model [[1]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md).
> 2. Yes, there are questions in the "explodinggradients/fiqa - ragas_eval" dataset whose answers depend on the context. Therefore, if you renew the context, you should also renew the ground truth [[1]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md).
> 
> _To continue talking to [Dosu](https://dosu.dev), mention @dosu._

what's your suggestion on how to renew ground truth answer? using a larger LLM?

> @gcy0926 Hello there! I'm a bot designed to help you with bugs, questions, and contributions. I'm here to assist you while you wait for a human maintainer.
> 
> 1. Yes, you should index the documents and renew the context retrieved by your model when using the context recall metric and other retrieval metrics on your own embedding model [[1]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md).
> 2. Yes, there are questions in the "explodinggradients/fiqa - ragas_eval" dataset whose answers depend on the context. Therefore, if you renew the context, you should also renew the ground truth [[1]](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md).
> 
> _To continue talking to [Dosu](https://dosu.dev), mention @dosu._

And another question, is the source doc for  explodinggradients/fiqa ragas_eval in the corpus subset? which is 57.6k rows.

<!-- Answer -->
To renew ground truth answers, using a larger LLM is indeed a good approach. The `ragas` framework supports this by allowing you to replace the default OpenAI models with custom LLMs and embeddings. You can pass custom LLMs and embeddings when instantiating the `TestsetGenerator` and use them in the evaluation process. Here’s an example:

```python
from ragas.testset.generator import TestsetGenerator

# Initialize custom LLM and embeddings
from langchain_together import Together
from langchain_together.embeddings import TogetherEmbeddings

together_key = "<your-key-here>"

embeddings = TogetherEmbeddings(model="togethercomputer/m2-bert-80M-8k-retrieval")

together_completion = Together(
    model="NousResearch/Nous-Hermes-2-Mixtral-8x7B-SFT",
    temperature=0.7,
    max_tokens=4000,
    top_k=1,
    together_api_key=together_key
)

# Use custom LLM and embeddings in TestsetGenerator
generator = TestsetGenerator.from_langchain(
    generator_llm=together_completion,
    critic_llm=together_completion,
    embeddings=embeddings,
)
```

This setup allows you to leverage larger or more specialized LLMs for generating ground truth answers, ensuring the quality of your RAG pipeline <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customisations/bring-your-own-llm-or-embs.md)</sup>.

Regarding your second question, to verify if the source documents for the "explodinggradients/fiqa - ragas_eval" dataset are included in the corpus subset of 57.6k rows, you can follow these steps:

1. **Load the Corpus Subset**: Load the corpus subset of 57.6k rows into a DataFrame.
2. **Load the Source Documents**: Load the source documents from the "explodinggradients/fiqa - ragas_eval" dataset.
3. **Check for Inclusion**: Check if the source documents are included in the corpus subset by comparing the content or identifiers of the documents.

Here is a sample code to perform these steps:

```python
import pandas as pd

# Load the corpus subset of 57.6k rows
corpus_df = pd.read_csv('path_to_corpus_subset.csv')

# Load the source documents from the "explodinggradients/fiqa - ragas_eval" dataset
source_documents = pd.read_csv('path_to_source_documents.csv')

# Check for inclusion
included = source_documents['document_id'].isin(corpus_df['document_id'])

# Verify if all source documents are included
all_included = included.all()

print(f"All source documents are included in the corpus subset: {all_included}")
```

Replace `'path_to_corpus_subset.csv'` and `'path_to_source_documents.csv'` with the actual paths to your datasets. The `document_id` should be replaced with the appropriate column that uniquely identifies each document in your datasets <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/arize.ipynb)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


It seems the issue was answered, closing this now.