Repository: explodinggradients/ragas
Issue: using the embeddings
Root cause: Documentation Deficiency

I have been working with RAGAS for a while, I have a special case that needs an opinion for this.
I already have a dataset that contains the embeddings for context, answer, and the question. and I noticed that the evaluate function uses textual data for running the evaluation is there any way that I can use the embeddings instead of textual data, if I customize the evaluate function then I have to also customize the metrics that I want to use.
if I am correct it is gonna be too time-consuming and I do not want to pay for an API key to generate the embeddings that I already have.


<!-- Answer -->
You can use embeddings instead of textual data in the RAGAS `evaluate` function without extensive customization. You can pass your precomputed embeddings directly to the `evaluate` function using the `embeddings` parameter. This allows you to bypass the need for generating new embeddings or using an API key.

Here's how you can do it:

1. **Prepare Your Precomputed Embeddings**: Ensure your embeddings are structured in a way that the `evaluate` function can use. This typically involves implementing a class that mimics the expected interface, such as having methods like `embed_documents` and `embed_query`.

2. **Pass Precomputed Embeddings to Evaluate**: When calling the `evaluate` function, pass your precomputed embeddings directly:

   ```python
   # Assuming `precomputed_embeddings` is an instance of your custom embeddings class
   evaluation_report = evaluate(ragas_data, metrics=metrics, embeddings=precomputed_embeddings)
   ```

By doing this, the `evaluate` function will use your provided embeddings and will not invoke the default embedding generation process, thus avoiding recomputation of embeddings <sup>[[1]](https://github.com/explodinggradients/ragas/issues/1400)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/evaluation.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu.
no you are wrong do not give me wrong answer

<!-- Answer -->
It looks like this issue needs your specialized knowledge, @jjmachan, could you please step in and provide your guidance?


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


hey @amin-kh96 that is a very interesting usecase!

while you don't have to customize the metrics, you will have to write your own https://docs.ragas.io/en/stable/references/embeddings/#ragas.embeddings.BaseRagasEmbeddings

this means each time ragas tried to compute the embedding for something it will hit you're vector store instead of the cloud. You can also write it so that it will compute the embedding if none is found.

I can help you implement this too but this should work, let me know if you have any doubts? 
cheers 🙂 

@jjmachan Thanks but what about the evaluate function that takes textual data only? what can I do for that? 
I already created baseragsembeddings and baseragasLLM to bypass the error for APIkey.

@jjmachan
This is my code. I created the subclass of baseragsembeddings and basaragsLLM.
but the problem now is that I already paid for the embeddings to be generated but the evaluate function force me to use textual data. please take a look at my code.
```py
import json
import numpy as np
from datasets import Dataset
import typing as t
import torch

import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.metrics.pairwise import cosine_similarity
from scipy.spatial.distance import euclidean

from langchain_core.outputs import LLMResult, Generation
from langchain_core.outputs import LLMResult
from transformers import AutoModel, AutoTokenizer


from ragas.llms.prompt import PromptValue
from ragas.embeddings import BaseRagasEmbeddings
from ragas.metrics import context_utilization
from ragas.llms import BaseRagasLLM
from ragas.llms.prompt import PromptValue
from langchain_core.outputs import LLMResult, Generation
from ragas import evaluate

# Load the ground truth data
file_path = 'C:\\Users\\Amin\\git\\ragas-prototype\\src\\assets\\GT.json'
with open(file_path) as f:
    ground_truth_data = json.load(f)

# Load the question and the answer and the chunks
file_path = 'C:\\Users\\Amin\\git\\ragas-prototype\\src\\assets\\user_llm_interaction_embeddings_c1521dd5_b819_4241_b3a4_3e5c1388037c.json'
with open(file_path) as f:
    llm = json.load(f)

# Initialize an empty list to hold the new dataset
data_set = []

# Iterate through the list and combine every two dictionaries
for i in range(0, len(llm), 2):
    combined_dict = {
        "text_vector_1": llm[i].get("text_vector", []),
        "text_vector_2": llm[i + 1].get("text_vector", []),
        'chunks': llm[i + 1].get('chunks', [])
    }
    data_set.append(combined_dict)

def map_chunks(data_set, ground_truth_data):
    for item in data_set:  # Iterate over each dictionary in data_set
        c = []  # Reset c for each item
        for chunk_id in item['chunks']:  # Loop through 'chunks' in the current dictionary
            for element in ground_truth_data:  # Loop through ground_truth_data
                if element['id'] == chunk_id:  # Match chunk_id with element's id
                    c.append(element['text_vector'])  # Append the matching text_vector to c
        item['chunks'] = c  # Replace the original 'chunks' (ids) with the mapped text_vector values

    return data_set  # Return the updated data_set

data_set = map_chunks(data_set, ground_truth_data)

# Assuming data_set is a list of dictionaries
ragas_data = [
    {
        "question": entry["text_vector_1"],  # Assuming this is a list of strings
        "answer": entry["text_vector_2"],  # Assuming this is a list of strings
        "contexts": entry["chunks"]  # Assuming this is a list of lists of strings
    }
    for entry in data_set
]

# Create the required structure that  structures the data for the Hugging Face Dataset creation.
formatted_data = {
    "question": [entry["question"] for entry in ragas_data],
   "contexts": [entry["contexts"] for entry in ragas_data],
    "answer": [entry["answer"] for entry in ragas_data]
}


# Define the column_map to match custom columns to expected ones
column_map = {
    "question": "question",  # Match to the correct column
    "answer": "answer",      # Match to the correct column
    "contexts": "contexts"   # Match to the correct column
}

# Create a Dataset using the Hugging Face `datasets` library
ragas_dataset = Dataset.from_dict(formatted_data)

model_name = 'distilbert-base-uncased'

class CustomHuggingFaceRagasEmbeddings(BaseRagasEmbeddings):
    def __init__(self, model_name: str, custom_embeddings: list = None):
        """
        Initialize the Custom Hugging Face Ragas Embeddings with the specified model and custom embeddings.
        
        Parameters:
            model_name (str): The name of the Hugging Face model to use (e.g., 'distilbert-base-uncased').
            custom_embeddings (list): A list of pre-computed custom embeddings (optional).
        """
        model_name = 'distilbert-base-uncased'
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModel.from_pretrained(model_name)
        self.custom_embeddings = custom_embeddings  # Store the custom embeddings

    def embed_documents(self, texts: list) -> np.ndarray:
        """
        Generate embeddings for a list of documents.
        
        Parameters:
            texts (list): A list of documents to embed.

        Returns:
            np.ndarray: An array of embeddings for the documents.
        """
        if self.custom_embeddings is not None:
            # If custom embeddings are provided, return those instead
            return np.array(self.custom_embeddings)
        
        # Generate new embeddings using the model if no custom embeddings are available
        inputs = self.tokenizer(texts, return_tensors='pt', padding=True, truncation=True)

        with torch.no_grad():
            outputs = self.model(**inputs)

        # Use the pooled output or the CLS token as the embedding
        embeddings = outputs.last_hidden_state[:, 0, :]  # CLS token for sentence embedding
        return embeddings.numpy()  # Convert to NumPy array

    def embed_query(self, query: str) -> np.ndarray:
        """
        Generate an embedding for a single query.
        
        Parameters:
            query (str): The query to embed.

        Returns:
            np.ndarray: The embedding for the query.
        """
        # If custom embeddings are provided, generate embedding based on those
        if self.custom_embeddings is not None:
            # You might want to handle how to relate the query to your custom embeddings
            raise NotImplementedError("Custom query embeddings are not supported with provided custom embeddings.")
        
        # Generate a new embedding using the model
        inputs = self.tokenizer(query, return_tensors='pt', padding=True, truncation=True)

        with torch.no_grad():
            outputs = self.model(**inputs)

        # Use the pooled output or the CLS token as the embedding
        embedding = outputs.last_hidden_state[:, 0, :]  # CLS token for single query embedding
        return embedding.numpy()  # Convert to NumPy array
    
# Initialize the custom embeddings class
custom_embeddings = CustomHuggingFaceRagasEmbeddings(ragas_dataset)


ragas_embeddings = CustomHuggingFaceRagasEmbeddings(model_name=model_name, custom_embeddings=custom_embeddings)

# Define the custom LLM class
class CustomRagasLLM(BaseRagasLLM):
    def __init__(self, api_key: str = None):
        """
        Initialize the custom LLM, optionally using an API key if necessary.
        """
        self.api_key = api_key

    def _call(self, prompt: str) -> str:
        """
        Process the prompt and return a result. This can be customized to
        use a local model or perform any required logic.
        """
        if not self.api_key:
            return f"Processed: {prompt} (without API key)"
        else:
            # Handle LLM response if using an API
            return f"Processed: {prompt} (with API key: {self.api_key})"
       
    def generate_text(
        self,
        prompt: PromptValue,
        n: int = 1,
        temperature: float = 1e-8,
        stop: t.Optional[t.List[str]] = None,
        callbacks: t.List = []
    ) -> LLMResult:
        # Synchronous generation logic
        text = self._call(prompt)
        return LLMResult(generations=[Generation(text=text)])
    async def agenerate_text(
        self,
        prompt: PromptValue,
        n: int = 1,
        temperature: float = 1e-8,
        stop: t.Optional[t.List[str]] = None,
        callbacks: t.List = []
    ) -> LLMResult:
        """
        Asynchronous method to generate text. This should allow for async processing.
        """
        # Simulate an asynchronous call, here we directly call the sync method for now
        text =await self._call(prompt)
        return LLMResult(generations=[Generation(text=text)])     
    
    
custom_llm =CustomRagasLLM(api_key=None) 
```

please note that i needed the result so temporarily I used the euclidian distance and cosine similarity for evaluation. just ignore that part. I wanna use evaluate function with the dataset that consists of embeddings for question, answer, and context.

@amin-kh96 I don't understand what the problem is - please make it clear and short

> Thanks but what about the evaluate function that takes textual data only? what can I do for that?

we only support textual data, so I don't understand what the question is. Do put a bit more effort into writing the comments here so that we can be more effective in helping you. I hope you understand 🙂 

@jjmachan
 I already have the embeddings(for question, context, and answer). I want to use  RAGAS with the embeddings that I have. but my problem is that the "evaluate" function forces me to use str format (which means textual data).
do you have any solution that lets the "evaluate" function support embeddings rather than textual data?
if it is not possible what approach do you suggest to me?

@amin-kh96 that approach will not work

what will work is this 

> hey @amin-kh96 that is a very interesting usecase!
> 
> while you don't have to customize the metrics, you will have to write your own https://docs.ragas.io/en/stable/references/embeddings/#ragas.embeddings.BaseRagasEmbeddings
> 
> this means each time ragas tried to compute the embedding for something it will hit you're vector store instead of the cloud. You can also write it so that it will compute the embedding if none is found.
> 
> I can help you implement this too but this should work, let me know if you have any doubts? cheers 🙂

implement a custom `BaseRagasEmbedding` so that when evaluate() calls `embed_text` it checks the embedding for that text from where ever you have and returns it. if not found it will fetch from openai or anyother service

> 报价回复能不能调用xinference起得嵌入模型得接口呢？



@dosu
I have creatted a baseragasllm. what should I do with this error?
Exception has occurred: AttributeError
'PromptValue' object has no attribute 'text'
  File "C:\Users\Amin\git\ragas-prototype\src\rag_eva.py", line 252, in generate_text
    prompt_text = prompt.text if isinstance(prompt, PromptValue) else str(prompt)
                  ^^^^^^^^^^^
AttributeError: 'PromptValue' object has no attribute 'text'


<!-- Answer -->
To resolve the `AttributeError: 'PromptValue' object has no attribute 'text'`, you need to ensure that the `PromptValue` object has the correct attribute being accessed in your `generate_text` method. In your code, you are trying to access `prompt.text`, but it seems that the correct attribute might be `prompt.prompt_str` based on the Ragas library's usage.

Here's what you can do:

1. **Check the `PromptValue` Class**: Verify if the `PromptValue` class has an attribute named `prompt_str`. If it does, you should use `prompt.prompt_str` instead of `prompt.text`.

2. **Modify the `generate_text` Method**: Update your `generate_text` method to use the correct attribute. Here's an example of how you might adjust your code:

   ```python
   def generate_text(
       self,
       prompt: PromptValue,
       n: int = 1,
       temperature: float = 1e-8,
       stop: t.Optional[t.List[str]] = None,
       callbacks: t.List = []
   ) -> LLMResult:
       # Use the correct attribute from PromptValue
       prompt_text = prompt.prompt_str if isinstance(prompt, PromptValue) else str(prompt)
       text = self._call(prompt_text)
       return LLMResult(generations=[Generation(text=text)])
   ```

Ensure that `prompt.prompt_str` is valid and correctly defined in the `PromptValue` class. If `prompt_str` is not the correct attribute, you will need to find the appropriate attribute that contains the prompt text <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/tests/unit/llms/test_llm.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
what should I do with this error?
12 validation errors for LLMResult
generations.0.0.Generation
  Input should be a valid dictionary or instance of Generation [type=model_type, input_value=('text', 'Processed: Give...: \n (without API key)'), input_type=tuple]
    For further information visit https://errors.pydantic.dev/2.9/v/model_type
generations.0.0.function-after[set_text(), ChatGeneration]
  Input should be a valid dictionary or instance of ChatGeneration [type=model_type, input_value=('text', 'Processed: Give...: \n (without API key)'), input_type=tuple]
    For further information visit https://errors.pydantic.dev/2.9/v/model_type
generations.0.0.GenerationChunk

<!-- Answer -->
The error you're encountering suggests that the `LLMResult` is expecting a list of `Generation` objects, but it seems like the input provided is not in the correct format. Specifically, the error indicates that the input should be a valid dictionary or an instance of `Generation`, but it appears to be a tuple instead.

Here's how you can address this issue:

1. **Ensure Correct Initialization of `Generation` Objects**: When creating `Generation` objects, make sure you are passing the correct parameters. The `Generation` object should be initialized with a dictionary or directly as an instance of `Generation`.

2. **Modify the `generate_text` Method**: Check your `generate_text` method to ensure that you are correctly creating `Generation` objects. Here's an example of how you might adjust your code:

   ```python
   def generate_text(
       self,
       prompt: PromptValue,
       n: int = 1,
       temperature: float = 1e-8,
       stop: t.Optional[t.List[str]] = None,
       callbacks: t.List = []
   ) -> LLMResult:
       # Use the correct attribute from PromptValue
       prompt_text = prompt.prompt_str if isinstance(prompt, PromptValue) else str(prompt)
       text = self._call(prompt_text)
       # Ensure you are creating a Generation object correctly
       generation = Generation(text=text)
       return LLMResult(generations=[[generation]])
   ```

3. **Check the Structure of `LLMResult`**: The `LLMResult` should be initialized with a list of lists of `Generation` objects. Make sure that the structure you are passing matches this expectation <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/tests/unit/llms/test_llm.py)</sup>.

By ensuring that the `Generation` objects are correctly instantiated and passed to `LLMResult`, you should be able to resolve the validation errors.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
this is customragasllm when I run the whole code I see failed to parse output. 
my code:
@dataclass
class CustomRagasLLM(BaseRagasLLM):
    api_key: t.Optional[str] = None

    def __post_init__(self):
        # Set the run config if necessary
        if self.run_config is None:
            self.run_config = RunConfig()   # Ensure this aligns with your Ragas setup

    async def _call(self, prompt: str) -> str:
        """
        Process the prompt and return a result as a string.
        """
        if not self.api_key:
            return f"Processed: {prompt} (without API key)"
        else:
            return f"Processed: {prompt} (with API key: {self.api_key})"
        
    def generate_text(
        self,
        prompt: t.Union[str, PromptValue],
        n: int = 1,
        temperature: float = 1e-8,
        stop: t.Optional[t.List[str]] = None,
        callbacks: t.List = []
    ) -> LLMResult:
        """
        Synchronous text generation.
        """
        # Ensure `prompt` is converted to a string if it’s a `PromptValue`
        prompt_text = prompt.prompt_str if isinstance(prompt, PromptValue) else str(prompt)
        
        print("generate_text is called with prompt:", prompt_text)  # Debugging

        # Generate a sample text response
        text = f"Processed: {prompt_text} (without API key)" if not self.api_key else f"Processed: {prompt_text} (with API key)"
        
        # Construct and return LLMResult
        result = LLMResult(generations=[[Generation(text=text)]])
        print("Returning LLMResult:", result)  # Confirming output structure
        return result

    async def agenerate_text(
        self,
        prompt: t.Union[str, PromptValue],
        n: int = 1,
        temperature: float = 1e-8,
        stop: t.Optional[t.List[str]] = None,
        callbacks: t.List = []
    ) -> LLMResult:
        """
        Asynchronous text generation.
        """
        # Convert prompt to string if it’s a PromptValue
        prompt_text = prompt.prompt_str if isinstance(prompt, PromptValue) else str(prompt)
        
        print("agenerate_text is called with prompt:", prompt_text)  # Debugging

        # Use the `_call` method to process asynchronously
        text = await self._call(prompt_text)
        
        # Construct and return LLMResult
        result = LLMResult(generations=[[Generation(text=text)]])
        print("Returning LLMResult:", result)  # Confirming output structure
        return result



@dosu
this is customragasllm when I run the whole code I see failed to parse output.
my code:

class CustomRagasLLM(BaseRagasLLM):
api_key: t.Optional[str] = None

def __post_init__(self):
    # Set the run config if necessary
    if self.run_config is None:
        self.run_config = RunConfig()   # Ensure this aligns with your Ragas setup

async def _call(self, prompt: str) -> str:
    """
    Process the prompt and return a result as a string.
    """
    if not self.api_key:
        return f"Processed: {prompt} (without API key)"
    else:
        return f"Processed: {prompt} (with API key: {self.api_key})"
    
def generate_text(
    self,
    prompt: t.Union[str, PromptValue],
    n: int = 1,
    temperature: float = 1e-8,
    stop: t.Optional[t.List[str]] = None,
    callbacks: t.List = []
) -> LLMResult:
    """
    Synchronous text generation.
    """
    # Ensure `prompt` is converted to a string if it’s a `PromptValue`
    prompt_text = prompt.prompt_str if isinstance(prompt, PromptValue) else str(prompt)
    
    print("generate_text is called with prompt:", prompt_text)  # Debugging

    # Generate a sample text response
    text = f"Processed: {prompt_text} (without API key)" if not self.api_key else f"Processed: {prompt_text} (with API key)"
    
    # Construct and return LLMResult
    result = LLMResult(generations=[[Generation(text=text)]])
    print("Returning LLMResult:", result)  # Confirming output structure
    return result

async def agenerate_text(
    self,
    prompt: t.Union[str, PromptValue],
    n: int = 1,
    temperature: float = 1e-8,
    stop: t.Optional[t.List[str]] = None,
    callbacks: t.List = []
) -> LLMResult:
    """
    Asynchronous text generation.
    """
    # Convert prompt to string if it’s a PromptValue
    prompt_text = prompt.prompt_str if isinstance(prompt, PromptValue) else str(prompt)
    
    print("agenerate_text is called with prompt:", prompt_text)  # Debugging

    # Use the `_call` method to process asynchronously
    text = await self._call(prompt_text)
    
    # Construct and return LLMResult
    result = LLMResult(generations=[[Generation(text=text)]])
    print("Returning LLMResult:", result)  # Confirming output structure
    return result

<!-- Answer -->
To resolve the "failed to parse output" error in your `CustomRagasLLM` code, ensure that the output string generated by your language model is correctly formatted according to the expected output model. This error is typically raised by the `RagasOutputParser` when it cannot parse the output string into the specified `OutputModel`.

Here are some steps to troubleshoot and resolve the issue:

1. **Check Output Format**: Ensure that the output generated by your language model matches the expected format defined by the `OutputModel`. The `RagasOutputParser` uses this model to parse the output string.

2. **Debug Output String**: Log or print the `output_string` before parsing to verify its content. This can help identify if the output is malformed or missing expected fields.

3. **Review OutputModel Definition**: Verify that the `OutputModel` is correctly defined and that all required fields are present in the output string.

4. **Handle Parsing Exceptions**: The code should log an error message when parsing fails. You can add additional logging to capture more details about the `output_string` and the specific parsing error.

5. **Automatic Output Correction**: The `RagasOutputParser` includes a mechanism to automatically correct the output format if it doesn't initially meet the expected schema. Ensure that this mechanism is enabled and configured correctly <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/prompt/multi_modal_prompt.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/prompt/pydantic_prompt.py)</sup>.

By ensuring that your output string is correctly formatted and that the `OutputModel` is properly defined, you should be able to resolve the parsing error. Additionally, make use of the automatic correction feature to handle minor formatting issues.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
I see this as the out put why it gives me such a this result?
what I should do?

ato.
Evaluating: 100%|████████████████████████████████████████████████████████████████████████| 3/3 [08:24<00:00, 168.30s/it]
Applying monkey patch for column: context_utilization, Scores: [nan, nan, nan]
{'context_utilization': 0.0000}
PS C:\Users\Amin\git\ragas-prototype>



<!-- Answer -->
The output "Applying monkey patch for column: context_utilization, Scores: [nan, nan, nan]" indicates that the `_compute_score` method in the `LLMContextRecall` class is returning `np.nan` because the denominator (`denom`) is zero. This typically happens when there are no valid classifications returned by the LLM, resulting in an empty `responses` list.

To resolve this issue, you should:

1. **Verify LLM Configuration**: Ensure that your LLM is correctly configured and capable of returning valid classifications. Check the input data being fed to the LLM to ensure it is correct and complete.

2. **Add Logging**: Implement additional logging to track the inputs and outputs of the LLM. This can help identify if the LLM is receiving the correct prompts and if it is generating the expected responses.

3. **Check Input Data**: Make sure that the input data to the LLM is properly formatted and contains all necessary information for the LLM to generate valid responses.

By addressing these areas, you should be able to resolve the issue with the `context_utilization` scores being `nan` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_recall.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
i still see this:
Failed to parse output: Answer not found in new_ragas_dataset.. Returning None.
Failed to parse output: Answer not found in new_ragas_dataset.. Returning None.
Failed to parse output: Answer not found in new_ragas_dataset.. Returning None.
Failed to parse output: Answer not found in new_ragas_dataset.. Returning None.
Evaluating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00,  1.99it/s] 
c:\Users\Amin\git\ragas-prototype\.venv\Lib\site-packages\ragas\evaluation.py:277: RuntimeWarning: Mean of empty slice
  value = np.nanmean(self.scores[cn])
RAGAS Evaluation Report:
{'context_utilization': nan}
PS C:\Users\Amin\git\ragas-prototype> 



<!-- Answer -->
The error "Failed to parse output: Answer not found in new_ragas_dataset" suggests that the evaluation process is unable to find the expected "Answer" in your dataset. This could be due to a mismatch between the expected structure of the dataset and the actual data provided.

Here are some steps to troubleshoot and resolve this issue:

1. **Verify Dataset Structure**: Ensure that your dataset is correctly structured according to the expected format. The `new_ragas_dataset` should be structured as instances of `SingleTurnSample` or `MultiTurnSample`, with the necessary fields populated, such as `user_input`, `retrieved_contexts`, `reference_contexts`, `response`, `reference`, etc. <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/dataset_schema.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/components/eval_dataset.md)</sup>.

2. **Check for Missing Data**: Make sure that all required fields are present in each sample of your dataset. Missing or incorrectly named fields can lead to parsing errors.

3. **Review Output Parsing**: Ensure that the output from your LLM is correctly formatted and contains the expected "Answer" that matches the entries in your dataset. The `RagasOutputParser` relies on this to parse the output correctly.

4. **Debugging**: Add logging to your code to print out the dataset and the outputs being parsed. This can help identify any discrepancies between the expected and actual data.

5. **Handle NaN Values**: The warning about the "Mean of empty slice" indicates that the evaluation is encountering NaN values. This can happen if the scores are not being computed correctly due to missing or invalid data. Ensure that your dataset and LLM outputs are complete and valid to avoid NaN results <sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/evaluation.py)</sup>.

By ensuring that your dataset is correctly structured and that the LLM outputs are properly formatted, you should be able to resolve these parsing errors.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
does  Ragas library expects each question-answer pair to be grouped as individual dictionaries? 

<!-- Answer -->
Yes, the Ragas library expects each question-answer pair to be structured as individual dictionaries within your dataset. Each dictionary should contain the necessary fields such as the question, answer, and any relevant context. This structure allows the evaluation functions to process each pair independently and compute the required metrics accurately. Make sure your dataset is organized in this manner to ensure compatibility with the Ragas evaluation process.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
i am using the ragas version 0.1.6 does what you said about the dataset that should be individual dictionaries also work for this version.

<!-- Answer -->
Yes, in Ragas version 0.1.6, each question-answer pair should be structured as individual dictionaries within the dataset. This is required by the `from_list` and `from_dict` methods in the `RagasDataset` class, where each dictionary represents a sample, either as a `SingleTurnSample` or `MultiTurnSample`, depending on the presence and type of the `user_input` field <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/dataset_schema.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
I face this error :
i think its debugged now. the next error is this:
Exception has occurred: AttributeError
'list' object has no attribute 'rename_columns'
  File "C:\Users\Amin\git\ragas-prototype\src\rag_eva.py", line 353, in <module>
    evaluation_report = evaluate(new_ragas_dataset, metrics=metrics, embeddings=ragas_embeddings, llm=custom_llm, column_map=column_map)  # Pass your custom LLM here
    
    what should I DO?

<!-- Answer -->
The error you're encountering, `'list' object has no attribute 'rename_columns'`, suggests that the `new_ragas_dataset` is being passed as a list, but the `evaluate` function expects a `Dataset` or `EvaluationDataset` object that supports the `rename_columns` method.

Here's how you can resolve this issue:

1. **Convert List to Dataset**: If `new_ragas_dataset` is currently a list of dictionaries, you need to convert it into an `EvaluationDataset` using the `from_list` method. This will ensure that it is in the correct format for the `evaluate` function.

   ```python
   from ragas.dataset_schema import EvaluationDataset

   # Convert your list to an EvaluationDataset
   new_ragas_dataset = EvaluationDataset.from_list(new_ragas_dataset)
   ```

2. **Ensure Correct Dataset Type**: Make sure that `new_ragas_dataset` is an instance of `EvaluationDataset` or `Dataset` before passing it to the `evaluate` function.

3. **Check Column Mapping**: Ensure that the `column_map` parameter is correctly defined to map your dataset's column names to the expected names by the evaluation framework.

By converting your list to an `EvaluationDataset`, you should be able to resolve the error and proceed with the evaluation.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
but i am using ragas version 0.1.6
there is no from ragas.dataset_schema import EvaluationDataset
also how or where I can see how is the format of  Dataset

<!-- Answer -->
In Ragas version 0.1.6, you can refer to the `EvaluationDataset` class to understand the format of a dataset. Although there is no direct import statement like `from ragas.dataset_schema import EvaluationDataset`, the class is defined within the library and provides several methods to interact with the dataset.

Here's a summary of the `EvaluationDataset` class and its relevant parts:

- **Attributes:**
  - `samples`: A list of `BaseSample` objects, which can be either `SingleTurnSample` or `MultiTurnSample`.

- **Methods:**
  - `validate_samples(samples)`: Validates that all samples are of the same type.
  - `get_sample_type()`: Returns the type of the samples in the dataset.
  - `to_hf_dataset()`: Converts the dataset to a Hugging Face Dataset.
  - `to_pandas()`: Converts the dataset to a pandas DataFrame.
  - `features()`: Returns the features of the samples.
  - `from_list(mapping)`: Creates an `EvaluationDataset` from a list of dictionaries.
  - `from_dict(mapping)`: Creates an `EvaluationDataset` from a dictionary.
  - `to_csv(path)`: Converts the dataset to a CSV file.
  - `to_jsonl(path)`: Converts the dataset to a JSONL file.
  - `from_jsonl(path)`: Creates an `EvaluationDataset` from a JSONL file.

- **Sample Types:**
  - `SingleTurnSample`: Represents evaluation samples for single-turn interactions with attributes like `user_input`, `retrieved_contexts`, `reference_contexts`, `response`, `multi_responses`, `reference`, and `rubric`.
  - `MultiTurnSample`: Represents evaluation samples for multi-turn interactions with attributes like `user_input`, `reference`, `reference_tool_calls`, `rubrics`, and `reference_topics`.

You can use these methods and attributes to understand and manipulate the dataset format in Ragas <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/dataset_schema.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
I convert the dataset i mean my dataset to pandas dataframe but I still see the error.

<!-- Answer -->
Converting your dataset to a pandas DataFrame is not sufficient for the `evaluate` function in Ragas version 0.1.6, as it expects an `EvaluationDataset` or a `Dataset` object. Here's how you can convert your dataset to the correct format:

1. **Convert to EvaluationDataset**: If your dataset is currently a pandas DataFrame, you need to convert it into an `EvaluationDataset`. You can do this by first converting the DataFrame to a list of dictionaries and then using the `from_list` method.

   ```python
   from ragas.dataset_schema import EvaluationDataset

   # Convert DataFrame to a list of dictionaries
   data_list = df.to_dict(orient='records')

   # Convert the list to an EvaluationDataset
   evaluation_dataset = EvaluationDataset.from_list(data_list)
   ```

2. **Ensure Correct Structure**: Make sure that each dictionary in your list has the necessary fields like `user_input`, `reference_contexts`, and `reference` for `SingleTurnSample`, or the appropriate fields for `MultiTurnSample`.

3. **Pass to Evaluate Function**: Once you have the `EvaluationDataset`, you can pass it to the `evaluate` function.

By ensuring your dataset is in the correct format, you should be able to resolve the error and proceed with the evaluation.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
Exception has occurred: AttributeError
'list' object has no attribute 'rename_columns'
  File "C:\Users\Amin\git\ragas-prototype\src\rag_eva.py", line 350, in <module>
    evaluation_report = evaluate(new_ragas_dataset, metrics=metrics, embeddings=ragas_embeddings, llm=custom_llm)  # Pass your custom LLM here
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'rename_columns'

<!-- Answer -->
The error `'list' object has no attribute 'rename_columns'` indicates that `new_ragas_dataset` is being passed as a list, but the `evaluate` function expects a `Dataset` or `EvaluationDataset` object that supports the `rename_columns` method.

To resolve this issue, you need to convert your list of dictionaries into an `EvaluationDataset`. Here's how you can do it in Ragas version 0.1.6:

1. **Convert List to EvaluationDataset**: Use the `from_list` method to convert your list of dictionaries into an `EvaluationDataset`.

   ```python
   from ragas.dataset_schema import EvaluationDataset

   # Assuming new_ragas_dataset is a list of dictionaries
   evaluation_dataset = EvaluationDataset.from_list(new_ragas_dataset)
   ```

2. **Pass the EvaluationDataset to Evaluate**: Once you have the `EvaluationDataset`, you can pass it to the `evaluate` function.

   ```python
   evaluation_report = evaluate(evaluation_dataset, metrics=metrics, embeddings=ragas_embeddings, llm=custom_llm)
   ```

By converting your list to an `EvaluationDataset`, you should be able to resolve the error and proceed with the evaluation.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu
i see this in the terminal when i run my code:
or the v1 compatibility namespace if you are working in a code base that has not been fully upgraded to pydantic 2 yet.   from pydantic.v1 import BaseModel

and also in my internal ragas source code i have a class called : class RagasoutputParser(PydanticOutputParser):

can these two be realted to eachother? or cuase the error?
at the end when I run the whole code the out put is this:

Failed to parse output: Answer not found in new_ragas_dataset.. Returning None.
Evaluating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00, 2.44it/s]
c:\Users\Amin\git\ragas-prototype.venv\Lib\site-packages\ragas\evaluation.py:277: RuntimeWarning: Mean of empty slice
value = np.nanmean(self.scores[cn])
RAGAS Evaluation Report:
{'context_utilization': nan}

<!-- Answer -->
The message you're seeing about importing `BaseModel` from `pydantic.v1` suggests that your code or dependencies are using Pydantic v1 compatibility mode. This is typically done when a codebase has not been fully upgraded to Pydantic v2. The `RagasOutputParser` class, which extends `PydanticOutputParser`, might be affected by this if it relies on Pydantic models for parsing outputs.

The error "Failed to parse output: Answer not found in new_ragas_dataset" indicates that the evaluation process is unable to find the expected "Answer" in your dataset. This could be due to a mismatch between the expected structure of the dataset and the actual data provided. Here are some steps to address these issues:

1. **Check Dataset Structure**: Ensure that your dataset is correctly structured with all necessary fields. Each sample should be an instance of `SingleTurnSample` or `MultiTurnSample`, with fields like `user_input`, `retrieved_contexts`, `reference_contexts`, `response`, and `reference` properly populated <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/components/eval_dataset.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/dataset_schema.py)</sup>.

2. **Review Output Parsing**: The `RagasOutputParser` might be failing to parse the output due to a mismatch in expected and actual output formats. Ensure that the output from your LLM is correctly formatted and contains the expected "Answer" that matches the entries in your dataset.

3. **Handle NaN Values**: The warning about "Mean of empty slice" indicates that the evaluation is encountering NaN values. This can happen if the scores are not being computed correctly due to missing or invalid data. Ensure that your dataset and LLM outputs are complete and valid to avoid NaN results <sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/index.md)</sup>.

4. **Pydantic Compatibility**: If your codebase is using Pydantic v1 compatibility mode, ensure that all Pydantic models and parsers are correctly defined and compatible with this mode. This might involve checking the `RagasOutputParser` and any other Pydantic models for compatibility issues.

By addressing these areas, you should be able to resolve the parsing errors and improve the evaluation results.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*