Repository: explodinggradients/ragas
Issue: AttributeError('StringIO' object has no attribute 'classifications')
Root cause: Interface Contract Mismatch

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I'm using the latest ragas version and have been encountering the AttributeError('StringIO' object has no attribute 'classifications') error message when evaluating metrics. 

I'm using chatglm APIs and wonder if there is a compatibility issue.

Ragas version: 0.2.5
Python version: 3.12

**Code to Reproduce**
```
from datasets import Dataset
import pandas as pd
from ragas import evaluate, RunConfig

from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI
from langchain_openai import OpenAIEmbeddings
from langchain_community.chat_models import ChatZhipuAI
from langchain_community.embeddings import ZhipuAIEmbeddings

from ragas.metrics import (
    Faithfulness,
    ResponseRelevancy,
    LLMContextRecall,
    LLMContextPrecisionWithReference,
)

def read_excel(file_path, sheet_name='Sheet1'):
    data = pd.read_excel(file_path, sheet_name=sheet_name)
    data.fillna('', inplace=True)
    data = [each_row._asdict() for each_row in data.itertuples(index=False)]
    return data

def write_xlsx(file_path, data, sheet_name='Sheet1'):
    data = pd.DataFrame(data)
    data.to_excel(file_path, sheet_name=sheet_name, index=False)

def get_dataset(data):
    questions = []
    answers = []
    contexts = []
    ground_truths = []

    for each in data:
        questions.append(each['query'])
        answers.append(each['answer'])
        contexts.append([each['context']])
        ground_truths.append(each['reference'])

    data = {
        "user_input": questions, 
        "response": answers, 
        "retrieved_contexts": contexts,
        "reference": ground_truths
    }
    dataset = Dataset.from_dict(data)
    return dataset

def setup_llm_and_embedder():
    llm = LangchainLLMWrapper(ChatZhipuAI(
        base_url="https://open.bigmodel.cn/api/paas/v4/", 
        api_key="xxx",  # API Key
        model="glm-4-plus",  
        max_tokens= 8000
    ))

    text_embedder = LangchainEmbeddingsWrapper(ZhipuAIEmbeddings(
        api_base="https://open.bigmodel.cn/api/paas/v4/",
        api_key="xxx",  # API Key
        model="embedding-2"
        
    ))
    return llm, text_embedder

if __name__ == "__main__":
    file = 'xxx'
    save_file = 'xxx'
    data = read_excel(file)
    dataset = get_dataset(data)
    llm, text_embedder = setup_llm_and_embedder()
    run_config = RunConfig(
        max_retries=5,
        max_wait=120,
        timeout=500,
        max_workers=8
    )
    result = evaluate(
        dataset = dataset,
        llm = llm,
        run_config=run_config,
        embeddings = text_embedder,
        metrics=[
            Faithfulness(llm=llm),
            ResponseRelevancy(llm=llm),
            LLMContextRecall(llm=llm),
            LLMContextPrecisionWithReference(llm=llm),
        ],
    )
    df = result.to_pandas()
    write_xlsx(save_file, df)
```

**Error trace**
Evaluating:   2%|█▍                                                                                 | 14/792 [00:51<33:06,  2.55s/it]Exception raised in Job[10]: AttributeError('StringIO' object has no attribute 'classifications')
Evaluating:   5%|████▏                                                                            | 41/792 [02:50<1:32:51,  7.42s/it]Exception raised in Job[42]: AttributeError('StringIO' object has no attribute 'classifications')
Evaluating:   5%|████▎                                                                            | 42/792 [03:01<1:42:38,  8.21s/it]Exception raised in Job[46]: AttributeError('StringIO' object has no attribute 'classifications')
Evaluating:   7%|█████▋                                                                             | 54/792 [04:01<46:35,  3.79s/it]Exception raised in Job[54]: AttributeError('StringIO' object has no attribute 'classifications')
Evaluating:   7%|██████                                                                             | 58/792 [04:16<43:52,  3.59s/it]Exception raised in Job[50]: AttributeError('StringIO' object has no attribute 'classifications')
Evaluating:   8%|███████                                                                            | 67/792 [04:46<37:19,  3.09s/it]Exception raised in Job[62]: AttributeError('StringIO' object has no attribute 'classifications')
**Expected behavior**


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Also having this same problem while evaluating ragas faithfulness through the giskard.rag.evaluate function.

Looks like it occurs because of [fix_output_format_prompt object](https://github.com/explodinggradients/ragas/blob/99d1a47b9025ea46625d3f4985309eff24025c63/src/ragas/prompt/pydantic_prompt.py#L385) contains `StringIO` as `output_model` instead of type defined in `pydantic_object` field of `RagasOutputParser` class. I can`t get a logic of FixOutputFormat class for now.

@jjmachan @shahules786
Any inputs on this issue? Thanks

Im also facing the same issue.

I am also facing the same issue while evaluating ragas metrics through the giskard.rag.evaluate function.

I solved the particular problem I had while evaluating via giskard. When saving model results in an AgentAnswer object I was consolidating all of the contexts into a single stringz, whereas they should be saved as a list of strings. Converting to a list of strings solved the problem for me.

Here is a relevant extract from my model class - see comments in all caps

```
    def wrap_rag_model(
                self,
                question: str, 
                history=[]):
            messages = []
            for message in history:
                if message["role"] == "user":
                    messages.append({"inputs":{"chat_input":message['content']}})
                elif message["role"] == "assistant":
                    messages[-1]["outputs"] = {"chat_output":message["content"]}
            
            # Generate a response using Azure OpenAI
            response = self.call(user_prompt=question, history=messages)
    
            # Ensure that documents is a list of strings
            documents = self.get_response_context(response) if self.get_response_context(response) else []
            documents = [str(d) for d in documents]
            # Instead of returning a simple string, we return the AgentAnswer object which
            # allows us to specify the retrieved context which is used by RAGAS metrics
            return AgentAnswer(
                message=self.get_response_text(response),
                documents=documents # HERE ENSURE DOCUMENTS IS A LIST OF STRINGS
            )
    
    def scan_rag_model(
        self,
        testset: QATestset,
        knowledgebase: KnowledgeBase,
        ragas_metrics: list
        ):
    
        self.rag_scan = evaluate(
            self.wrap_rag_model,  #HERE THE FUNCTION MUST RETURN AN AgentAnswer object
            testset=testset,
            knowledge_base=knowledgebase,
            metrics=ragas_metrics,
            agent_description=self.description
            )
        
        return self.rag_scan
        

hey folks - taking a look at this now

> I solved the particular problem I had while evaluating via giskard. When saving model results in an AgentAnswer object I was consolidating all of the contexts into a single stringz, whereas they should be saved as a list of strings. Converting to a list of strings solved the problem for me.
> 
> Here is a relevant extract from my model class - see comments in all caps
> 
> ```
>     def wrap_rag_model(
>                 self,
>                 question: str, 
>                 history=[]):
>             messages = []
>             for message in history:
>                 if message["role"] == "user":
>                     messages.append({"inputs":{"chat_input":message['content']}})
>                 elif message["role"] == "assistant":
>                     messages[-1]["outputs"] = {"chat_output":message["content"]}
>             
>             # Generate a response using Azure OpenAI
>             response = self.call(user_prompt=question, history=messages)
>     
>             # Ensure that documents is a list of strings
>             documents = self.get_response_context(response) if self.get_response_context(response) else []
>             documents = [str(d) for d in documents]
>             # Instead of returning a simple string, we return the AgentAnswer object which
>             # allows us to specify the retrieved context which is used by RAGAS metrics
>             return AgentAnswer(
>                 message=self.get_response_text(response),
>                 documents=documents # HERE ENSURE DOCUMENTS IS A LIST OF STRINGS
>             )
>     
>     def scan_rag_model(
>         self,
>         testset: QATestset,
>         knowledgebase: KnowledgeBase,
>         ragas_metrics: list
>         ):
>     
>         self.rag_scan = evaluate(
>             self.wrap_rag_model,  #HERE THE FUNCTION MUST RETURN AN AgentAnswer object
>             testset=testset,
>             knowledge_base=knowledgebase,
>             metrics=ragas_metrics,
>             agent_description=self.description
>             )
>         
>         return self.rag_scan
>         
> ```

I tried this in my case and it still did not work (I am using giskard)

Issue is still occurring as of 2.10: #1831 

I'm seeing issues specifically with `Faithfulness`. This method works perfectly fine for very simple contexts `List[str]` but as the content inside the list gets increasingly complex something goes wrong. Doesn't matter if I force cast the content within the list to `str`. It still fails downstream.

Would actually be great if there was more verbose on what exactly goes wrong as it becomes quite an impossible task to debug...

---
EDIT: No wait, it seems to fail even with very simple `List[str]` contexts and maybe it is another metric causing it... No idea whats wrong anymore...

So after debugging this way too long, I managed to get around the issue for one of my applications by computing the `context_recall` metric separately. No idea why that was the issue. But if I do something like this, I am able to compute the metrics:
```
score = ragas.evaluate(
    dataset,
    metrics=[
        answer_correctness,
        faithfulness,
        answer_similarity,
        context_precision,
        answer_relevancy,
        # context_recall,   # <- DONT include this here
    ],
    llm=[...],
    embeddings=[...],
    raise_exceptions=True,
)
result = score.to_pandas()

# compute context recall separately
context_recall_score = ragas.evaluate(
    dataset,
    metrics=[context_recall],  # <- Include this metric here instead
    llm=[...],
    embeddings=[...],
    raise_exceptions=True,
)

# merge context recall score with result
result["context_recall"] = context_recall_score.to_pandas()["context_recall"]
```

And of course, there is no need to use `evaluate()` in the second step for this exact case, but maybe others experience issues with multiple metrics and this could be a quick-fix until the real issue has been resolved. Hopefully this gets resolved very soon!

Tested with `ragas==0.2.10`.

---
NOTE: This is by no means the fix to the solution but rather a temporary workaround/hack.

> So after debugging this way too long, I managed to get around the issue for one of my applications by computing the `context_recall` metric separately. No idea why that was the issue. But if I do something like this, I am able to compute the metrics:
> 
> ```
> score = ragas.evaluate(
>     dataset,
>     metrics=[
>         answer_correctness,
>         faithfulness,
>         answer_similarity,
>         context_precision,
>         answer_relevancy,
>         # context_recall,   # <- DONT include this here
>     ],
>     llm=[...],
>     embeddings=[...],
>     raise_exceptions=True,
> )
> result = score.to_pandas()
> 
> # compute context recall separately
> context_recall_score = ragas.evaluate(
>     dataset,
>     metrics=[context_recall],  # <- Include this metric here instead
>     llm=[...],
>     embeddings=[...],
>     raise_exceptions=True,
> )
> 
> # merge context recall score with result
> result["context_recall"] = context_recall_score.to_pandas()["context_recall"]
> ```
> 
> And of course, there is no need to use `evaluate()` in the second step for this exact case, but maybe others experience issues with multiple metrics and this could be a quick-fix until the real issue has been resolved. Hopefully this gets resolved very soon!
> 
> Tested with `ragas==0.2.10`.
> 
> NOTE: This is by no means the fix to the solution but rather a temporary workaround/hack.

In my case I get "'StringIO' object has no attribute 'statements'", an error related to faithfulness. Unfortunately, as I feared, this workaround did not work for me. 

closed with [fix: output parser bug by jjmachan · Pull Request #1864 · explodinggradients/ragas](https://github.com/explodinggradients/ragas/pull/1864) will be released with v0.2.12 🙂 

I'm closing this for now but if the issue is still persisting, please do let me know - really sorry about the delay

> get "'StringIO' object has no attribute 'statements'", an error related to faithfulness. Unfortunately, as I feared, this workaround did not work for me.

👍 

Hi @jjmachan,

I also get this error. On my end, after digging in the problem, I realised that single quotes happened to be in the JSON output generated by the judge LLM, making json parsing fail. It happens in Faithfulness metric because the judge LLM often cite extracted context in its reason field, with single quotes unfortunately. What I suggest is to slightly modify the Faithfulness prompt: https://github.com/explodinggradients/ragas/pull/1874 

thanks for the fix @michaelromagne - suggested a small change but lets get this merged in 🥳