Repository: explodinggradients/ragas
Issue: failed to parse output when using baidu qianfan to evaluate
Root cause: Documentation Deficiency

- [x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**my Question**
“failed to parse output”  when using baidu qianfan to evaluate，I just want to know how to solve this problem and would appreciate more case studies!

**Code Examples**
As you can see my code below, the error reported after running it is as follow：
![image](https://github.com/user-attachments/assets/2ab4be65-4357-4aa2-a9e5-13fa1e906c06)

```python

import pandas as pd
import numpy as np

df = pd.read_excel('data.xlsx')

c=['l1','l2','l3','l4','l5']
contexts=[]
for index,row in df.iterrows():
    contexts.append([x for x in [row['l1'],row['l2'],row['l3'],row['l4'],row['l5']] if pd.notna(x)])
question=df['question'].tolist()
answer=df['answer'].tolist()
groud_truth=[''] * len(question)

data={
    'question':question,
    'answer':answer,
    'contexts':contexts,
    'groud_truth':groud_truth
}

from datasets import Dataset
dataset = Dataset.from_dict(data)

############################

from langchain_community.chat_models import QianfanChatEndpoint
from ragas.llms import LangchainLLMWrapper
from langchain_community.embeddings import QianfanEmbeddingsEndpoint

import os
os.environ["QIANFAN_AK"] = ""
os.environ["QIANFAN_SK"] = ""


chat = QianfanChatEndpoint(model='ERNIE-Speed-128K')
v_llm = LangchainLLMWrapper(chat)



v_embeddings = QianfanEmbeddingsEndpoint(

           )


from ragas import evaluate
from ragas.metrics import (
    #answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
)

faithfulness.llm = v_llm
#answer_relevancy.llm = v_llm
#answer_relevancy.embeddings = v_embeddings
context_recall.llm = v_llm
context_precision.llm = v_llm


result = evaluate(
    dataset,
    metrics=[
        context_precision,
        faithfulness,
        #answer_relevancy,
        context_recall,
    ],
)

df = result.to_pandas()
df.head()
``` 

**Additional context**
I wonder if this project can provide some examples of using custom LLMs for evaluation, especially LLMs that Langchain provides interface support for accessing, and provide solution guides for common errors reported in custom LLMs, which I have encountered so far, including the failed to parase output mentioned in this issue, and other multiple issues.


<!-- Greeting -->
@Qinlilseven 你好! 我是Dosu，一个可以帮你解决错误、回答问题，甚至成为贡献者的机器人。我会在这里帮助你，等待人类维护者的到来。

<!-- Answer -->
The "failed to parse output" error in the Ragas evaluation framework can be caused by several issues:

1. **Improper JSON Formatting**: Ensure the output is a well-formatted JSON that adheres to the required schema.

2. **Schema Mismatch**: Verify that the output matches the expected JSON schema, including all required fields and correct data types.

3. **RagasOutputParserException**: This exception is raised if parsing fails after a specified number of retries <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/output_parser.py)</sup>.

To address these issues, you can:

- Check the output format and ensure it matches the expected schema.
- Implement logging to capture the exact output being parsed, which can help identify formatting issues.
- Review the retry mechanism in the code to ensure it's correctly handling parsing attempts.

For resource management issues like unclosed `aiohttp.ClientSession` objects, ensure that sessions are properly closed after use, possibly using context managers (`async with`) to manage their lifecycle.

Unfortunately, I couldn't find specific case studies or examples in the Ragas documentation related to custom LLMs. You might need to explore the documentation further or experiment with different configurations.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


https://docs.ragas.io/en/latest/getstarted/rag_evaluation/#choosing-evaluator-llm mentioned choosing evaluator llm，but no further details were mentioned