Repository: explodinggradients/ragas
Issue: After upgrading the version from 0.2.15 to 0.3.6, there was an error message displayed.
Root cause: Configuration Error

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
After upgrading my ragas version from 0.2.15 to 0.3.6, I encountered an error message. As follows:  LM returned 1 generations instead of requested 3. Proceeding with 1 generations. What should I do?
**Code Examples**
 # 创建LLM实例
    llm = ChatOpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
        model="deepseek-chat",
        logprobs=True
    )
   



    # 构建评估数据集
    eval_data = {
        'user_input': [
            '人工智能的三个主要分支是什么？',
            '什么是向量数据库？它在RAG系统中有什么作用？',
            '解释一下大语言模型的微调过程'
        ],
        'retrieved_contexts': [
            ['人工智能主要包括机器学习、深度学习、自然语言处理和计算机视觉等领域。'],
            ['向量数据库是专门用于向量数据存储和检索的系统。在RAG应用中，向量数据库存储文档嵌入，通过计算相似度快速找到匹配查询的文档。'],
            ['大语言模型微调是在已预训练的模型基础上，使用特定数据集进行额外训练的过程，目的是使模型适应特定任务或领域。']
        ],
        'response': [
            '人工智能的三个主要分支是机器学习、深度学习和自然语言处理。',
            '向量数据库是专门存储和检索向量嵌入的数据库系统，在RAG系统中用于高效地检索语义相似的文档。',
            '大语言模型的微调是在预训练模型的基础上，使用特定任务的数据进一步训练模型的过程。'
        ],
        'reference': [
            '人工智能的三个主要分支是机器学习、自然语言处理和计算机视觉。',
            '向量数据库是专门设计用于存储和检索向量嵌入的数据库系统。在RAG系统中，它用于存储文档的向量表示，并通过相似性搜索快速检索与查询最相关的文档。',
            '大语言模型的微调是一个过程，其中预训练的基础模型使用特定领域或任务的数据进行额外训练，以优化其在特定应用场景的性能。'
        ],
        
    }
    comprehensive_dataset = Dataset.from_dict(eval_data)


    from ragas.metrics import (
        context_recall,
        context_precision,
        answer_correctness,
        answer_similarity,
        answer_relevancy,
        faithfulness
    )

    # 定义评估指标
    metrics = [
        faithfulness,
        answer_relevancy,
        answer_similarity,
        answer_correctness,
        context_precision,
        context_recall,

    ]
 
    # 综合评估
    comprehensive_score = evaluate(
        comprehensive_dataset,
        metrics=metrics,
        llm=llm,
        embeddings=embeddings
        # embeddings=jina_embedding
    )
    # 结果分析
    result_df = comprehensive_score.to_pandas()

**Additional context**
After upgrading the version from 0.2.15 to 0.3.6, there was an error message displayed.


@jietang00723-cloud Do you mean the deprecation warning for LangChain LLM?

Could you share the error message exactly?

> [@jietang00723-cloud](https://github.com/jietang00723-cloud) Do you mean the deprecation warning for LangChain LLM?你是否指的是 LangChain LLM 的弃用警告？
> 
> Could you share the error message exactly?你能准确分享错误信息吗？
Hello, I'm glad you replied. The problem I encountered was merely a warning message. Although this information doesn't have much impact on my usage, I still want it to disappear. The warning message is as follows:
‘’LM returned 1 generations instead of requested 3. Proceeding with 1 generations.‘’

<img width="1555" height="100" alt="Image" src="https://github.com/user-attachments/assets/3551c3fd-e6e1-4062-aa81-1d17705f20e8" />

I'm also facing the same problem, on the same ragas version (0.3.6).
I believe this message comes from here:
https://github.com/explodinggradients/ragas/blob/8b190e16b49b2e3f4d31af70426536fb59beec6f/src/ragas/prompt/pydantic_prompt.py#L267-L271

Also, I'm using `langchain_openai`'s `AzureOpenAIEmbeddings`. 

The issue most likely is that some LLM providers/configurations may not support or may limit the number of concurrent batched requests, returning only 1 generation instead of the requested 3.

You can either, set a lower strictness, `answer_relevancy.strictness = 1`

or migrate to Ragas's native LLM interface for better support:

```py
from ragas.llms import llm_factory

llm = llm_factory(
    "openai",
    api_key=openai_api_key,
    base_url=openai_api_base,
    model="deepseek-chat"
)

comprehensive_score = evaluate(
    comprehensive_dataset,
    metrics=metrics,
    llm=llm,
    embeddings=embeddings
)
```

Thank you. I think I have a general idea of what the problem is. By the way, I'd like to mention that the "faithfulness" indicator is relatively insensitive to the logical reasoning contained in the answers and often makes incorrect judgments. For instance, the model performs context-based refinement, but the judgment of fidelity is not accurate.

Thanks for bringing it up @jietang00723-cloud 

Would you like to discuss faithfulness in a separate issue?

If this issue is resolved for you, we can close it then.

The test code `test_bypass_n_true_async_does_not_pass_n` in `tests/unit/llms/test_llm.py` demonstrates a robust workaround for this issue. The recommended approach is to wrap the LangChain LLM using LangchainLLMWrapper and pass the `bypass_n=True` parameter within the wrapper.

sample code looks like:
```
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# 1. create an origin LangChain LLM（without  bypass_n）
self.eval_llm = ChatOpenAI(**llm_kwargs)
self.eval_embeddings = OpenAIEmbeddings(**embedding_kwargs)

# 2. Use LangchainLLMWrapper to wrap LLM，an set bypass_n=True
self.ragas_llm = LangchainLLMWrapper(
    langchain_llm=self.eval_llm, 
    bypass_n=True  # Set bypass_n here
)

# 3. Use wrapped LLM in evaluate function
eval_results = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_recall, context_precision],
    llm=self.ragas_llm,  # 使用包装后的 LLM
    embeddings=self.eval_embeddings,
)
```

Unsupported provider: 'deepseek'. Supported: openai, anthropic, google, litellm

llm = llm_factory(
    "openai",
    api_key=deepseek_api_key,
    base_url=deekseek_base_url,
    model="deepseek-chat"
)

@paperClub-hub 

Please open a separate issue for this.

is the original question fixed in newer versions? I'm getting the same output but maybe there is a better fix. Im using Azure OpenAI.

> is the original question fixed in newer versions? I'm getting the same output but maybe there is a better fix. Im using Azure OpenAI.

Are you using the new metrics collections and still getting error?
If so, please open a new issue about it alongwith the version using and logs.