zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: How to use ChatAnthropicVertex without replying on LangchainLLMWrapper
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I'm struggling with `LangchainLLMWrapper` deprecation warning.
The current version of my faithfullness metric was implemented like this:
```python
from ragas.metrics import Faithfulness
from ragas import SingleTurnSample
from langchain_google_vertexai.model_garden import ChatAnthropicVertex

llm = ChatAnthropicVertex(
    credentials=my_credentials,
    project=my_project,
    location="us-east5",
    model="claude-3-5-sonnet-v2@20241022",
    temperature=0.0,
    top_p=0.0,
)

ragas_llm = LangchainLLMWrapper(llm)

scorer = Faithfulness(llm=ragas_llm)
sample = SingleTurnSample(
    user_input="qual a capital do brasil?",
    response="a capital do brasil é sao paulo",
    retrieved_contexts=["sao paulo é a capital do brasil", "brasilia é a capital do brasil"],
)

score = await scorer.single_turn_ascore(sample)
print(score)
```

And it's working, but I keep getting this deprecating warning message:
```
DeprecationWarning: LangchainLLMWrapper is deprecated and will be removed in a future version. Use the modern LLM providers instead: from ragas.llms.base import llm_factory; llm = llm_factory('gpt-4o-mini') or from ragas.llms.base import instructor_llm_factory; llm = instructor_llm_factory('openai', client=openai_client)
```

The thing is: I cannot/don't know how to fix it. I tried using `llm_factory` and `instructor_llm_factory`without success. Can you help me by providing an example of how to use `ChatAnthropicVertex` correctly? I checked the documentation and could not find it.

Ragas version: 0.3.7

@dcaled We're removing the deprecation warning with this [PR](https://github.com/explodinggradients/ragas/pull/2387). 

Also, `instructor_llm_factory` has been removed. `llm_factory` will be available going forward to remove this confusion as well.

All of this will be out in this week release. 

Thank you for the response @anistark
Repository: explodinggradients/ragas
Issue: dynamically check which LLM class is it and if its langchain and not wrapped, just wrap it ourselves
Root cause: Validation Gap

Hey, I noticed that `langchain.llms.OpenAI.generate()` and `langchain.llms.Bedrock.generate()` are supposed to take `prompt: List[str]` instead of `List[ChatPromptTemplate]` that is currently being passed. 

Although the OpenAI class doesn't seem to error with the wrong input type, the Bedrock class does. It errors at `langchain.llms.bedrock._human_assistant_format()`.

Similarly, `langchain.chat_models.BedrockChat.generate()` `messages` argument should take type `List[List[BaseMessage]]` instead of `ChatPromptTemplate`.  This causes an error in `langchain.chat_models.anthropic. _convert_one_message_to_text`.


Thanks again!

Actually, this might just be me forgetting to wrap the LLM in `LangchainLLM` 😅

yeah that is mostly the reason although we are trying to to figure out how to avoid that gotcha 😅 
if you have any suggestion please do let us know - about how we can make it much more easier for users to use this the correct way.

On option I had in mind was to dynamically check which LLM class is it and if its langchain and not wrapped, just wrap it ourselves. should avoid the gotchas here
Repository: explodinggradients/ragas
Issue: how to control Ragas0.3.6 only generate simple test data
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


@z1quovadis-max Could you please expand on what exactly are you looking for?

Did you mean one single test data or single-hop questions? Didn't quite understand.
Repository: explodinggradients/ragas
Issue: Can I build a knowledge graph using Neo4j and then pass it on to Ragas for use?
GitHub labels: question

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
The node types in the knowledge graph of ragas are relatively few, but the node types I need to construct are quite numerous. This does not fully meet my requirements. Could we use another knowledge graph library for construction and then hand it over to ragas to generate the answers?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us?
Repository: explodinggradients/ragas
Issue: Non-deterministic scores
Root cause: Configuration Error

Hey, apologies for creating multiple issue threads so rapidly, but I like this library and hope it gains more traction!

Some metrics are giving different results each time they are ran even with temperature set to 0. For example, Faithfulness and AnswerCorrectness.

I believe this might be due to using LangChain `ChatOpenAI` instead of `OpenAI`, where the latter appears more deterministic with temperature set to 0, but I'm not positive about this.

I got the same issue as well, every time I run the evaluation, it will give slight or significant different scores for each question sometime. Any advice @shahules786  ? 

Is it normal ? if so, what is the reason behind this ? to what extent the difference can be tolerated ?
Should not be giving the same score for each time we run evaluation ?

I guess the new seed feature would be helpful here

Hi @austinmw @ihgumilar , thanks for sharing your thoughts & concerns. The scores can vary due to the non-deterministic nature of LLMs, I have seen this happening more with closed-sourced services. The new `seed` feature from OpenAI might be useful, but I have yet to experiment with it. But I would want something more generalization beyond OpenAI.  

To cut down the noise you can try to run the aggregate results across more data points @ihgumilar 

Self-consistency checks can be a good way to improve reproducibility.

Thanks a lot for the suggestion guys @shahules786 @austinmw . Any suggestion what we can do ?
@austinmw's suggestion could be implemented easily I believe, we can just add seed parameter. But not sure for other models.

Cheers

I'm experiencing the same issue. I looked into this and in my case it seems that the passed temperature value is not respected, it's either set to 0.2 or 1e-8.
https://github.com/explodinggradients/ragas/blob/v0.0.22/src/ragas/llms/langchain.py#L200

1e-8 is very close to 0 but not zero. Is this a bug or an intended feature?

Also there seems to be a lack of consistency between the async and non-async implementation as for async it's always set to 0:
https://github.com/explodinggradients/ragas/blob/v0.0.22/src/ragas/llms/langchain.py#L146

I'm also experiencing score fluctuations between runs. Is the seed feature implemented yet?

@shahules786 FYI I fixed the seed but it didn't work...
Repository: explodinggradients/ragas
Issue: Get explanation along with scores
Root cause: Documentation Deficiency

The evaluate call on a data set and list of metrics returns only the scores. 

Isn't there any way to also get a corresponding explanation? Especially if the scores are low and I have hundreds of questions, it is almost impossible to rationalize them manually.

Any help would be greatly appreciated. 


Continuing from #2296 

I guess you're trying to access `.get('reason')` directly on the prompt output. However, faithfulness metric's prompt output (NLIStatementOutput) is a Pydantic model with a statements field (a list). The reasoning is nested one level deeper: `statements[i].reason`, not at the top level.

Try to access something like `results.traces[i][metric_name][prompt_name]['output']`

Hey , I’m working on an open source tool called vero-eval, for testing and debugging RAG/Agents. It generates test dataset through multiple edge cases, and user-personas by understanding your application's use-case, calculates a lot of metric scores for each block, and generates a report explaining the scores, what went wrong in which case and fix suggestions

We’re early and constantly working on feedback, would love to know how you find it

Repo (pip install available) is here if you want to poke around: https://github.com/vero-labs-ai/vero-eval 
Would appreciate any feedback. Thanks already

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: The formula for recall was mistakenly described as the formula for precision.
GitHub labels: documentation

question

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
The document appears to have mistakenly described the recall formula as the precision formula.

URL: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/sql/
Path: docs/concepts/metrics/available_metrics/sql.md
Repository: explodinggradients/ragas
Issue: Can RAGAS custom metrics return a value of type str (string)?
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Can RAGAS custom metrics return a value of type str (string)?

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


@02hao09 Yes, but only as discrete metrics with predefined string values, not arbitrary strings.



It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Make embeddings optional in AnswerCorrectness metric when similarity weight is 0
Root cause: Validation Gap

## Problem

The new collections-based `AnswerCorrectness` metric requires embeddings even when they're not used (when `weights[1] == 0`).

**Current Behavior:**
```python
# Embeddings always required, even if unused
scorer = AnswerCorrectness(
    llm=llm, 
    embeddings=embeddings,  # Required even with weights=[1.0, 0.0]
    weights=[1.0, 0.0]  # 100% factuality, 0% similarity
)
```

**Legacy Behavior:**
The legacy implementation only created embeddings when `weights[1] != 0`, making them truly optional for factuality-only use cases.

## Proposed Solution

Make embeddings optional and validate based on weights:

```python
def __init__(
    self,
    llm: "InstructorBaseRagasLLM",
    embeddings: Optional["BaseRagasEmbedding"] = None,
    weights: List[float] = [0.75, 0.25],
    ...
):
    if weights[1] > 0 and embeddings is None:
        raise ValueError(
            "Embeddings required when similarity weight > 0. "
            "Provide embeddings or set weights=[1.0, 0.0] for factuality only."
        )
    
    self.embeddings = embeddings
```

This maintains flexibility while providing clear error messages when embeddings are actually needed.

**File:** `src/ragas/metrics/collections/_answer_correctness.py`
Repository: explodinggradients/ragas
Issue: Are SemanticSimilarity and AnswerSimilarity supposed to be different?
Root cause: Architectural Constraint

## Problem

The collections API has two implementations that appear to be identical:
- `SemanticSimilarity` ([_semantic_similarity.py](src/ragas/metrics/collections/_semantic_similarity.py))
- `AnswerSimilarity` ([_answer_similarity.py](src/ragas/metrics/collections/_answer_similarity.py))

**The implementations are 100% identical** - same cosine similarity algorithm, same parameters. The only differences are:
1. Class name
2. Default metric name: `"semantic_similarity"` vs `"answer_similarity"`

## Context

In the legacy API ([_answer_similarity.py](src/ragas/metrics/_answer_similarity.py#L109)), `AnswerSimilarity` was simply a subclass alias of `SemanticSimilarity`:

class AnswerSimilarity(SemanticSimilarity):
    name: str = "answer_similarity"When migrating to collections API, both were implemented as separate standalone classes with duplicated code.

## Current State

- `SemanticSimilarity` has [documentation](docs/concepts/metrics/available_metrics/semantic_similarity.md) and is in the metrics index
- `AnswerSimilarity` has no documentation and is not in the metrics index
- Both are exported in `ragas.metrics.collections.__all__`

## Question

Should we consolidate these, or is there a reason to keep both as separate implementations?

Good catch. Can merge them together with `AnswerSimilarity` removed. with docs updated.
Repository: explodinggradients/ragas
Issue: docs: examples with Amazon Bedrock
GitHub labels: documentation

Add documentation and examples of ragas with Amazon Bedrock. Please assign this issue to me.

Hey @arm-diaz , thank you. This will go under [customisation](https://docs.ragas.io/en/latest/howtos/customisations/index.html). 

Can you @shahules786 check the current documentation? I got an error with harmfulness. It's there something you guys would like to change/add to it? 

Hi, is this currently using `BedrockChat.generate()`? If so I think it should be faster with `BedrockChat.batch()`

![image](https://github.com/explodinggradients/ragas/assets/12224358/8a744742-df7b-449f-bb2c-c146f80fee90)


**Edit:** Actually nvm, I think there's already an async call I missed.

hey I'm closing this because we have the docs now at https://docs.ragas.io/en/latest/howtos/customisations/aws-bedrock.html.

feel free to reopen or comment below if there is more to be done :)
Repository: explodinggradients/ragas
Issue: Link for office hours is broken
GitHub labels: bug

The link to office hours, but in ragas.io

<img width="945" height="169" alt="Image" src="https://github.com/user-attachments/assets/9f9e2188-2572-48cb-b47a-1f2b0baf3b70" />

and in the README of this repo, redirects to a 404 page

Thanks for reporting @raffaem 

The new link is https://cal.com/team/vibrantlabs/office-hours
Will update the docs as well.
Repository: explodinggradients/ragas
Issue: How to set the max_tokens for LLM (Large Language Model)?
Root cause: Documentation Deficiency

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
The max_tokens setting I set in the chatopenai client didn't work, resulting in a token limit exceeded during the evaluation. How should I set a valid max_tokens?

Ragas version: v0.2.15
Python version: 3.12.5

**Code to Reproduce**
Exception raised in Job[1]: LLMDidNotFinishException(The LLM generation was not completed. Please increase the max_tokens and try again.)
Exception raised in Job[2]: LLMDidNotFinishException(The LLM generation was not completed. Please increase the max_tokens and try again.)
**Error trace**
ChatOpenAI(
                api_key=os.getenv("OPENAI_API_KEY"),
                base_url=os.getenv("OPENAI_API_BASE"),
                model=os.getenv("MODEL"),
                temperature=0,
                top_p=0.1,
                max_tokens=8192,
                timeout=420
            )
**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


@jietang00723-cloud Please update to latest ragas 0.3.9 

```py
from ragas.llms import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o", max_tokens=8192)
```

I set it up like this, but it seems that the setting didn't work. Additionally, I would like to ask how to set max_token using llm_facy. thanks

@jietang00723-cloud 

```py
from openai import OpenAI
from ragas.llms import llm_factory

client = OpenAI(api_key="your-api-key")

# Set max_tokens through llm_factory via **kwargs
llm = llm_factory(
    model="gpt-4o",
    client=client,
    max_tokens=8192
)
```

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Docs: Quick Start content vs. code disconnect
Root cause: Documentation Deficiency

  - [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I'm going through the Quick Start guide and it appears to be a disconnect between the code installed by the `quickstart` script and the code listed on the Quick Start page.

The code that gets installed when running `ragas quickstart rag_eval` has:
```python
def load_dataset():
    dataset = Dataset(...)

    data_samples = [
        {
            "question": "What is ragas 0.3",
            "grading_notes": "- experimentation as the central pillar - provides abstraction for datasets, experiments and metrics - supports evals for RAG, LLM workflows and Agents",
        },
...    ]

    for sample in data_samples:
        row = {"question": sample["question"], "grading_notes": sample["grading_notes"]}
        dataset.append(row)

    # make sure to save it
    dataset.save()
    return dataset
```
The code on the [Quick Start](https://docs.ragas.io/en/stable/getstarted/quickstart/#customize-your-evaluation) page suggests to add:
```python
def load_dataset():
    """Load test dataset for evaluation."""
    data_samples = [
        SingleTurnSample(
            user_input="What is Ragas?",
            response="",  # Will be filled by querying RAG
            reference="Ragas is an evaluation framework for LLM applications",
            retrieved_contexts=[],
        ),
...        # Add more test cases here
    ]

    dataset = EvaluationDataset(samples=data_samples)
    return dataset
```
but the two pieces of code don't see to fit together. The first one loads a dataset from file and the second one uses `SingleTurnSample` which is not clear how it works.

Secondly, it also talks about an `_init_clients()` function that is missing in the installed code.

Ragas version: `0.3.9`
Python version: `3.13.3`

**Code to Reproduce**
See above

**Error trace**
N/A

**Expected behavior**
The installed code fits well with the code listed on the Quick Start page.

**Additional context**
N/A

Thanks @rvernica for reporting this. Will get the docs updated.
Repository: explodinggradients/ragas
Issue: RAGAS Examples improve_rag ModuleNotFoundError
Root cause: External Dependency Breakage

-  [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
```
> uv pip install "ragas-examples[improverag]"
Audited 1 package in 13ms
> python -m ragas_examples.improve_rag.rag
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File ".../.venv/lib/python3.13/site-packages/ragas_examples/improve_rag/rag.py", line 15, in <module>
    from langchain.docstore.document import Document
ModuleNotFoundError: No module named 'langchain.docstore'
```
and
```
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File ".../.venv/lib/python3.13/site-packages/ragas_examples/improve_rag/rag.py", line 17, in <module>
    from langchain.text_splitter import RecursiveCharacterTextSplitter
ModuleNotFoundError: No module named 'langchain.text_splitter'
```

Ragas version: `0.3.9`
Python version: `3.13.3`

**Code to Reproduce**
See above 

**Error trace**
See above

**Expected behavior**
Script runs

**Additional context**
The fix is:
```python
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
```

This example needs update. Thanks for reporting this @rvernica
Repository: explodinggradients/ragas
Issue: examples/improve_rag/eval temperature and top-p errors with GPT-5 mini
Root cause: Configuration Error

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Default values set for `temperature` and `top_p` by the `llm_factory/InstructorLLM` do not work with GPT-5 mini used in the `improve_rag/eval`.

Current version of [improve_rag/eval.py](https://github.com/explodinggradients/ragas/blob/f0eff321cb21c271722531be500d1c2be6e9e567/examples/ragas_examples/improve_rag/evals.py) example fails with:

```xml
<generation number="1">
<exception>
    Error code: 400 - {'error': {'message': "Unsupported value: 'temperature' does not support 0.01 with this model. Only the default (1) value is supported.", 'type': 'invalid_request_error', 'param': 'temperature', 'code': 'unsupported_value'}}
</exception>
<completion>
    None
</completion>
</generation>
```
and
```xml
<generation number="1">
<exception>
    Error code: 400 - {'error': {'message': "Unsupported parameter: 'top_p' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'top_p', 'code': 'unsupported_parameter'}}
</exception>
<completion>
    None
</completion>
</generation>
```

Ragas version: `0.3.9`
Python version: `3.13.3`

**Code to Reproduce**
See above

**Error trace**
See above

**Expected behavior**
Successful evaluation

**Additional context**
The fix is to customize the `llm_factory` instantiation in [eval.py](https://github.com/explodinggradients/ragas/blob/f0eff321cb21c271722531be500d1c2be6e9e567/examples/ragas_examples/improve_rag/evals.py#L209) to:
```python
llm_factory("gpt-5-mini", client=openai_client, temperature=1, top_p=None)
```