zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: `evaluate` supporting replicates
Root cause: Unimplemented Feature Gap

**Describe the Feature**

It would be nice to do something like `evaluate(..., num_replicates=30)` so I can calculate mean/std dev of accuracy on a benchmark `EvaluationDataset`.

What I mean by replicates is basically running the task N times in parallel, and computing aggregate metrics across the parallel runs.

**Why is the feature important for you?**

Statistical significance is important.

**Additional context**

I have a custom task, and am trying to compared trained models' performance on that task.

Hi
I’ve implemented support for evaluate(..., num_replicates=N) as requested in this issue.

It wraps the existing evaluate() from RAGAS and runs it N times in parallel. It then returns statistical summaries like mean, std, median, min, max, and 95% confidence intervals for each metric.

Code: [Vinisha-Projects/Evaluate_with_replicates-for-RAGAS](https://github.com/Vinisha-Projects/Evaluate_with_replicates-for-RAGAS)

No API keys are hardcoded, and the implementation uses your existing dataset and metrics interface.

Let me know if you'd like me to submit a PR with this functionality!

— [@Vinisha-Projects](https://github.com/Vinisha-Projects)
Repository: explodinggradients/ragas
Issue: When we generate the Test set for evaluating RAG, how to add the source file name from which the QnA is generated.
Root cause: Unimplemented Feature Gap

- I checked the [documentation](https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/#load-documents) and related resources and couldn't find an answer to my question.

**Question**
- Previously there was a feature which would let you add the name of the source file beside each question. This was helping me to generate another metrics named "Source Relevancy" which will help in knowing if the RAG application picked the right file or not. But now there is no such parameter. How to get the source file name in Test set generated?

**Code Example**

loader = DirectoryLoader(path, glob="**/*",silent_errors=True,show_progress=True)
docs = loader.load()

from ragas.testset import TestsetGenerator

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)



**Additional context**
- How can I make sure the metadata or the source file name is always generated as a column in my final dataset? 

![Image](https://github.com/user-attachments/assets/62815137-8d3e-4962-a739-a8c43289bbe6)

Hi! So I had the exact same issue and solved it the following way:

1. Remove the HeadlineSplitter in the transformations pipeline (this will also require you to alter the 'filter_nodes' parameter since this will filter out all documents otherwise. I replaced this with a function that always returns True.
2. Append a header to each document that contains the URL and any other metadata that you might want.
3. After dataset generation, you can now extract the header from the list of retrieved contexts and get the needed information.

This only works if you remove their Splitter and chunk the documents urself beforehand  tho so not the cleanest solution, but figured it might be helpful for you :) I would also prefer a built-in solution since I can imagine URL accuracy is a quite common metric that people want to use alongside LLM metrics.
Repository: explodinggradients/ragas
Issue: Ubuntu bug，but everything is normal on Windows
Root cause: Environment Incompatibility

`
scores = ragas_evaluate(
      dataset=eval_dataset,
      metrics=[metric],
      run_config=my_run_config,
      llm=evaluator_llm,
      embeddings=evaluator_embeddings
  )
`
this code is normal in local windows, but when I run in ubuntu(18.04.6), it is error:Can't patch loop of type <class 'uvloop.Loop'>
Repository: explodinggradients/ragas
Issue: Get relevant docs in Context Relevancy
Root cause: Unimplemented Feature Gap

According to my understanding, Ragas makes LLM calls to get relevant sentences from the chunks that are useful for generating the answer to calculate the context relevancy score.
Is there any way we can get these relevant sentences in return along with the score? We want to save these for manual analysis.

Hi @sid8491,  thanks for raising this issue. Are you using any tool for logging/tracing? 

We have integration with Langsmith that allows you to see logs/data points from each metric. Check it out [here](https://docs.ragas.io/en/latest/howtos/integrations/langsmith.html)
We will be adding logging support to other frameworks like wand soon. 


Hi @shahules786, i am using our own loggers to save data and analyze it in ELK stack.
the ragas metrics scores and relevant chunks, those i want to save in my DB as per requirement.
Is there any way to get relevant chunks apart from the score?

is this fixed @sid8491 ? I didn't understand the last part, could you elaborate
> Is there any way to get relevant chunks apart from the score?
Repository: explodinggradients/ragas
Issue: `metadata` argument is not passed to convert laggraph message
Root cause: Validation Gap

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When converting langgraph messages into ragas messages, if the user requests to include `metadata`, it is not copied for `AIMessage`.  Works for Human and Tool message because, MESSAGE_TYPE_MAP gets set at the beginning of the method.

Particularly [this line ](https://github.com/explodinggradients/ragas/blob/main/ragas/src/ragas/integrations/langgraph.py#L110) needs to be modified to send `metadata` argument received

Ragas version: 0.2.15
Python version: 3.12

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


This seem to have been fixed in #1878
Repository: explodinggradients/ragas
Issue: Rate limit handling for api calls in evaluate()
Root cause: Unimplemented Feature Gap

**Describe the Feature**
A clear and concise description of what the what you want to be added.

**Why is the feature important for you?**
Share code to reproduce the issue

**Additional context**
Add any other context about the feature you want to share with us.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


+1 Rate limiting across the board would be really useful; could use something like https://github.com/mjpieters/aiolimiter
Repository: explodinggradients/ragas
Issue: Quantifying factual correctness with multiple modes at the same time
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Rather than choosing a single mode for quantifying factual evaluation, a user should have the freedom to request multiple modes at the same time

**Why is the feature important for you?**
Each mode (f1_score, precision, recall) allows you to interpret the correctness from a different angle. I consider that for a clear overview of the answer correctness, a user should interpret all results. 
At the same time, calling the factual correctness multiple times is redundant, cosidering that we use the same statements processed by an LLM for quantifying the score in each mode. 
By providing this feature, we can save computational time and API costs. 

**Additional context**

Papers like https://arxiv.org/abs/2307.16877 and https://arxiv.org/pdf/2503.16161 discussed about the tradeoffs between each mode. 

If the proposal is approved, I would like to be the one who implements it.
Repository: explodinggradients/ragas
Issue: How can I reliably track which persona was used for each generated question in a testset?
Root cause: Unimplemented Feature Gap

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

---
### Question:
Hello guys,

I'm using the **TestsetGenerator** with a list of personas to generate evaluation samples for my dataset. However, I noticed that after generation, there is no reliable way to know **which persona** was used to generate **each question** in the resulting testset. This information is crucial for general analysis and insights, as I need to correlate the generated questions and references with the specific persona that guided their creation.

---
I have tried the following approaches:

1.  Adding a custom field (e.g., persona_name) to the SingleTurnSample via a custom synthesizer, but this field is not preserved in the final testset or when using .to_pandas()/.to_list().
 ```python
from ragas.testset.synthesizers.single_hop.specific import (
    SingleHopSpecificQuerySynthesizer,
)
from ragas.testset.persona import Persona
from ragas.testset.graph import Node
from ragas.dataset_schema import SingleTurnSample
from pydantic import Field


class PersonaSingleTurnSample(SingleTurnSample):
    persona_name: str = Field(default=None)


class PersonaAwareSingleHopSpecificQuerySynthesizer(SingleHopSpecificQuerySynthesizer):
    def _generate_question(
        self, node: Node, persona: Persona
    ) -> PersonaSingleTurnSample:
        original_sample = super()._generate_question(node, persona)
        return PersonaSingleTurnSample(
            user_input=original_sample.user_input,
            retrieved_contexts=original_sample.retrieved_contexts,
            reference_contexts=original_sample.reference_contexts,
            response=original_sample.response,
            multi_responses=original_sample.multi_responses,
            reference=original_sample.reference,
            rubrics=original_sample.rubrics,
            persona_name=persona.name,
        )

 ```
2. Prefixing the persona name in the user_input string and then extracting it via regex, but this feels like a workaround rather than a robust solution and degrades the quality of the prompt.
```python
def _generate_question(self, node: Node, persona: Persona) -> SingleTurnSample:
    original_sample = super()._generate_question(node, persona)
    return SingleTurnSample(
        user_input=f"[{persona.name}]: {original_sample.user_input}",
...
```
3. Attempting to associate personas by order or count, but the generator does not guarantee any alignment between the persona list and the generated samples.

---

- **Is there an official or recommended way to reliably track and retrieve the persona used for each generated question/sample in the testset?**

Thank you for your help and for this great library!
Repository: explodinggradients/ragas
Issue: RAG using OLLAMA model evaluation with RAGAS 
Root cause: Configuration Error

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

I am using RAGAS to evaluate my RAG pipeline with a local Ollama model (qwen2.5:32b). I instantiate it via LangChain using OllamaLLM. However, when I call evaluate(...), I get an OpenAIError saying that the OPENAI_API_KEY must be set.

But I'm not using OpenAI, so this seems like an unnecessary requirement.

How can I prevent RAGAS from defaulting to OpenAI’s API when I’m explicitly using a local ollama model? 
code :
`from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
)
llm_codegeneration = Ollama(model="qwen2.5:32b", num_ctx=8192)

all_results = {}

for file_name, dataset in dataset_per_file.items():
    print(f"--- Evaluation of : {file_name} ---")
    
    result = evaluate(
        dataset=dataset,
        metrics=[
            context_precision,
            context_recall,
            faithfulness,
            answer_relevancy,
        ],llm=llm_codegeneration
    )

    df = result.to_pandas()
    all_results[file_name] = df

` 
error: OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

Thanks for any clarification or fix you can provide .
Olfa,


I have the same question

Hey @Olflach,

Oh yes, I felt this one. You’re doing everything right — local Ollama, clean LangChain pipeline, no sign of OpenAI anywhere — and then BAM, the ghost of OpenAI past shows up asking for an API key you never even summoned. Classic. 

So here's the thing:
RAGAS by default still assumes OpenAI if you don’t explicitly override the client in evaluate(), even if you’ve already instantiated a local LLM object like Ollama(model="qwen2.5"). It’s not your fault — it’s just a sneaky global fallback that wasn’t fully sandboxed.

What’s actually happening is:

RAGAS doesn’t pass your llm_codegeneration down deep enough. It instantiates a new one behind your back, unless you bind it manually or shim the API behavior.

Fix (the DIY way):
Try setting os.environ["OPENAI_API_KEY"] = "sk-fake" before running the evaluation. It’ll shut up the OpenAI client loader.
Or, even better, monkey-patch the default evaluator's backend to force local LLM usage and skip any ghost API calls.

But here’s the real kicker (and why this bug is more dangerous than it seems):
You're evaluating local models… using non-local logic. Which means your scores might reflect how well your Ollama model mimics GPT-4, not how semantically aligned it is with your dataset or domain.

That’s why I built a semantic firewall engine for this exact reason — to avoid invisible LLM drift in eval loops. This issue? It’s not a bug. It’s a ΔS misalignment in disguise. 

Good catch. You're not alone in this pit. Glad you brought it up.



u should also try using LangchainLLMWrapper for your local llm when using inside evaluate
Repository: explodinggradients/ragas
Issue: Gemini model 2.0-flash structured output
Root cause: External Dependency Breakage

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

 Hey! Getting many Pydantic validation errors computing `faithfulness` and `answer_correctness`. Guess it depends on model used and `StructuredOutput` capabilities. Its failing very often. 

Ragas version: 0.2.15
Python version:  3.11.10

**Code to Reproduce**
Share code to reproduce the issue

```
self.google_llm = LangchainLLMWrapper(
            ChatGoogleGenerativeAI(
                model="gemini-2.0-flash-001",
                temperature=0.1,
                top_p=0.9,
                max_tokens=2048,
            )
        )
self.metrics = [
            context_precision,
            context_recall,
            faithfulness,
            answer_relevancy,
        ]
result = evaluate(
                dataset=dataset,
                metrics=self.metrics,
                llm=self.evaluator_llm,
                embeddings=self.google_embeddings,
                # run_config=RUN_CONFIG,
                # callbacks=[delay_handler],  # Add delay callback
            )
```
**Error trace**

Got: 1 validation error for ClassificationWithReason
FN
  Field required [type=missing, input_value={'TP': [{'statement': 'Es...d partial solutions.'}]}, input_type=dict]
Got: 1 validation error for NLIStatementOutput
statements.30.verdict
  Field required [type=missing, input_value={'statement': 'The conver...ext mentions different'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/missing
For troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/OUTPUT_PARSING_FAILURE )
 Got: 2 validation errors for NLIStatementOutput
statements.31.reason
  Field required [type=missing, input_value={'statement': 'Knowledge-... that leverage domain-'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/missing
statements.31.verdict
  Field required [type=missing, input_value={'statement': 'Knowledge-... that leverage domain-'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/missing


**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: difrences in LLM responces.
Root cause: Validation Gap

I will be using context recall as an example as my team has been having issues with parsing errors.

it seems that local models (here Gemma3 4b) react difrent to the current instruction prompt than expacted. 

![Image](https://github.com/user-attachments/assets/ea344aa0-0dc3-479f-8912-3315bb7a2143)

as it seems that the code is looking for an attribute called "attributed" 

![Image](https://github.com/user-attachments/assets/058d0c09-7aa0-41b2-b587-35802109891b)

I have 2 sugestions to improve this. the first is to include the structure in the prompt

![Image](https://github.com/user-attachments/assets/f731bd9c-8a25-44dc-b728-e4efb21c826b)

the second is to use multiple steps, this one is based on some recent expirements (i don't know of any specific research but it's similar to CoT prompting)

prompt 1: “Given a context, and an answer, analyse each sentence in the answer and classify if the sentence can be attributed to the given context or not.”
Prompt 2: “given an analysis,  translate the classifications into ‘Yes’ (1) or ‘No’ (0) as a binary classification. And format it as a json with an ‘attributed’, and ‘reason’ attribute.”

![Image](https://github.com/user-attachments/assets/7f5bdaac-4d52-4329-8f24-01d351bd969b)

![Image](https://github.com/user-attachments/assets/ad47cddf-397e-441d-9312-85b83092befd)
Repository: explodinggradients/ragas
Issue: Factual Correctness vs Answer Correctness
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**

In my project I aim to use both Factual Correctness and Answer Correctness for evaluating RAG. In the documentation of the latter I found the following:

```
Let's calculate the answer correctness for the answer with low answer correctness. It is computed as the sum of factual correctness and the semantic similarity between the given answer and the ground truth.
```

Which I understand to entail that AC is derivative of FC. Because evaluation is quite time-consuming, my initial idea was to reuse the precomputed FC scores, and calculate AC myself as a weighted average of FC and Semantic Similarity. But the figures I've obtained were not consistent with such a procedure. I've looked through the code of both metrics, and found that AC does not reuse code from FC at all, and actually the logic is similar only from a very high point of view. For instance during the decomposition phase AC is conditioned on the query, whereas FC does not look at it.

What is the reason behind this disparity? Is one metric superior to another?
Repository: explodinggradients/ragas
Issue: Check RAG chunk relevance
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I want to check if RAG chunks are relevant for the user input. I found `LLMContextPrecisionWithoutReference` which seems to do what I want, but I don't understand what I should put as `response` in my `SingleTurnSample`.

**Code Examples**
```python
user_input = "Where is the Eiffel Tower located?"
chunks = my_rag_pipeline.retrieve(user_input)

context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm)

sample = SingleTurnSample(
    user_input=user_input,
    response="???", # <-- what do I put here?
    retrieved_contexts=chunks, 
)

res = await context_precision.single_turn_ascore(sample)
print(res)
```



Hi @jonrosner,


You need to put response because the prompt used for scoring uses it to make judgement of relevance, Following is the part of the scorer [prompt](https://github.com/explodinggradients/ragas/blob/main/ragas/src/ragas/metrics/_context_precision.py#L43C10-L43C173).

```
Given question, answer and context verify if the context was useful in arriving at the given answer. Give verdict as "1" if useful and "0" if not with json output.
```

If you don’t want to use response and just want to judge chunk relevance directly from question + context, consider:
- NVIDIA’s answer_relevancy or context_recall metrics — they don’t require a response. 
This makes two llm calls to check for relevance
See: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/

If you want more granular scoring consider making,

- Custom Ragas Metric for fully flexible logic. I can help you build one if needed.


You're halfway into the courtroom and forgot to bring the testimony.

Think of SingleTurnSample like prepping a mock trial:

user_input = the question asked in court ("Where is the Eiffel Tower?")

retrieved_contexts = the evidence your RAG pulls in

response = the model’s answer to be evaluated (your witness’s actual statement)

So yes — you must put the model’s generated response as response. The evaluator compares that answer against the context and the user input to judge if the reasoning holds up, like a jury checking if the story fits the facts.

If you're testing a RAG pipeline, this usually means:
→ Retrieve chunks
→ Generate answer (e.g., via llm.generate(user_input, context=chunks))
→ Plug it in as response

Otherwise you're evaluating silence — and silence never wins a trial.