zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Prompt ${xx_prompt} failed to parse output: The output parser failed to parse the output including retries.
Root cause: Validation Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?
When I perform Rag evaluation, I have the following issues:
1. Prompt context_precision_prompt failed to parse output
2. Prompt claim_decomposition_prompt failed to parse output
3.  Prompt n_l_i_statement_prompt failed to parse output
4.  Prompt extract_entities_prompt failed to parse output
5.  Prompt statement_generator_prompt failed to parse output
May I ask how to solve this


**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


I get the same error when running evaluate with context_precision and context_recall : "Prompt fix_output_format failed to parse output" and "Prompt context_recall_classification_prompt failed to parse output". The response given by the LLM is quite unstable because with the same input data and script, this won't always occur: on rare occasions, the evaluation goes through the entire dataset without raising any error. Maybe increasing the number of retries could help solve the problem

Ragas version 0.2.11

I have the same problem. When the length of reference and response is very long (response 665 characters, reference 2745 characters), llm_context_precision_with_reference,context_recall and fectual_correctness are all nan, and then the error message is: [ERROR] Prompt xxx failed to parse output: The output parser failed to parse the output including retries.

I get the same error，this happens in some jobs,not always occur,how to solve it ?
Repository: explodinggradients/ragas
Issue: How can I evaluate image retrieval?
Root cause: Documentation Deficiency

I've started using ColPali retrieval and would like to evaluate multimodal retrieval. Does Ragas support that?


Hi @simjak,

Yes, we have a couple of multi-modal metrics like [Multi-modal Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/multi_modal_faithfulness/) and [Multi-modal Relevance](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/multi_modal_relevance/). If you find any other metrics you'd like us to support, let us know!

hey @simjak 🙂 👋🏽 

like @sahusiddharth we have a few basic ones but would love to provide more support. Do try out the available ones and let me know how it goes? would be happy to help out anyways we can too


+1 (Might be helpful if Ragas supports video and image based prompt evaluations focused on gemini models)
Repository: explodinggradients/ragas
Issue: Question for open api key
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.


I have a question. In my current environment, I cannot use OpenAI API keys. Is there a way to use Ragas with different LLM APIs without using OpenAI API keys?




Hi @jinsoo96,

Yes, you can. For more details, check our documentation here: [Ragas Docs](https://docs.ragas.io/en/stable/howtos/customizations/customize_models/). Alternatively, you can use Langchain to use any model of your choice.


Thank you for your response! However, I understand that using the langchainWrapper feature in ragas, and later generating test sets, requires an OpenAI API key based on my review of ragas' source code.
Is it possible to use different embedding models or LLM models instead?

hey @jinsoo96 you might be still using the default LLM or embedding somewhere - did you check out the indepth guide https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/#a-deeper-look. You just have to figure out where you are using the default LLMs and then replace that

The issue is that I cannot import various functions like ragas langchainLLMWrapper in my current cloud development environment due to OpenAI API key transmission errors in the source code.
I'd like to know:

If I want to modify the source code directly (hardcode), which parts should I modify and would this approach be acceptable?
Are there any alternative solutions besides hardcoding?

The main concern is finding a way to use these ragas features without relying on the OpenAI API key requirement, either through source code modification or alternative methods. What would be the best approach to solve this problem?

see @jinsoo96 that should not be an issue because we did make sure a while back that ragas works without OpenAI - but maybe this is a bug that has happended too

could you share the code snippets you are trying to run? I'll check this out in detail

I'll check my environment settings again and verify if there are any bugs. Thank you for your thorough responses and helpful answers. Have a great day! :)

you too - hope you get this sorted 🙂 

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: Incorrect or confusing documentation about ContextPrecision metric
Root cause: Documentation Deficiency

[x ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

Documentation: The page https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/ has a section called 'Non LLM Based Context Precision' which then describes a method that uses an LLM.

```
NonLLMContextPrecisionWithReference metric is can be used when you have both retrieved contexts and also reference contexts associated with a user_input. To estimate if a retrieved contexts is relevant or not this method uses the LLM to compare each of the retrieved context or chunk present in retrieved_contexts with each ones present in reference_contexts.
```

This information seems to be conflicting and inconsistent. 



Ragas version: n/a
Python version: n/a

**Code to Reproduce**
n/a

**Error trace**

**Expected behavior**
Would expect a nonLLM based method to be documented as not requiring an LLM


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hi @richarda23,  

Thank you so much for bringing this to our attention! I looked into the underlying [code](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_precision.py#L189C1-L250C21) to double-check, and you’re absolutely correct this method does **not** use any LLM. The mention of LLM in the documentation is indeed a mistake, and I really appreciate you pointing this out.  

I’ve raised a PR to update the documentation with the correct details.
Thanks again for helping improve the project! 🙌
Repository: explodinggradients/ragas
Issue: List of new functionalities in version 0.2
GitHub labels: documentation

question

I checked the [documentation](https://docs.ragas.io/en/stable/howtos/migrations/migrate_from_v01_to_v02/) and related resources and couldn't find an answer to my question.

**Question:**
There has been a Groundbreaking improvement in version 0.2 that the Chat History will be considered while calculating the Metrics. But unfortunately, it is not mentioned in the documentation.

Also, details on Chat History for how many iterations is considered, also can be added.

Similarly, list of such major enhancements is missing. 

Something like, **WHAT'S NEW IN 0.2**, will help.

This clear information helps the Data Scientists decide whether to migrate to 0.2 or not

**Please add the requested information in your response to this issue**

This might help.

https://docs.ragas.io/en/latest/howtos/migrations/migrate_from_v01_to_v02/


> This might help.
> 
> https://docs.ragas.io/en/latest/howtos/migrations/migrate_from_v01_to_v02/

@Vidit-Ostwal - Checked it but it is not clear

hey @GenAI-Rocky generated something for you here: https://www.perplexity.ai/search/site-docs-ragas-io-i-checked-t-i.Sbk2auQ4q6pBj1EJ.TXw

but I think what you are looking for is the dataset scheme that chanaged with v0.2? that is mentioned here https://docs.ragas.io/en/latest/concepts/components/eval_dataset/

let me know if that helps 🙂 

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: AnswerCorrectness does not pass llm to underlying faithfulness & similarity classes
Root cause: Configuration Error

Hi,

Happy with the new release and I'm eager to try out the new answer similarity and answer correctness metrics. One thing I noticed is that the llm argument you can pass to AnswerCorrectness is not used when instantiating the underlying faithfulness and similarity classes to calculate the metrics.
See below code

```
from ragas.metrics import AnswerCorrectness, AnswerSimilarity, Faithfulness
answer_similarity = AnswerSimilarity(llm=azure_model)
faithfulness = Faithfulness(llm=azure_model)
answer_correctness = AnswerCorrectness(llm=azure_model)

# code to create dataset, for this example called ragas_ds["eval"]

# calculating the metrics
f_score = faithfulness.score(ragas_ds["eval"])  # works!
s_score = answer_similarity.score(ragas_ds["eval"]) # works!
a_score = answer_correctness.score(ragas_ds["eval"])  # error! must provide engine or deployment_id
```

it does work when you pass the already instantiated faithfulness and similarity to correctness like the following:
```
answer_correctness = AnswerCorrectness(answer_similarity=answer_similarity, faithfulness=faithfulness)
a_score = answer_correctness.score(ragas_ds["eval"])  # works!
```


However, I would expect it to work as well when passing an llm as argument. I suspect the following block in answer_correctness.py needs a little tweak along the following lines
```
    def __post_init__(self: t.Self):
        if self.answer_similarity is None:
            self.answer_similarity = AnswerSimilarity(llm=<reference to self.llm or something like that>)
        if self.faithfulness is None:
            self.faithfulness = Faithfulness(llm=<reference to self.llm or something like that>)
```
this would also require a check that, if faithfulness and similarity arent passed to correctness, at least the llm argument should be passed. Hope it helps, keep up the good work!


Regards, Koen

thnks you so much for bringing this issue up - will make a patch for this asap and merge this in 😃
Repository: explodinggradients/ragas
Issue: difference between retrieved contexts  and reference contexts
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.
what is the difference between the retrieved contexts  and reference contexts that extract from generate Synthetic test data?


@dosu do you know this?, explain with an example as well

<!-- Answer -->
The difference between retrieved contexts and reference contexts in the context of generating synthetic test data is as follows:

- **Retrieved Contexts**: These are the contexts that the system retrieves for a given query. They represent the information identified as relevant to the user's input query. In the `SingleTurnSample` class, this is represented by the attribute `retrieved_contexts`, which is a list of strings.

- **Reference Contexts**: These are the contexts considered as the correct or expected ones for a given query. They serve as a benchmark or ground truth against which the retrieved contexts can be evaluated. In the `SingleTurnSample` class, this is represented by the attribute `reference_contexts`, which is also a list of strings.

The key difference lies in their roles: retrieved contexts are what the system actually retrieves, while reference contexts are what ideally should have been retrieved <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/dataset_schema.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/f65c1796-a75f-41f9-b653-4d2af2dd375f?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/f65c1796-a75f-41f9-b653-4d2af2dd375f?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/f65c1796-a75f-41f9-b653-4d2af2dd375f?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/f65c1796-a75f-41f9-b653-4d2af2dd375f?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/f65c1796-a75f-41f9-b653-4d2af2dd375f?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/f65c1796-a75f-41f9-b653-4d2af2dd375f?feedback_type=other)</sup>


@dosu, explain this with example as well
Repository: explodinggradients/ragas
Issue: Where to specify "max_tokens" - getting " LLMDidNotFinishException(The LLM generation was not completed. Please increase try increasing the max_tokens and try again.)" error
Root cause: Documentation Deficiency

Hi all,
   I am unable to figure out from the documentation where to specify the `max_tokens` value to avoid getting the following error:

` LLMDidNotFinishException(The LLM generation was not completed. Please increase try increasing the max_tokens and try again.`

Also what can be done to avoid the following error:

`The LLM did not return a valid classification.`

Thanks,
Paddy

Hi @tvsathish,

I wasn't able to locate an option to directly configure the max_tokens setting within Ragas. It's possible that this feature is not exposed in the current version.

Here’s an example of how you can set the `max_tokens`:

```python
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", max_tokens=100))
print(llm.langchain_llm.max_tokens)
```

This should output:

```
100
```

Let me know if this works for you. 

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Instability in answer relevance score
Root cause: Configuration Error


For evaluating my RAG pipeline, in real time, I am implementing 'Answer Relevance' score. But what I found that for same set of question, answer, and context it is generating different scores. And these scores are varying by 10-15%. I understand as it is a probabilistic measure (using LLM, re-engineering questions from answer etc.), the score may not be the same always. But varying 10-15% is a trust issue. Any solution to this? 

hey @laughinghugs this is a problem we are aware of and hoping to solve it with approaches like the once mentioned in https://blog.ragas.io/aligning-llm-as-judge-with-human-evaluators.

@shahules786 would you have any suggestions 

The variation in scores you're seeing (10-15%) is expected to some extent since LLM-based evaluations introduce a degree of randomness. However, such a large variation can be a **trust issue**, as you mentioned.

One possible cause is the way the 'Answer Relevance' score is computed—specifically, the process of **re-engineering questions from the answer** before measuring cosine similarity. If this step isn't generating a **diverse enough set of questions**, the cosine similarity metric may not be capturing the true relevance consistently, leading to unstable scores.

### **Potential Issue Affecting Stability**
I've found that the **temperature setting in LangchainLLMWrapper is being ignored**, and it is **always set to a very low value (~0)** regardless of what the user specifies, which in this case i believe medium to high temperature. This means that instead of generating multiple diverse questions, the system is **producing the same question repeatedly**—which can lead to artificially high or low cosine similarity scores, depending on the case.

I believe that fixing this issue: **[Answer Relevance Issue](https://github.com/explodinggradients/ragas/issues/1812)** could **help stabilize the score variance**. With a properly diverse set of generated questions, the cosine similarity measurement should be more robust and better reflect the intended behavior.

### **Quick Fix You Can Try**
If you want a **quick win**, you can explicitly pass the temperature in your implementation:
```
response = await self.question_generation.generate(
    data=prompt_input,
    llm=self.llm,
    temperature=self.llm.langchain_llm.temperature,  # Ensure temperature is set
    callbacks=callbacks,
)
````

This is not the most elegant solution, but it should increase diversity in generated questions if you set your llm with high temperature, which might help stabilize your scores.

Let me know if you try this, and if it improves your results!

@galrettig - Thanks for the reply. Let me try it. I will post the outcome.
Repository: explodinggradients/ragas
Issue: does it support deepseek
GitHub labels: good first issue

question

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Does RAGAS support deepseek, if can, please add demo in doc.



Hi @landhu,

We are planning to have support for DeepSeek soon. 
We'll make sure to add a demo in the documentation once it's ready.

hey @landhu - we actually do support deepseek via langchain

these docs should help you
- https://api-docs.deepseek.com/faq#does-your-api-support-langchain
- https://docs.ragas.io/en/stable/howtos/customizations/customize_models/

hope this helps and let us know how it goes 🙂 

like @sahusiddharth we'll also have a demo notebook too on how to leverage it soon
Repository: explodinggradients/ragas
Issue: Discord invite links are invalid
GitHub labels: bug

[✅] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When I click on any of the Discord links on the Ragas documentation or on the Ragas' X account, I get the following message:

"This invite may be expired or you might not have the permission to join"

Also, the link in the [Get Started page ](https://docs.ragas.io/en/stable/getstarted/?h=discord) is missing.

hey @carlosgsouza thanks for pointing this out but I'm not sure why this is happening. 

the link in question https://discord.com/invite/5djav8GGNZ does seem to work and we have users joining in as well but I've had some users telling me it is not working for some of them too

could you confirm that is the link that u are facing an issue with?

Hi! That is the same link I had tried. It still doesn't work for me.

![Image](https://github.com/user-attachments/assets/3e2b63f6-7c2b-48c0-912c-80319737165c)

created a new one for you, can you check @carlosgsouza https://discord.gg/5djav8GGNZ

I'll replace the others if its working


Closing as resolved.
Repository: explodinggradients/ragas
Issue: ModuleNotFoundError: No module named 'ragas.testset.evolutions'
Root cause: Documentation Deficiency

im testing out ragas to generate questions but i keep getting this error when trying to implement the types of questions i want to generate


**Code Examples**
```py
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context

generator = TestsetGenerator.from_langchain(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings=ollama_emb
)

distribution = {simple: 0.5, reasoning: 0.25, multi_context: 0.25}
testset = generator.generate_with_langchain_docs(documents,
                                                 test_size=10,
                                                 distributions=distribution,
                                                 raise_exceptions=False)
```

gives error: ---------------------------------------------------------------------------
```
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[16], line 3
      1 # ✅ Use Groq LLMs instead of Ollama
      2 from ragas.testset import TestsetGenerator
----> 3 from ragas.testset.evolutions import simple, reasoning, multi_context
      5 generator_llm = ChatGroq(model_name="gemma2-9b-it", api_key=groq_api_key)
      6 critic_llm = ChatGroq(model_name="llama3-8b-8192", api_key=groq_api_key)

ModuleNotFoundError: No module named 'ragas.testset.evolutions
```

Hi @sahlebrahim,

Could you let me know which version of the Ragas library you're using?

getting same error with ragas Version: 0.2.13

hey @saurabh-itg @sahlebrahim these are in v0.1 and was removed in v0.2. The updated docs are given here https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/

can you try that out and let me know if it works?


@jjmachan   getting this error  while following the steps mentioned on the updated documentation from the link shared ....

Error: Not able to get any relationship among the nodes once the nodes are created ...I have used local downloaded hugging face embedding model 

![Image](https://github.com/user-attachments/assets/fb3fea85-9534-47b6-a76e-d456dcc1106a)
![Image](https://github.com/user-attachments/assets/a6a33982-cbef-4dde-be7d-9f342048efda)
Repository: explodinggradients/ragas
Issue: Retrieval metrics : improvements 
Root cause: Unimplemented Feature Gap

- [ ] Add guardrails to metrics to check for the correct format.  Score NaN if returned otherwise
- [ ] Differentiate between context relevance and precision 
Relevance -> reference free ( can be used to optimize chunking/ ranking ) 
Precision -> works with ground truth 


Hi @shahules786, thank you for these improvements, we were also a little bit confused by the different definition of context recall and precision, glad to know that it's going to be changed.
BTW, we have a question concerning the Relevance->reference free improments, what do you mean by reference free in this case?
Thanks for your response in advance.
Repository: explodinggradients/ragas
Issue: Generator Testset Portuguese Support
GitHub labels: question

module-testsetgen

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

Is there a roadmap for including support for other languages ​​in the Testset Generator? I would like to generate datasets in Portuguese



Hi @jpfcabral,

The Testset Generator already supports generating datasets in Portuguese. More details on language adaptation can be found here: [Non-English Testset Generation](https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_language_adaptation/).