zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: How to integrate Ragas with Ragflow for evaluation?
GitHub labels: question

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

How to integrate Ragas with Ragflow for evaluation? Is there any documentation I can look at? 




<!-- Greeting -->
Hey @RyanTree-HS! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

<!-- Answer -->
I couldn't find specific information on integrating Ragas with Ragflow in the Ragas documentation or related resources. It seems there might not be direct guidance or examples available for this integration within the repository. You might need to explore the general components and modules involved in the Ragas evaluation process, such as metrics, language models, and the executor, to see how they can be adapted for use with Ragflow <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/evaluation.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/components/index.md)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: Answer Correctness not working after pydantic update
Root cause: External Dependency Breakage

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

Ragas version: 0.2.0
Python version: 3.12

**Error trace**
AttributeError('StatementsAnswers' object has no attribute 'dicts')

**Additional context**
The dicts method was deleted from the StatementAnswers class after the pydantic update 95f8d7bf590f19f1d0fc302d3e64556ae5e49f18

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
@hubkrieb Hello there! I'm here to help you with any bugs, questions, or contributions you might have. Let's tackle this together!

<!-- Answer -->
To resolve the `AttributeError` caused by the removal of the `dicts` method from the `StatementsAnswers` class after the pydantic update in Ragas version 0.2.0 with Python 3.12, you should replace the `dicts` method with the `model_dump` method. The `model_dump` method is used to serialize the model to a dictionary, which is the intended functionality of the removed `dicts` method.

Here's how you can modify the code:

Replace any instance of `answers.dicts()` with `answers.model_dump()`.

For example, in the `_compute_score` method, change:

```python
faithful_statements = sum(
    1 if answer.verdict else 0 for answer in answers.model_dump()
)
num_statements = len(answers.model_dump())
```

Additionally, ensure that any other usage of `dicts()` in the code is replaced with `model_dump()`, as seen in the `NoiseSensitivity` class methods like `_evaluate_statement_faithfulness` and `_decompose_answer_into_statements` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_faithfulness.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_noise_sensitivity.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: Faithfulness Errors
Root cause: Validation Gap

 - [x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I got the following error.

ERROR:ragas.executor:Exception raised in Job[9]: AttributeError('NoneType' object has no attribute 'generate')

**Code Examples**
```
import openai
from ragas.experimental.metrics._faithfulness import FaithfulnessExperimental
from ragas.metrics.base import get_segmenter
from ragas.llms import llm_factory
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
)

LLM_NAME = "gpt-4o-mini"
ragas_llm = llm_factory(model=LLM_NAME)

# データセットの準備
dataset = Dataset.from_dict(
    {
        "question": questions,
        "answer": answers,
        "contexts": contexts,
        "ground_truth": ground_truths,
    }
)

# Faithfulnessの言語を指定
faithfulness = FaithfulnessExperimental()
faithfulness.sentence_segmenter = get_segmenter(language="japanese", clean=False)
faithfulness.llm = ragas_llm

# Use the adapted faithfulness metric in your evaluation
score = evaluate(
    dataset,
    llm=ragas_llm,
    metrics=[
        faithfulness,
        ],
                 )
print(score)
```


<!-- Answer -->
The error you're encountering, `'NoneType' object has no attribute 'generate'`, suggests that the `llm` object might not be properly initialized or set. Here are a few steps to troubleshoot and resolve the issue:

1. **Check LLM Initialization**: Ensure that `ragas_llm` is correctly initialized by the `llm_factory` function. Verify that the `llm_factory` function is returning a valid LLM instance and not `None`.

2. **Verify LLM Assignment**: Make sure that `faithfulness.llm = ragas_llm` is correctly assigning the LLM to the `FaithfulnessExperimental` instance. The LLM should not be `None` at this point.

3. **Inspect `llm_factory`**: If `helicone` configuration is enabled, ensure that all necessary parameters like `default_headers` and `base_url` are correctly set and not `None` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/base.py)</sup>.

4. **Debugging**: Add print statements or use a debugger to check the state of `ragas_llm` after initialization to confirm it is not `None`.

By following these steps, you should be able to identify and fix the source of the error.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu @dosubo
I have almost similar code with the same model gpt 4o mini  and i counter the same error and when i checked what is inside llm_factor(model= MODEL_NAME) i found that
 LangchainLLMWrapper(run_config=RunConfig(timeout=180, max_retries=10, max_wait=60, max_workers=16, exception_types=<class 'openai.RateLimitError'>, log_tenacity=False, seed=42))
Repository: explodinggradients/ragas
Issue: Is there any example using local open source LLM model for evaluation?
Root cause: Documentation Deficiency

Using openAI api might not be feasible for some environment, could someone provide any reference or example link on running evaluation with local LLM model/?

<!-- Answer -->
Yes, there are examples and documentation for evaluating RAG pipelines using local models in RAGAS. You can refer to the article by Atita Arora titled ["Evaluating Retrieval Augmented Generation using RAGAS"](https://superlinked.com/vectorhub/articles/retrieval-augmented-generation-eval-qdrant-ragas) for an end-to-end tutorial. Additionally, the RAGAS documentation includes a notebook titled "Evaluating RAG pipelines with Openlayer and Ragas," which provides a step-by-step guide for setting up and evaluating RAG pipelines with local models <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/community/index.md)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/openlayer.ipynb)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: what langchain version should I use? got below issue.
Root cause: External Dependency Breakage

   from ragas.metrics.answer_relevance import AnswerRelevancy, answer_relevancy
  File "####\Python\Python311\Lib\site-packages\ragas\metrics\answer_relevance.py", line 10, in <module>
    from langchain.embeddings.base import Embeddings
ModuleNotFoundError: No module named 'langchain.embeddings.base'


my langchain version 0.0.261

i managed well with 0.0.288

seems like, we have to check the dependencies. Will do that and fix it. 

Thanks for bringing this up 🙂, much appreciated!
Repository: explodinggradients/ragas
Issue: Where ground truth comes from?
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Where does ground truth come from?

**Additional context**
I'm building a data set to run evaluations with RAGAS but I don't understand where ground truth comes from. Is this a human-annotated property? I understand the ground truth in RAGAS refers to the correct answer to the question. It's typically used for the context_recall metric. But how to we get this? Human in the loop? LLM generated? More documents from the retrieval? Thank you?


<!-- Greeting -->
Hey @jersonmorrow! I'm here to help you with any questions or issues you have. Let's tackle this together!

<!-- Answer -->
The ground truth in RAGAS evaluations comes from the expected answer for each question. It is typically a human-annotated property, representing the correct answer for each question in your dataset <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/applications/data_preparation.md)</sup>. 

For generating ground truth data, RAGAS recommends using an evolutionary generation paradigm, which includes methods like reasoning, conditioning, multi-context, and conversational approaches to ensure diverse and comprehensive evaluation datasets <sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/testset_generation.md)</sup>. 

These methods can involve human input, but they can also be enhanced with techniques like using LLMs or additional documents to refine the ground truth.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: The documentation is not up to date with the code
Root cause: Documentation Deficiency

[ x ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

The documentation is not on par with the code : https://docs.ragas.io/en/stable/howtos/customisations/ragas_custom_model.html

Ragas version: 0.1.21
Python version: 3.10

**Code to Reproduce**

```
from ragas.testset.evolutions import ComplexEvolution
```

**Error trace**

```
from ragas.testset.evolutions import ComplexEvolution
ModuleNotFoundError: No module named 'ragas.testset.evolutions'
```


**Expected behavior**

A successful import.

**Additional context**
 

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Answer -->
There are no specific mentions of the `ComplexEvolution` class or the `ragas.testset.evolutions` module in the available information for version 0.1.21. However, a similar issue was reported where a `ModuleNotFoundError` occurred due to naming conflicts or missing modules. 

One solution for a similar issue was to ensure that your script name does not conflict with the package name (e.g., renaming `ragas.py` to something else) <sup>[[1]](https://github.com/explodinggradients/ragas/issues/1080)</sup>.

Additionally, verify that the `ragas` package is correctly installed. You can try reinstalling it using:

```bash
pip install ragas
```

Or from the source:

```bash
pip install git+https://github.com/explodinggradients/ragas
```

If the module is still missing, it might be worth checking if there have been any recent changes or updates in the repository that could have affected the module's availability.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu my problem is that the documentation in https://docs.ragas.io/en/stable/howtos/customisations/ragas_custom_model.html states that the code should be 

```python
from ragas.testset.evolutions import ComplexEvolution
```
but it does not work.

Notifiy this to maintainers.

<!-- Answer -->
It seems like this issue requires a deeper look, @jjmachan could you please lend your expertise?


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


hey @nprime496 I'm not sure there is a problem - justed added a test to confirm

https://github.com/explodinggradients/ragas/pull/1460
Repository: explodinggradients/ragas
Issue: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
Root cause: External Dependency Breakage

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
when i import library from ragas i get 
metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

**Code Examples**
from ragas.metrics import faithfullness
from ragas import evaluate

**Additional context**
What version of ragas i can use ?


<!-- Greeting -->
Hey @vohoangkh4ng! I'm here to help you out with any bugs or questions you have. Let's tackle this issue together!

<!-- Answer -->
The error you're encountering is related to a metaclass conflict, which can occur when a class inherits from multiple base classes with different metaclasses. Unfortunately, I don't have specific information on metaclass conflicts or version requirements for the Ragas library.

To resolve this, you can try the following steps:

1. **Check for Updates**: Ensure you are using the latest version of Ragas. Sometimes, such conflicts are resolved in newer releases.

2. **Review Dependencies**: Check if there are any specific dependencies or Python version requirements that might affect metaclass behavior.

3. **Simplify Imports**: Try importing only the necessary components to isolate the issue.

If these steps don't help, consider checking the Ragas repository for any recent commits or issues related to metaclass conflicts.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


I encountered this issue too after migration to `langchain-core==0.3.x` and switching from `Pydantic` v1 to v2. It would be great if Ragas could be updated to be compatible.

I am getting the same error but with Django viewsets

@VMinB12 @Mukhsin0508 are you guys using v0.2+
we do support the libs you mentioned now with that update

@vohoangkh4ng 
There's a typo in your import: you wrote `faithfullness` instead of `faithfulness`

```
# Solution 1: Clean installation with specific versions
# First, uninstall existing packages
pip uninstall ragas langchain datasets -y

# Install specific versions known to work together
pip install ragas==0.0.22
pip install langchain==0.0.350
pip install datasets==2.14.7

# Your imports should now work
from ragas.metrics import faithfulness  # Note the correct spelling
from ragas import evaluate
```

> Check your Python version (Ragas works best with Python 3.8-3.10)
> b. Make sure you don't have conflicting packages in your environment
> c. Try running:
```
pythonCopypip freeze | grep -i "ragas\|langchain\|datasets"

```

thanks alot @Mukhsin0508 for the fix ❤️ 
closing this now
Repository: explodinggradients/ragas
Issue: when will ragas 0.2 be released ?
GitHub labels: question

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**

when will ragas 0.2 be released (approximatively) ?

**Additional context**

I use a lot of functionalities from `langchain>0.3` and ragas bounds me to use a lesser version. I tried and it's too much work backwards. I'd like to know when I can check the repo again. Also, 0.2 adds too much modifications to work with even the unstable branch.

<!-- Answer -->
I couldn't find any specific information about the release date for Ragas 0.2 in the available data. It seems there are no details on upcoming versions or features in the repository documentation or related resources. You might want to keep an eye on the repository for any updates or announcements regarding future releases.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: Supply a notebook with open source llm (e.g. llama 2)
Root cause: Documentation Deficiency

Can you please supply a notebook that demonstrates how to use ragas with open source llm ?

Thanks,
Eyal

Hey Eyal, thanks for bringing this up

We do have plans for this and is in our roadmap but it might take a while.  All our testing is with OpenAI so we can't ensure accuracy with opensource models. That being said I was wondering what were the reasons you wanted open-source models and how big of a model can you run at your end. Also, would you be okay running a ~30B model for evaluations, do you have a hosted version available right now?

This will help us get more sense of how to prioritise this

Hello,
I am in a similar situation, as I am building a RAG pipeline but I cannot use the OpenAI key (due to cost reasons); I am using a quantized version of Llama2 that fits in the free version of Colab. 
The idea I had was to use the same model for generation and evaluation, aware that the results won't be that accurate as in case I was using GPT3.5/GPT4, but still in some ways informative for optimization.  
Looking here (https://github.com/explodinggradients/ragas/blob/main/docs/guides/llms.ipynb) I thought it was possible to do that by changing the llm called by the metrics, but looking at your comment I don't know if it is convenient (?)
Do you have any suggestions? 

Hey @Robs1999, full support for opensource LLMs will be coming shortly, I'll update this once its in

Thanks a lot for the update @jjmachan! I was wondering if you could give us a rough idea of when we might expect full support for open-source language models? Even a rough estimate would be really helpful! 
Thanks again, and keep up the great work! 😊🚀

appologies for the delay but we have shipped this feature you can check the docs here https://docs.ragas.io/en/latest/howtos/customisations/llms.html#evaluating-with-open-source-llms

do check it out and feel free to open it this doesn't address the issue completely. 

> appologies for the delay but we have shipped this feature you can check the docs here https://docs.ragas.io/en/latest/howtos/customisations/llms.html#evaluating-with-open-source-llms
> 
> do check it out and feel free to open it this doesn't address the issue completely.

The above link forwards to 404 page. Do we have integrations supported with non OpenAI models now?

@manojkumarvohra do checkout this doc [Bring Your Own LLMs and Embeddings | Ragas](https://docs.ragas.io/en/stable/howtos/customisations/bring-your-own-llm-or-embs.html)
Repository: explodinggradients/ragas
Issue: Traditional NLP Metrics examples in docs are broken
Root cause: Documentation Deficiency

[X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
The examples from the '[Traditional NLP Metrics](https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/traditional/)' don't work on the current version.

Ragas version: 0.1.21
Python version: 3.11.10

**Code to Reproduce**
```python
from ragas.dataset_schema import SingleTurnSample
```
or
```python
from ragas.metrics._bleu_score import BleuScore
```


**Error trace**

```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ragas.dataset_schema'
```
and
```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ragas.metrics._bleu_score'
```

**Expected behavior**
Modules are imported successfully


**Additional context**
This works on `ragas==0.1.18`

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hey @dschuler36! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together!

<!-- Answer -->
The issue you're encountering with the 'Traditional NLP Metrics' examples in Ragas version 0.1.21 is due to missing modules `ragas.dataset_schema` and `ragas.metrics._bleu_score`. This problem arises because these modules or classes might have been removed or renamed in the latest version.

To resolve this, check the latest documentation or release notes for any changes in module structure or class names. You might need to adjust your imports based on the updated module paths or use an earlier version like `ragas==0.1.18` where these imports work as expected <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_bleu_score.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: Creating Testdataset from dictionary/json file
Root cause: Unimplemented Feature Gap

**Describe the Feature**
In some cases I think it would be interesting to generate a testdataset variable from a json file or dictionary with the necessary keys (question, answer, context...) in order to also evaluate LLMs on human-generated Testdataset for specific topics.

**Why is the feature important for you?**
Because I am implementing RAGAS on a really specific topic and some tests have to be carried out on exames generated by specialists in the company.

**Additional context**
I would be able to generate the code and a Pull request. Maybe the feature is already implemented but I have not found any method "from_json" or similar.


Hey, this is something we will be working on immediately after 0.2 release. I would love to know more about your request so that we can use this while adding the feature. 
If you have some time, please book a slot https://cal.com/shahul-ragas/30min
Repository: explodinggradients/ragas
Issue: Persist test set sample questions to TestSetGenerator object so as to not lose work
Root cause: Unimplemented Feature Gap

**Describe the Feature**
I am currently on a tier 1 account with Anthropic, which means I have a daily cap of 1M tokens (total - in and out).  When trying to generate a testset of 100 questions, I hit that daily token limit and the `generator.generate` call raises a RateLimitError.  However, the data is not preserved, which means I have now paid for 1M tokens but critically _do not have access to the results_.

Therefore, as a developer I would like the generated test questions to be persisted to the TestSetGenerator object as they are received so that I can access the testset up to the point that the Exception occurred.  Even better would be to catch the Exception and enter a "pause" state so that the next day I could run something like `generator.continue()` and pick up where I left off.


**Why is the feature important for you?**
See above

**Additional context**
Add any other context about the feature you want to share with us.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Hey @ahgraber thanks for bringing this up. As part of 0.2 we are introducing new test generation algorithm. We will add this to our immediate roadmap for improvements after 0.2 release. 
https://docs.ragas.io/en/latest/getstarted/rag_testset_generation/

Amazing!  I'm really looking forward to experimenting with v0.2

Hey @ahgraber It's out. Please let us know if you need any help with it.