zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: SimpleCriteriaScore not processing reference at all (how it is presented in Simple Criteria Scoring examples)
Root cause: Validation Gap

[X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I am creating SimpleCriteriaScore to process as it written in a documentation
```python
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import SimpleCriteriaScore


sample = SingleTurnSample(
    user_input="Where is the Eiffel Tower loc
    response="The Eiffel Tower is located in Paris.",
    reference="The Eiffel Tower is located in Egypt"
)

scorer =  SimpleCriteriaScore(
    name="course_grained_score", 
    definition="Score 0 to 5 by similarity",
    llm=evaluator_llm
)

await scorer.single_turn_ascore(sample)`
```
But it is not processing **reference**

due to the fact that ragas.metrics._simple_criteria.SimpleCriteriaScore._ascore takes in to evaluation only user_input, context and response but not reference regas scores code below as "0", even reference and response are the same.

Ragas version: 0.2.8
Python version: 3.12

**Code to Reproduce**
`from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import SimpleCriteriaScore


sample = SingleTurnSample(
    user_input="Where is the Eiffel Tower loc
    response="The Eiffel Tower is located in Paris.",
    reference="The Eiffel Tower is located in Paris"
)

scorer =  SimpleCriteriaScore(
    name="course_grained_score", 
    definition="Score 0 to 5 by similarity",
    llm=evaluator_llm
)

await scorer.single_turn_ascore(sample)`

**Expected behavior**
I expect that SimpleCriteriaScore validates response against a reference


<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Oops, just raised a fix for this https://github.com/explodinggradients/ragas/pull/1758

Hi @vklonin,

The fix has been implemented and the issue has been resolved. You can have a look at the code below. Let me know if you face any difficulty.

```python
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import SimpleCriteriaScore


llm = ChatOpenAI(model="gpt-4o-mini")
evaluator_llm = LangchainLLMWrapper(llm)

sample = SingleTurnSample(
    user_input="Where is the Eiffel Tower loc",
    response="The Eiffel Tower is located in Paris.",
    reference="The Eiffel Tower is located in Paris.",
)

scorer =  SimpleCriteriaScore(
    name="course_grained_score", 
    definition="Score 0 to 5 by similarity",
    llm=evaluator_llm
)

await scorer.single_turn_ascore(sample)
```

Output
```
5
```
Repository: explodinggradients/ragas
Issue: Incorrect evaluation results using EvaluatorChain integration
Root cause: Interface Contract Mismatch

Hello! I'm getting incorrect results using the EvaluatorChain. When using RAGAS directly with SingleTurnSample, the results are as expected. 

**Versions**
python_version: 3.12.3 

Name: ragas
Version: 0.2.8

Name: langchain-openai
Version: 0.2.12


**Code to reproduce**
```python
from ragas.integrations.langchain import EvaluatorChain
from ragas.metrics import faithfulness
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

evaluator_chain = EvaluatorChain(metric=faithfulness)
faithfulness.llm = LangchainLLMWrapper(
    ChatOpenAI(
        model_name="gpt-4o",
        temperature=0,
    )
)

sample = {
    "question": "What is the capital of France?",
    "answer": "The capital of France is Paris.",
    "contexts": [
        "Paris is the capital and most populous city of France."
    ]
}

results = evaluator_chain.invoke(sample)
```
Output
```
SCORE: 0.0 - this is incorrect
```

Code to reproduce using SingleTurnSample
```python
from ragas import evaluate 
from ragas.metrics import Faithfulness
from ragas import SingleTurnSample

llm = ChatOpenAI(
    temperature=0, 
    model_name="gpt-4o"
)
ragas_llm = LangchainLLMWrapper(llm)
sample = SingleTurnSample(
    user_input="What is the capital of France?", 
    response="The capital of France is Paris.",
    retrieved_contexts=["Paris is the capital and most populous city of France."]
)
scorer = Faithfulness(llm=ragas_llm)
await scorer.single_turn_ascore(sample)
```
Output
```
SCORE: 1.0
```



Hi @wriazati,

I looked into this issue, and found the evaluator chain expects the contexts to be in the form of list of Langchain `Document` objects. It should work fine if you pass a list of `Documents` in the contexts, like this:

```python
from langchain_core.documents import Document

sample = {
    "question": "What is the capital of France?",
    "answer": "The capital of France is Paris.",
    "contexts": [
        Document(page_content="Paris is the capital and most populous city of France.")
    ]
}

results = evaluator_chain.invoke(sample)

results
```

Output:
```
{'question': 'What is the capital of France?',
 'answer': 'The capital of France is Paris.',
 'contexts': [Document(page_content='Paris is the capital and most populous city of France.')],
 'faithfulness': 1.0}
```

We are moving away from using the Evaluation chain. You can find more details about the new approach in our [docs](https://docs.ragas.io/en/stable/howtos/integrations/langchain/)

Feel free to reach out if you have any feedback or questions about any of them!

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Version file not found
Root cause: Environment Incompatibility

[X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

I have installed ragas from git and added the path the PYTHONPATH
Trying to generate a test set using the code at: https://docs.ragas.io/en/v0.1.21/howtos/integrations/llamaindex.html
I get the error
ModuleNotFoundError: No module named 'ragas._version'

Ragas version: 0.2.8
Python version: 3.12

**Code**
from ragas.testset.synthesizers.generate import TestsetGenerator

**Error Trace** - file path to git repo shortened to PATH for privacy
Traceback (most recent call last):
  File "PATH\myScript.py", line 5, in <module>
    from ragas.testset.synthesizers.generate import TestsetGenerator
  File "PATH\src\ragas\__init__.py", line 1, in <module>
    from ragas.dataset_schema import EvaluationDataset, MultiTurnSample, SingleTurnSample
  File "PATH\src\ragas\dataset_schema.py", line 19, in <module>
    from ragas.sdk import RAGAS_API_URL, RAGAS_APP_URL, upload_packet
  File PATH\src\ragas\sdk.py", line 10, in <module>
    from ragas._version import __version__
ModuleNotFoundError: No module named 'ragas._version'

**Expected behavior**
Import modules

**Additional context**
I have a feeling this is a problem with how I installed ragas and not a bug with the module, but given that I haven't found any issues documenting this hopefully the answer can help others that make the same mistake.


Hi @ejoly3,

Have you tried reinstalling the ragas module?

```
pip install --upgrade ragas
```

If the issue persists, you can try running these commands, which helped resolve it for me:

```
pip install -U setuptools
pip install -e ".[dev]"
```

Let me know how it goes for you.

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: Unable to pass an evaluation criteria to a SimpleCriteriaScore
Root cause: Documentation Deficiency

[ X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Unable to pass an evaluation criteria to a SimpleCriteriaScore

Ragas version: 0.2.8
Python version: 3.12


**Expected behavior**
I want to set up my own simple validation in SimpleCriteriaScore but I am unable to, because in a code it is rewrited with some hardcoded validation criteria


self._definition = definition
**self.single_turn_prompt = single_turn_prompt or SingleTurnSimpleCriteriaPrompt()** - i may pass my instruction here
**self.multi_turn_prompt = multi_turn_prompt or MultiTurnSimpleCriteriaPrompt()** -  i may pass my instruction here

_update the instruction for the prompts with the definition_
```python
instruction = f"Evaluate the Input based on the criterial defined. Give a score between 0 and 5.\nCriteria Definition: **{self._definition}**"
self.single_turn_prompt.instruction = instruction
self.multi_turn_prompt.instruction = instruction
```
I also may pass my instruction in definition but it is also not working because of this instruction = f"Evaluate...


<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hi @vklonin,

We have a section in our [docs](https://docs.ragas.io/en/stable/howtos/customizations/metrics/_modifying-prompts-metrics/#modifying-instruction-in-default-prompt) that explains in detail how to modify the existing prompts for the Evaluation scorers. We have also made some changes in SimpleCriteriaScore metric code. Please have a look and let us know if the problem still persists.

You can use the following code as an example:

```python
scorer = SimpleCriteriaScore(
    name="",
    definition="",
    llm=evaluator_llm,
)

prompt = scorer.get_prompts()["single_turn_simple_criteria_prompt"]
prompt.instruction =   # your instruction here
scorer.set_prompts(**{"single_turn_simple_criteria_prompt": prompt})
```
Repository: explodinggradients/ragas
Issue: Error in generating test set
Root cause: Environment Incompatibility

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

Ragas version:
Python version:

**Code to Reproduce**
Share code to reproduce the issue
```py
import os
os.environ["OPENAI_API_KEY"] =“xxx”

from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI
from langchain_openai import OpenAIEmbeddings
generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
generator_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

from langchain.document_loaders import PyPDFLoader
loader = PyPDFLoader("E:\youcai\dingweikel\A backlight and deep learning based method for calculating the number of seeds per silique.pdf")
documents = loader.load()
docs = loader.load()

from ragas.testset import TestsetGenerator
generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
```

**Error trace**
![3](https://github.com/user-attachments/assets/ccf1f77b-c80f-4aa8-a053-6fd6fbc2a78c)



**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Can you share which version of ragas are you using @13874360430 

Uninstall first, then reinstall the new version
&nbsp;


1534931699
***@***.***



&nbsp;




------------------&nbsp;原始邮件&nbsp;------------------
发件人: ***@***.***&gt;; 
发送时间: 2025年1月4日(星期六) 晚上7:33
收件人: ***@***.***&gt;; 
抄送: ***@***.***&gt;; ***@***.***&gt;; 
主题: Re: [explodinggradients/ragas] Error in generating test set (Issue #1764)














 
I am facing the same issue, when I remove all the arguments it produces the following error
我面临同样的问题，当我删除所有参数时，它会产生以下错误
 "TypeError: TestsetGenerator.init() missing 4 required positional arguments: 'generator_llm', 'critic_llm', 'embeddings', and 'docstore'"
“TypeError： TestsetGenerator.init () 缺少 4 个必需的位置参数：'generator_llm'、'critic_llm'、'embeddings' 和 'docstore'”
 
It could be an old version of ragas. If so, how do I update to the new version?
它可能是拉格斯的旧版本。如果是这样，我该如何更新到新版本？
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
直接回复此电子邮件、在 GitHub 上查看或取消订阅。
You are receiving this because you were mentioned.
您收到此邮件是因为有人提到了您。  Message ID: ***@***.***&gt;

@13874360430 was this fixed for you?

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: Rubrics criteria scoring does not prompt a score per rubric criterium
Root cause: Algorithmic Error

[X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Rubrics scoring as described in the [doc page](https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/general_purpose/) does not prompt all the rubrics as described for the RubricsScore and InstanceRubricsScore. It will not actually prompt the scoring of each rubric criterium and average the score.

It will only do one prompt and collect one score.

Ragas version: 0.2.7
Python version: 3.11

**Code to Reproduce**
From documentation:

```
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import InstanceRubricsScore


SingleTurnSample(
    user_input="Where is the Eiffel Tower located?",
    response="The Eiffel Tower is located in Paris.",
    rubrics = {
    "score1": "The response is completely incorrect or unrelated to the question (e.g., 'The Eiffel Tower is in New York.' or talking about something entirely irrelevant).",
    "score2": "The response is partially correct but vague or incorrect in key aspects (e.g., 'The Eiffel Tower is in France.' without mentioning Paris, or a similar incomplete location).",
    "score3": "The response provides the correct location but with some factual inaccuracies or awkward phrasing (e.g., 'The Eiffel Tower is in Paris, Germany.' or 'It is located in Paris, which is a country.').",
    "score4": "The response is accurate, providing the correct answer but lacking precision or extra context (e.g., 'The Eiffel Tower is in Paris, France.' or a minor phrasing issue).",
    "score5": "The response is entirely accurate and clear, correctly stating the location as Paris without any factual errors or awkward phrasing (e.g., 'The Eiffel Tower is located in Paris.')."
}
)

scorer =  InstanceRubricsScore(llm=evaluator_llm)
await scorer.single_turn_ascore(sample)
```

**Error trace**
Step through the code and only 1 LLM scoring call is actually made.

**Expected behavior**
The rubrics scoring are evaluated separately and averaged as described.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hey @GillesJ,

We have made the necessary changes to both the code and documentation to fix the issue. Let me know if it works for you.
Repository: explodinggradients/ragas
Issue: Documentation (Un)stable?
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Question**
Is it just me or the documentation especially the import statements are not working for me.
This is supremely frustating i am trying to install it again for the third time hoping it would work. (Third time's the charm)

**Additional context**
Anything else you want to share with us? 


Same, I've been able to find a lot of the functions in the import statements elsewhere in the source code. I've had a lot of other bugs that I haven't been able to fix come up though.

Hi @tiwarikaran @ejoly3,

We are really sorry for the trouble, and we completely understand how this can be frustrating. It would be very helpful if you could share a few examples where this issue is happening. This will help us fix it quickly.

![Image](https://github.com/user-attachments/assets/dc909bd2-6d1b-47d8-a7be-be84a0950253)


Does this image help

Hi @tiwarikaran,

You can try importing it like this:

```python
from ragas.dataset_schema import SingleTurnSample
```

Hey @tiwarikaran Can you share the output of 
`pip freeze | grep ragas`

And if this is 0.1.x please do `pip install -U ragas`. The reason is that you may be using old version of ragas. 

> from ragas.dataset_schema import SingleTurnSample

I tried doing that here's the output

![Image](https://github.com/user-attachments/assets/88934c54-3fa3-4b20-9433-679c1da68ce0)



> pip freeze | grep ragas

using this version of ragas atm 

ragas==0.1.22

hey @tiwarikaran since you are using an older version do check the older docs? https://docs.ragas.io/en/v0.1.21/

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Factual Correctness F1, Precision and Recall at the same time.
Root cause: Unimplemented Feature Gap

**Describe the Feature**
The Factual Correctness metric takes a long time to compute and it is often interesting to inspect precision and Recall in addition to F1. That's why it would be interesting if instead of making 3 different calls to the same metric, it could be done in one call and return all 3 at the same time. I have seen that something similar happens with metrics like Rouge, but in that case it is not so necessary because it takes much less time.

**Why is the feature important for you?**
Because for a testset of only 50 questions it takes about 5 minutes to run these three metrics, so it is not scalable, nor is it compatible with other metrics that also take longer. For example, running the same with 10 metrics can take up to 1 hour. Whenever you can simplify LLM calls I think it is crucial to do so.

Thanks you a lot!


Hi @Javier-Jimenez99,

Thank you for sharing your suggestion! Given your use case, it might be more efficient for you to implement a custom metric to calculate Precision, Recall, and F1 in a single call. You can find detailed guidance on how to create custom metrics in the [documentation](https://docs.ragas.io/en/stable/howtos/customizations/metrics/_write_your_own_metric/#custom-metrics).

I hope this helps! Please let me know if you have any questions or need further assistance.

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: Answer correctness not working
Root cause: Validation Gap

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
Answer correctness not working

Ragas version:
Python version:

**Code to Reproduce**
```
from datasets import Dataset 
from ragas.metrics import faithfulness, answer_correctness
from ragas import evaluate

data_samples = {
    'question': ['What is France capital?', 'What is India capital?'],
    'answer': ['France capital is Delhi', 'India capital is spain'],
    'ground_truth': ['France capital is Paris.', 'India capital is Delhi']
}
dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset,metrics=[answer_correctness])
score.to_pandas()
```
**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


You didn't specify the error type, Python version, or RAGAS version, so I can't tell what it is, but if it's a "metaclass conflict" issue, it should be fixed by upgrading the langchain library.

Error Type in VertexAI
`TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases`

Solve
`pip install --upgrade ragas langchain`

thanks a lot of the helping out @hundredeuk2 🙌🏽 

@ekaanshkhosla were you able to solve it? could you share the error trace if it was not solved?

Closing after 8 days of waiting for the additional info requested.
Repository: explodinggradients/ragas
Issue: How to generate out of context questions to evaluate the hallucination of a LLM 
Root cause: Documentation Deficiency

[*] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
To evaluate the hallucination of a LLM, we would like to generate several out of context questions using Ragas. I checked the doc and wonder if I could use the persona concept as follows. 

**Code Examples**
```python

from ragas.testset.persona import Persona

persona_external = Persona(
    name="External",
    role_description="Don't know much about the company and ask a lot of unrelated questions.",
)

```




hey @parkerzf I think you could modify testset generation to get this

@shahules786 what do you think?
Repository: explodinggradients/ragas
Issue: making convert_to_ragas_messages  capable of dealing with tool call AIMessages that have List as argument  
Root cause: Validation Gap

**Describe the Feature**
`convert_to_ragas_messages ` func in Ragas uses  "r.ToolCall" when an lanchain AIMessage is a tool call. And the func deals only with tool calls that have as args (str, int, float) and throws pydantic error when it comes to another type.

- So it would be perfect if we have `convert_to_ragas_messages ` func deals also with tool calls that have args of  type `List `

**Why is the feature important for you?**
Because in alot of usecases we have tools in langchain that passes List type to there args  and not having support prevent from eval important tool calls.

**Code**
```
[
HumanMessage(content='PRODUCT REF 1243', additional_kwargs={}, response_metadata={}, name='user'),
AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_8ByB2q1Dp7lb6VkYjOu5aWHZ', 'function': {'arguments': '{"id_search_queries":["PRODUCT REF 1243"]}', 'name': 'getter'}, 'type': 'function'}]}, ....)
]


ragas_messages = convert_to_ragas_messages(state["messages"])
sample = MultiTurnSample(user_input=ragas_messages)
```

**Additional context**
The code it will throw an error if its excuted and thats because of the tool call args are type List and not  (str, int, float) .

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


Hi @AMMISAIDFaical,

I hope you're doing well. We've implemented the changes you requested. Please feel free to let me know if you encounter any further issues.
Repository: explodinggradients/ragas
Issue: How to use RagasEvaluatorChain in the latest version
Root cause: Documentation Deficiency

In version V0.2.8, using from ragas.langchain.evalchain import RagasEvaluatorChain will result in an error message saying ModuleNotFoundError: No module named 'ragas.langchain'

hey @pengxin233 we are actually moving away from using the evaluator chain. can you check the updated docs for this. 

https://docs.ragas.io/en/stable/howtos/integrations/langchain


let me know if there is any further issues with this 🙂 (really sorry about the delay!)

It seems the issue was answered, closing this now.