zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: error when apply_transformers
Root cause: Validation Gap

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

apply_transforms never success if anything went wrong in the whole processing..

This code is just copy from the document, only llm model and embedding model changed.
takes six hours and 10 dollars to run, but got nothing!!!

remind, not all steps fails, only part of them, but I got nothing finally.

If everything ok is required to finish the whole task, you should stop at the first failure, not run to the world's ending, wasting my time and money, for nothing!

<img width="1146" alt="Image" src="https://github.com/user-attachments/assets/833e91e4-f670-4ce7-a43d-504be6849c48" />

<img width="1283" alt="Image" src="https://github.com/user-attachments/assets/98777e19-84a5-4cba-90b9-87a17e45ec65" />

<img width="1397" alt="Image" src="https://github.com/user-attachments/assets/1247dff5-6525-40ea-93fd-00c291069240" />

<img width="1385" alt="Image" src="https://github.com/user-attachments/assets/ea5288e9-5c2a-47ab-98c2-d2f8d1a21473" />

Ragas version:
Python version:
3.11

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.

If everything ok is required to finish the whole task, you should stop at the first failure, not run to the world's ending, wasting my time and money, for nothing!


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


I use the stable version of ragas, seems to be v0.2.14
Repository: explodinggradients/ragas
Issue: Testset(samples=[])
Root cause: Documentation Deficiency

I executed the following code, but the generated dataset is empty. What could be the issue? My Ragas version is 0.2.14.

code{from ragas.testset import TestsetGenerator

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
}


Hi @txygjx,

The issue you're facing might be due to missing or incorrect transformations when Ragas creates the knowledge graph. Here's a simple overview of how Ragas generates test sets:

First, Ragas builds a knowledge graph from your documents, consisting of nodes and relationships. From this graph, Ragas generates test questions.

There are two main types of questions:
- Single-hop questions: Simple questions based directly on individual nodes.
- Multi-hop questions: More complex questions created by connecting multiple nodes through relationships.

If your dataset is empty, the knowledge graph might lack the necessary properties or relationships to generate questions.

We are preparing a detailed tutorial on this topic, which will soon be available here:

- https://ragas--1959.org.readthedocs.build/en/1959/howtos/applications/singlehop_testset_gen/ (currently not live, coming soon)
- https://github.com/explodinggradients/ragas/pull/1959

I hope this helps!
Repository: explodinggradients/ragas
Issue: Strange calculation of Recall in Factual Correctness
Root cause: Algorithmic Error

When calculate factual correctness because of fuzzy matching, the following could happen: 

4 claims in response
2 claims in ground truth 

Let's say 2 claims in response are good, and 2 are not. Conclusion `TP=2`, `FP=2`. 
Then from 2 claims in ground truth 1 is good and 1 is not. Code correctly calculates `FN=1`. 

And 
`recall  = 2/(2+1) = 2/3 `

But, if total number of claims in ground truth is 2, how could `recall` faction have denominator of "3". With recall we ask how many claims in ground truth were covered, and in this case it is 1, so recall would be 1/2 I would say. The 3 above comes by mixing TP calculated for the precision.  


If you take the example with the sun given in answer correctness class https://github.com/explodinggradients/ragas/blob/d5da272c94b2adb5ba60a8583b024027244f46b6/src/ragas/metrics/_answer_correctness.py#L59

We would come to same: 

2 claims in answer
5 claims in ground truth 

TP=1, FP=1, FN=5, recall = 1/6 ... but is it? I would argue it is 0/5 in this case. 

I would love to hear some thoughts on this.
Repository: explodinggradients/ragas
Issue: Non-English QA generation is not working
Root cause: Documentation Deficiency

according to the document
https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_language_adaptation/

```python
import asyncio

from llama_index.core import SimpleDirectoryReader
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.openai_like import OpenAILike
from ragas.embeddings import LlamaIndexEmbeddingsWrapper
from ragas.llms import LlamaIndexLLMWrapper
from ragas.testset import TestsetGenerator
from ragas.testset.persona import Persona
from ragas.testset.synthesizers.single_hop.specific import (
    SingleHopSpecificQuerySynthesizer,
)
from ragas.testset.transforms.extractors.llm_based import NERExtractor
from ragas.testset.transforms.splitters import HeadlineSplitter

from settings import settings

# set_global_handler("simple")

documents = SimpleDirectoryReader(
    "./Sample_non_english_corpus"
).load_data()

generator_llm = LlamaIndexLLMWrapper(
    OpenAILike(
        model=settings.OPENAI_MODEL,
        api_key=settings.OPENAI_API_KEY,
        api_base=settings.OPENAI_BASE_URL,
        is_chat_model=True,
    )
)

generator_embeddings = LlamaIndexEmbeddingsWrapper(
    OllamaEmbedding(
        model_name=settings.EMBED_MODEL,
    )
)

personas = [
    Persona(
        name="curious_student",
        role_description="A student who is curious about the world and wants to learn more about different cultures and languages",
    ),
]

transforms = [HeadlineSplitter(), NERExtractor(llm=generator_llm)]

generator = TestsetGenerator(
    llm=generator_llm,
    embedding_model=generator_embeddings,
    persona_list=personas,
)

distribution = [
    (SingleHopSpecificQuerySynthesizer(llm=generator_llm), 1.0),
]
for query, _ in distribution:
    prompts = asyncio.run(query.adapt_prompts("chinese", llm=generator_llm))
    query.set_prompts(**prompts)


testset = generator.generate_with_llamaindex_docs(
    documents,
    testset_size=5,
    transforms=transforms,
    query_distribution=distribution,
)

# df = testset.to_pandas()
# print(df.head())

dataset = testset.to_evaluation_dataset()
print("Query:", dataset[0].user_input)
print("Reference:", dataset[0].reference)

# os.makedirs("outputs", exist_ok=True)
# df.to_excel(
#     f"outputs/medical_insurance_testset-{datetime.now().strftime('%Y%m%d%H%M%S')}.xlsx"
# )
```
Repository: explodinggradients/ragas
Issue: Error: Attribute error:'float' object has no attribute 'response'
Root cause: Validation Gap

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

I am using llama index with azure open ai to evaluate
When I am calling evaluate function with query engine,llm,dataset I am getting attribute error mentioning 'float' object has no attribute "response error" with an exception raised in job[1]: AttributeError('NoneType' object has no attribute 'search')

from ragas.integrations.llama_index import evaluate

Result=evaluate (query_engine,metrics,dataset)

![Image](https://github.com/user-attachments/assets/891ba15b-6098-4781-9639-d6c7c3e6d400)

![Image](https://github.com/user-attachments/assets/2859eb77-60a7-448f-b1bb-93668895556a)
Repository: explodinggradients/ragas
Issue: Questions About Metrics Changes from Ragas v0.1 to v0.2
Root cause: Documentation Deficiency

[V] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I have compared the descriptions of metrics in the documentation for Ragas [v0.1](https://docs.ragas.io/en/v0.1.21/concepts/metrics/) and [v0.2](https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/) and have some questions:  

1. If all v0.1 metrics are [still supported](https://docs.ragas.io/en/stable/howtos/migrations/migrate_from_v01_to_v02) in v0.2, why are metrics like "Answer Relevance" and "Context Utilization" not mentioned in the v0.2 documentation? Is this an oversight in the documentation, or are these metrics no longer supported in v0.2?  

2. Building on the first question, could it be that some metrics have simply been renamed? For example, is "Answer Relevance" in v0.1 equivalent to "Response Relevancy" in v0.2? If so, could you clarify which metrics have been renamed and what their new names are?  

3. In v0.2, metrics are categorized under different groups (e.g., RAG, Nvidia, General Purpose). I understand this may be for better organization due to the expanded functionality, but have the definitions of any metrics changed as a result of this re-categorization?  

4. I have found two different images explaining the metrics. Could you confirm which one is correct or up-to-date?  

|  Metrics | V0.1  | V0.2 |
|  ----  | ----  | ----  |
| Faithfulness | V | V(RAG)|
| Answer Relevance | V | |
| Context Precision | V | V(RAG)|
| Context utilization | V | |
| Context Recall | V | V(RAG)|
| Context entities recall | V | V(RAG) |
| Noise Sensitivity | V | V(RAG)|
| Answer semantic similarity | V | |
| Answer Correctness | V | |
| Answer Critique | V | V(General Purpose)|
| Domain Specific Evaluation | V | |
| Summarization Score | V | |
| Response Relevancy |  | V(RAG) |
| Answer Accuracy |  | V(Nvidia) |
| Context Relevancy |  | V(Nvidia) |
| Response Groundedness |  | V(Nvidia) |
| Factual Correctness |  | V(Natual Language Comparison) |
| Semantic Similarity |  | V(Natual Language Comparison) |

![Image](https://github.com/user-attachments/assets/b9c8fbd4-e229-46b6-ae4a-701bf16588da)

![Image](https://github.com/user-attachments/assets/6dc70966-e644-4862-baf5-0467db4aefbf)


Thank you for your clarification!
Repository: explodinggradients/ragas
Issue: BaseRagasLLM overriding temperature results in PydanticPrompt.generate generating identical responses due to temperature being mapped to effectively 0 when n is 1
Root cause: Algorithmic Error

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When using ragas to evaluate answer relevancy metric, all generated questions are identical. 
I have tracked this down across ResponseRelevancy._ascore -> PydanticPrompt.generate -> BaseRagasLLM.generate. 
The temperature gets set to effectively 0 if n = 1, but n IS ALWAYS 1 if PydanticPrompt.generate is being used.

A potential solution to this problem would be for _ascore to use .generate_multiple, instead of looping using range and using multiple OF .generate.

That still leaves me wondering, why the temperate from the LLM that gets passed to ragas is not respected, and 0.3 or 1e-8 is being used based on n?

Ragas version: 0.2.14
Python version: 3.12.3

**Code to Reproduce**
Run ragas.evaluate using ragas.metrics.answer_relevancy as metric.

**Expected behavior**
Generating varied questions.
Repository: explodinggradients/ragas
Issue: Ensure question variedness when using ResponseRelevancy
Root cause: Algorithmic Error

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
ResponseRelevancy does not guarantee varied questions. This makes strictness effectively pointless.

Ragas version: 0.2.14
Python version: 3.12.3

**Code to Reproduce**
Run ragas.evaluate using ragas.metrics.answer_relevancy as metric.

**Expected behavior**
Generating varied questions.

**Proposed solutions**

- Generate all questions at the same time, relying on the LLM to respect the request for variedness.
- Keep generating questions, with a limited amount of retries, until a unique set of questions is generated. This has the potential to get stuck in a loop and should only be implemented if the first solution is not possible.
Repository: explodinggradients/ragas
Issue: Misleading documentation for "LLM Based Context Precision without reference"
GitHub labels: bug

documentation

good first issue

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

I think the documentation for LLM Based Context Precision without reference is wrong. The doc is available at: https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/context_precision/#context-precision-without-reference

It says:
> `LLMContextPrecisionWithoutReference` metric can be used when you have both retrieved contexts and also _reference contexts_ associated with a `user_input`.

I believe the emphasized part is wrong, since this evaluation works without reference context, as the name and heading suggests.

Looking at the code also confirms this:

```python
@dataclass
class LLMContextPrecisionWithoutReference(LLMContextPrecisionWithReference):
    name: str = "llm_context_precision_without_reference"
    _required_columns: t.Dict[MetricType, t.Set[str]] = field(
        default_factory=lambda: {
            MetricType.SINGLE_TURN: {"user_input", "response", "retrieved_contexts"}
        }
    )

    def _get_row_attributes(self, row: t.Dict) -> t.Tuple[str, t.List[str], t.Any]:
        return row["user_input"], row["retrieved_contexts"], row["response"]

```
... it only requires: `user_input`, `response` `retrieved_contexts`.

Ragas version:
Python version:

**Code to Reproduce**
Share code to reproduce the issue

**Error trace**

**Expected behavior**
A clear and concise description of what you expected to happen.

Updated documentation, for ex:
> `LLMContextPrecisionWithoutReference` metric can be used when you have retrieved contexts (`retrieved_contexts`) associated with a `user_input`.

**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hi, I'll be happy to update the doc if someone could confirm that my understanding is indeed accurate.

Hi @muhammadyaseen, I’d like to work on this and fix the misleading documentation for LLMContextPrecisionWithoutReference. I’ll submit a PR soon. 🙂
Repository: explodinggradients/ragas
Issue: New metric: to quantify the RAG Improvement
Root cause: Unimplemented Feature Gap

It would be beneficial to have an evaluation metric that measures the improvement brought by the RAG. This metric should perform the following:


1. Calculate the distance between the RAG-generated answers and the ground truths.
2. Calculate the distance between the LLM stand-alone generated answers and the ground truths.
3. Determine the improvement based on the two distances mentioned above. The resulting score could range from [-1, 1], since the improvement could potentially be negative.

Inputs required:

1. Ground truths
2. LLM stand-alone generated answers
3. RAG generated answers

Hey @wywdiablo , this is interesting. We worked on a similar idea during the paper but couldn't quantify the difference effectively.  How do you think this metric will help in development? 

@shahules786 I believe this metric will be important for certain use cases. For example, in one case we aim to enhance the answer by integrating the LLM's knowledge with a new external data source. A challenge arises when the RAG forms an answer based solely on the data we provided. If we offer a different prompt template in the RAG, such as, "You are a professor in the XXX domain," it seems only activate the LLM's inherent knowledge rather than drawing from the external data source. There should be a metric to inform the developer of this, allowing for better control.
Repository: explodinggradients/ragas
Issue: AttributeError when I try to use ContextRelevance, LLMContextPrecisionWithoutReference to evaluate the test dataset
Root cause: Validation Gap

**Describe the bug**
I got an AttributeError: property' object has no attribute 'get' when I try to use evaluate() function to evaluate a dataset with ContextRelevance, LLMContextPrecisionWithoutReference metrics. It's working when I only use faithfulness. And it's working when I evaluate a SingleTurnSample-

Ragas version: newest version
Python version: newest version

**Code to Reproduce**
result = evaluate(eval_dataset, metrics=[faithfulness, AnswerRelevancy], llm = evaluator_llm)

**Error trace**
File <command-7263321066632266>, line 27
     22 eval_dataset = EvaluationDataset.from_pandas(df_bge_pd)
     23 print(eval_dataset)
---> 27 result = evaluate(eval_dataset, metrics=[faithfulness, AnswerRelevancy], llm = evaluator_llm)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-f55819bb-501c-45c7-93b3-dc3c0125c4ca/lib/python3.11/site-packages/ragas/validation.py:60, in validate_required_columns(ds, metrics)
     58 metric_type = get_supported_metric_type(ds)
     59 for m in metrics:
---> 60     required_columns = set(m.required_columns.get(metric_type, []))
     61     available_columns = set(ds.features())
     62     if not required_columns.issubset(available_columns):

The evaluation dataset I created is using eval_dataset = EvaluationDataset.from_pandas(df) because the data source is a pandas dataframe

I ran into the same problem with BleuScore, but it was because I wasn't passing it properly.

```
results = evaluate(eval_dataset, metrics=[BleuScore])
> 'property' object has no attribute 'get'
results = evaluate(eval_dataset, metrics=[BleuScore()])
> {'bleu_score': ...}
```

Perhaps a more detailed error message would help here! Or, even better, if the user passes a class instead of an object, perhaps initialise it for them?

Hi @Haoyuxiaohan, 

Were you able to resolve it? Thanks @ziggycross for helping us out. Yes, we need to pass the object of metrics in the list of metrics when doing evaluations.

Take reference form the below example.
```py
import pandas as pd
from ragas.dataset_schema import EvaluationDataset

sample_dict = {'user_input': {0: 'When was Einstein born?'},
 'response': {0: 'Albert Einstein was born in 1879.'},
 'reference': {0: 'Albert Einstein was born in 1879.'},
 'retrieved_contexts': {0: []}}

df = pd.DataFrame(data = sample_dict)

dataset = EvaluationDataset.from_pandas(dataframe=df)

result = evaluate(dataset , metrics=[ContextRelevance(llm=evaluator_llm)])
```
Repository: explodinggradients/ragas
Issue: MultiTurnSample do not cover multiple ToolMessage in a row.
Root cause: Validation Gap

[X] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
MultiTurnSample do not consider scenarios when there are several parallel tool calls after just one AIMessage.

Ragas version: 0.2.14
Python version: 3.11

**Code to Reproduce**

```python
from ragas.dataset_schema import MultiTurnSample
from ragas.messages import HumanMessage, AIMessage, ToolMessage, ToolCall


sample_input = [
    HumanMessage(
        content="Can you provide me with details about Einstein's theory of relativity?"
    ),
    AIMessage(
        content="Got it! Let me fetch more details from 'General Theory of Relativity by A. Einstein'.",
        tool_calls=[
            ToolCall(
                name="document_retrieve",
                args={"document": "General Theory of Relativity by A. Einstein"},
            ),
            ToolCall(
                name="document_retrieve",
                args={"document": "A. Einstein biography"},
            ),
        ],
    ),
    ToolMessage(
        content="Found relevant documents: 1. Relativity: The Special and the General Theory, 2. General Theory of Relativity by A. Einstein."
    ),
    ToolMessage(content="Found relevant documents: 1. A. Einstein biography"),
    AIMessage(content="I found some documents on Einstein's theory of relativity..."),
]
sample = MultiTurnSample(user_input=sample_input)

```


**Error trace**
```
  File ".venv/lib/python3.11/site-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for MultiTurnSample
user_input
  Value error, ToolMessage instances must be preceded by an AIMessage instance. [type=value_error, input_value=[HumanMessage(content="Ca...='ai', tool_calls=None)], input_type=list]
    For further information visit https://errors.pydantic.dev/2.10/v/value_error
```


**Expected behavior**
I would expect several ToolMessage to be supported after just one AIMessage.
Repository: explodinggradients/ragas
Issue: Llama4 as Evaluator
Root cause: Documentation Deficiency

**Describe the Feature**
Llama4 is recently released and it is benchmarking as one of the best evaluators.

**Why is the feature important for you?**
With 10M of context window, I believe it will be game-changer evaluator.

**Additional context**
As its newly released, it will give a boost to Ragas also if we implement it faster and bring it show 😄 


Hi @PrashantDixit0, we will soon be releasing a tutorial showing evaluations with Llama 4

If you are planning to do evaluations with Llama and using TogetherAI Api you can do it like this,

```
from langchain_together import ChatTogether
from ragas.llms import LangchainLLMWrapper

llm = ChatTogether(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
)
evaluator_llm = LangchainLLMWrapper(llm)
```
if you are using something else if it is supported by Langchain, just wrap it with LangchainLLMWrapper and you are good to go.
Repository: explodinggradients/ragas
Issue: Error code 500 Internal Server Error
Root cause: External Dependency Breakage

 [ERROR] [ragas.utils] [RagasID: a-fb6708849c364267bd4cfedcea81454f, 
App-Version: 0.2.14]
 [API_ERROR] Request failed. Status Code: 500, URL: https://api.ragas.io/api/v1/alignment/evaluation,
 Error Message: 
API Message: An internal server error occured
means the alignment evaluation endpoint is crashing on Ragas' backend.
Repository: explodinggradients/ragas
Issue: Can you confirm if Vertex AI (Text Bison) is supported with Ragas?
Root cause: Documentation Deficiency

Hi, thanks for this tool. Can you tell us when Ragas will support LLMs like Vertex AI models like Text-bison?

yep you can use Vertext AI models via langchains [Vertext AI modle](https://python.langchain.com/docs/integrations/llms/google_vertex_ai_palm).

```py
from langchain.llms import VertexAI
from ragas.llms import LangchainLLM

llm = LangchainLLM(VertextAI())
# now you can swap it out for other LLMs
```

to figure out how to customise Ragas llm check [this doc](https://docs.ragas.io/en/latest/howtos/customisations/llms.html)

feel free to reopen if you face any more problems

Hi, I am facing the same issue. 

This solution isn't working:

```
from langchain.llms import VertexAI
from ragas.llms import LangchainLLM

llm = LangchainLLM(VertextAI())
```