zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: Nvidia Accuracy calculation code is not consistent with the docs
Root cause: Documentation Deficiency

[ x ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
In the Nvidia Accuracy metric calculation, the User Answer and Reference Answer were supposed to be swapped (according to docs) in steps 1 and 2. However, it's not really swapping, as both titles and values are being swapped.

Ragas version: 0.2.14


Hi @vitorsabbagh, are you just referring to the example in the docs? 

the example calculation is not the best example because both the response and reference is the same. 

If we look under the hood, it swapping is happening.

1. https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_nv_metrics.py#L109

```py
formatted_prompt = StringPromptValue(
                    text=self.template_accuracy1.format(
                        query=sample.user_input,
                        answer0="User Answer",
                        answer1="Reference Answer",
                        sentence_inference=sample.response,
                        sentence_true=sample.reference,
                    )
                )
```

2. https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_nv_metrics.py#L132

```py
                formatted_prompt = StringPromptValue(
                    text=self.template_accuracy2.format(
                        query=sample.user_input,
                        answer0="Reference Answer",
                        answer1="User Answer",
                        sentence_inference=sample.reference,
                        sentence_true=sample.response,
                    )
                )
```

notice `answer0` and `answer1` they are getting swapped.

Hi @sahusiddharth, you're right! However, notice that sentence_inference and sentence_true are also being swapped. It's a double swap.

Yes @vitorsabbagh, you are correct thank-you letting us know. I will raise a PR to fix this.

I believe the implementation is right.
The docs rather needs update to clarify the double swap.

@shahules786 Could you please verify once? 

> I believe the implementation is right. The docs rather needs update to clarify the double swap.
> 
> [@shahules786](https://github.com/shahules786) Could you please verify once?

Hi!

What’s the rationale behind using a double swap? Is it just reversing the order of ref and pred? I’d personally prefer running the same prompt twice with two different LLMs.

The original single-swap idea felt more promising :/


> The original single-swap idea felt more promising :/

Update: The newer metrics collections has single swap (AnswerAccuracy for example). We'll be deprecating the legacy metrics api soon.
Repository: explodinggradients/ragas
Issue: How do I put ragas outputs into a clean JSON, like the JSON that I can get from the dashboard?
Root cause: Documentation Deficiency

Hi.
I want to extract the output results from the Ragas result object on my local machine. How do I do that?

Hi @AmirPoursaberi, you can convert the evaluation result to pandas then can convert them to dict.

```py
from ragas.dataset_schema import EvaluationDataset
from ragas import evaluate 
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import AnswerAccuracy

from langchain_openai import ChatOpenAI
from ragas.llms import LangchainLLMWrapper

sample = SingleTurnSample(
    user_input="When was Einstein born?",
    response="Albert Einstein was born in 1879.",
    reference="Albert Einstein was born in 1879."
)

dataset = EvaluationDataset(samples=[sample])

llm = ChatOpenAI(model="gpt-4o-mini")
evaluator_llm = LangchainLLMWrapper(langchain_llm=llm)

result = evaluate(dataset=dataset, metrics=[AnswerAccuracy(llm=evaluator_llm)])
result.to_pandas().to_dict()
```

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Multi-Modal Testset Generation
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Add support for multi-modal testset generation in RAGAS. While RAGAS currently supports multi-modal evaluation through MultiModalRelevance and MultiModalFaithfulness metrics, it lacks the ability to generate multi-modal test cases. This feature would enable automatic generation of test cases that combine both text and image contexts, making it easier to evaluate multi-modal RAG systems comprehensively. Ideally it would also extract images from documents.

**Why is the feature important for you?**
Currently, while RAGAS can evaluate multi-modal RAG systems, users must manually create test cases that include images. This creates a disconnect between the evaluation and generation capabilities. Visual RAG appears to be the future. 

I am also looking to make my first contribution to the library! I could give this a go and create a PR.

@wabeg I am looking forward to your PR!
Repository: explodinggradients/ragas
Issue: ragas evaluator function is taking LLM model as none even it is passed in evaluator's function call
Root cause: Interface Contract Mismatch

Code:
%python
    # %pip install openai langchain
    
    from ragas.metrics import answer_relevancy, faithfulness, context_utilization
    from ragas.evaluation import evaluate
    from azure.identity import DefaultAzureCredential
    import openai
    
    import os
    from langchain.llms import AzureOpenAI
    
    # Set your Azure OpenAI endpoint and API key
    endpoint = <end point>
    api_key = <api_key>
    api_version = <api_version>
    
    # Configure OpenAI client
    os.environ["OPENAI_API_KEY"] = api_key
    
    # Wrap the AzureOpenAI client with Langchain's AzureOpenAI class
    evaluator_llm = AzureOpenAI(
        api_key=api_key,
        api_base=endpoint,
        api_type="azure",
        api_version=api_version,
        deployment=deployment
    )
    
    # Compute RAGAS Scores
    scores = evaluate(
        dataset,
        metrics=[faithfulness, answer_relevancy, context_utilization],
        llm=evaluator_llm
    )

Error:
 AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config'
    File , line 25
         16 evaluator_llm = AzureOpenAI(
         17     api_key=api_key,
         18     api_base=endpoint,
       (...)
         21     deployment=deployment
         22 )
         24 # Compute RAGAS Scores
    ---> 25 scores = evaluate(
         26     dataset,
         27     metrics=[faithfulness, answer_relevancy, context_utilization],
         28     llm=evaluator_llm
         29 )
         31 # Convert results back to DataFrame for easy viewing
         32 results_df = scores.to_pandas()
    File /local_disk0/.ephemeral_nfs/envs/pythonEnv-087f3445-139b-431e-884f-bffb273ec73f/lib/python3.11/site-packages/ragas/metrics/base.py:153, in MetricWithLLM.init(self, run_config)
        149 if self.llm is None:
        150     raise ValueError(
        151         f"Metric '{self.name}' has no valid LLM provided (self.llm is None). Please initantiate a the metric with an LLM to run."  # noqa
        152     )
    --> 153 self.llm.set_run_config(run_config)

Hi @manaligc,

Which ragas version are you using? If you are using the latest one.

I'd recommend importing the metrics class, Wrapping you Langchain LLM with ragas  `LangchainLLMWrapper`

something like this
```py
from ragas.metrics import AnswerAccuracy, ContextUtilization, Faithfulness
from langchain.llms import AzureOpenAI
from ragas.llms import LangchainLLMWrapper

evaluator_llm = LangchainLLMWrapper(AzureOpenAI(
    api_key=api_key,
    api_base=endpoint,
    api_type="azure",
    api_version=api_version,
    deployment=deployment
))

scores = evaluate(
    dataset,
    metrics=[AnswerAccuracy(llm=evaluator_llm), ContextUtilization(llm=evaluator_llm), Faithfulness(llm=evaluator_llm)],
    llm=evaluator_llm
)
```
Repository: explodinggradients/ragas
Issue: Seperate Evaluation for Retrieval and Generation both
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
How ragas can make it possible to evaluate retrieval and generation both separately with different metrics to analyse each of these step.


### Retrieval Metrics
- Context Precision
- Context Recall
- Context Relevance

### Generation Metrics
- Faithfulness
- Response Relevancy
- Noise Sensitivity
- Response Groundedness

Ragas also gives you options to make your own metrics using 
- Aspect Critic
- Rubric Score
- Simple Criteria 

We'll soon have this on docs as well, thanks to @KalyanKS-NLP. 

Thank you @sahusiddharth for responding.
I also want to contribute in Ragas and other projects of Explodinggradients. Let me know how can I contribute 👍
Repository: explodinggradients/ragas
Issue: How can we integrate Ragas with WebdriverIO
GitHub labels: question

Could you please advise how we can integrate Ragas with WebdriverIO

Hi, are there any update on the reported issue?

Hi Team, just following up on this again. could you please guide us on the above

@hameeda2 Could you please share more context?
What's the use case? How are you planning to integrate? Are you looking for any specific eval metric?
Repository: explodinggradients/ragas
Issue: Trouble using Faithfulness Metric with HHEM-2.1-Open
Root cause: Documentation Deficiency

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Can't figure out how to use Faithfulness metric with HHEM-2.1-Open (https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/?h=faithfulness#example)

**Code Examples**

```python
import asyncio
from ragas.dataset_schema import SingleTurnSample 
from ragas.metrics import FaithfulnesswithHHEM

async def main():

    sample = SingleTurnSample(
        user_input="When was the first super bowl?",
        response="The first superbowl was held on Jan 15, 1967",
        retrieved_contexts=[
            "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
        ]
    )

    scorer = FaithfulnesswithHHEM()
    result = await scorer.single_turn_ascore(sample)
    print(result)

if __name__ == "__main__":
    asyncio.run(main())

```

**Additional context**
Anything else you want to share with us? 

Just keep getting this error: `assert self.llm is not None, "LLM is not set"` but I'm following the documentation exactly. I thought the LLM would be set under the hood when using this metric that pulls automatically from HuggingFace?

Hi @awlevin,

Yes, your understanding is correct, the `FaithfulnesswithHHEM` metric automatically selects the model from HuggingFace, which is used for doing classification. 

However, under the hood, `FaithfulnesswithHHEM` requires a separate LLM to break the response into simpler statements before classification.

```py
from ragas.dataset_schema import SingleTurnSample 
from ragas.metrics import FaithfulnesswithHHEM
from ragas.llms import LangchainLLMWrapper
from langchain_openai import OpenAI

evaluator_llm = LangchainLLMWrapper(OpenAI())


sample = SingleTurnSample(
        user_input="When was the first super bowl?",
        response="The first superbowl was held on Jan 15, 1967",
        retrieved_contexts=[
            "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
        ]
    )
scorer = FaithfulnesswithHHEM(llm=evaluator_llm)
await scorer.single_turn_ascore(sample)
```

I hope this helps



Makes sense, thank you! This part of the docs threw me off:

![Image](https://github.com/user-attachments/assets/31321f2f-c8f9-4128-9aee-eddf42d8466e)

I'll close this issue now but let me know if there's an open source page somewhere that I should update, if that seems appropriate.
Repository: explodinggradients/ragas
Issue: RAGAS metrics vs Traditional Metrics
GitHub labels: question

Hi!

I would like to know if RAGAS metrics are as reliable as with traditional metrics, especially compared to retrieval metrics such as precision@k, recall@k, MRR, nDCG... Are there any studies/papers about LLMs as a judge that ensure that?

If you had the possibility to choose between LLMs as a judge and traditional metrics, what would you choose? (In the case you want to evaluate a RAG pipeline)

Thanks
Repository: explodinggradients/ragas
Issue: Documentation: howto about Non-English Testset Generation is configured to generate query in Spanish but generates queries in French
GitHub labels: bug

documentation

[x ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
It is a "documentation bug".

In the notebook [Non-English Testset Generation Notebook](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customizations/testgenerator/_language_adaptation.md), the howto shows how to generate queries in Spanish from Spanish wikipedia articles.

However, at the end of the notebook, the example queries are in French:

```
Query: Quelles sont les caractéristiques du Bronx en tant que borough de New York?
Reference: Le Bronx est l'un des cinq arrondissements de New York, qui est la plus grande ville des États-Unis. Bien que le contexte ne fournisse pas de détails spécifiques sur le Bronx, il mentionne que New York est une ville cosmopolite avec de nombreux quartiers ethniques, ce qui pourrait inclure des caractéristiques culturelles variées présentes dans le Bronx.
```

Ragas version: latest
Repository: explodinggradients/ragas
Issue: unable to apply transformation: Connection error
Root cause: Validation Gap

Description:

I am trying to generate test set with ragas and I am getting connection error when invoking generator.generate_with_langchain_docs with the generator object. I am using llm proxy for the llm.

Ragas version: 2.0

**Code to Reproduce**
#ignore un necessary imports below

pip install langchain-community
import os
!pip install ragas

from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI
from langchain_openai import OpenAIEmbeddings

from ragas.testset import TestsetGenerator
from langchain_community.document_loaders import DirectoryLoader
!pip install unstructured[local-inference]

pip install -qU langchain-huggingface
from langchain_community.embeddings import HuggingFaceEmbeddings
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
from sentence_transformers.quantization import quantise_embeddings
from langchain_community.embeddings import SentenceTransformerEmbeddings

from ragas.testset import TestsetGenerator
from langchain.text_splitter import TokenTextSplitter

path = "/content"
loader = DirectoryLoader(path, glob="**/*.txt")
docs = loader.load()


embeddings = HuggingFaceEmbeddings(model_name="mixedbread-ai/mxbai-embed-large-v1")


generator_llm_candy = ChatOpenAI(model_name = "gpt-4o",
                 openai_api_base = "",
                 openai_api_key  = "", # DO NOT HARDCODE YOUR KEY
                 model_kwargs    = {'user': "" }) # DO NOT HARDCODE A USER HERE

generator_llm = LangchainLLMWrapper(generator_llm_candy)
generator_embeddings = LangchainEmbeddingsWrapper(embeddings)


generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)

data_frame = dataset.to_pandas()

data_frame.to_csv('file1.csv')


**Error trace**
ERROR:ragas.testset.transforms.engine:unable to apply transformation: Connection error.
ERROR:ragas.testset.transforms.engine:unable to apply transformation: 'headlines' property not found in this node
ERROR:ragas.testset.transforms.engine:unable to apply transformation: Connection error.
ERROR:ragas.testset.transforms.engine:unable to apply transformation: node.property('summary') must be a string, found '<class 'NoneType'>'
ERROR:ragas.testset.transforms.engine:unable to apply transformation: Node c11e1996-b069-4197-a7dd-ba85818f1641 has no summary_embedding


<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


I can see issue like this https://github.com/explodinggradients/ragas/issues/1496.  But there the issue was not passing embedding model in the  TestsetGenerator api but here in my code I do the both.
Repository: explodinggradients/ragas
Issue: Context Precision Documentation
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**My Question**
https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/#llm-based-context-precision
In the docs of Context Precision without reference - it clearly states:
""```LLMContextPrecisionWithoutReference``` metric can be used when you have both ***retrieved contexts*** and also ***reference contexts*** associated with a user_input.""

However, in the code snippet, only the retrieved_context is used. Instead of the reference_contexts, as suggested in the text, the response is provided.

I'm trying to understand why reference_contexts aren't used here. From my understanding, the correct approach would be to use the reference contexts, as the documentation suggests, instead of the response — especially since we are calculating context recall. To calculate recall, we need a reference to determine which retrieved contexts are correct, so we can say, for example, that x% of the retrieved contexts are relevant.

Why isn't that the case here?
Repository: explodinggradients/ragas
Issue: Knowledge Graph node IDs should be content hashes for idempotency
Root cause: Unimplemented Feature Gap

**Describe the Feature**
Currently, IDs are UUIDs.  it would improve idempotency of knowledge graph generation if the IDs were based on hashes of the text; that way the same text automatically receives the same ID.  This is particularly valuable in the case of Splitters; assuming teh splitting algorithm is deterministic, the resulting chunks with a hash ID would mean that the same pipeline does not replicate the chunks (as I believe it currently does).

**Why is the feature important for you?**
Run knowledge graph generation twice - I believe you end up with 2x chunks because the chunk UUIDs do not recognize the chunk exists.

**Additional context**
Add any other context about the feature you want to share with us.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->
Repository: explodinggradients/ragas
Issue: evaluate() fails with *APIConnectionError(Connection error.)* when azure (AzureOpenAIEmbeddings, AzureChatOpenAI) langchain models are used
Root cause: Interface Contract Mismatch

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
While using azure models (AzureOpenAIEmbeddings, AzureChatOpenAI) the evaluate() module seems to fail with 
Error Message : Exception raised in Job[1]: APIConnectionError(Connection error.) 

Ragas version: Version: 0.2.15
Python version: 3.11.0

**Code to Reproduce**
```
import os
from langchain_openai.embeddings import AzureOpenAIEmbeddings
from langchain_openai.chat_models import AzureChatOpenAI
import httpx
from dotenv import load_dotenv

from ragas.metrics import FactualCorrectness, LLMContextPrecisionWithoutReference, Faithfulness
from ragas import SingleTurnSample, EvaluationDataset
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper


from ragas import evaluate
load_dotenv()


embeddings = AzureOpenAIEmbeddings(model="MYMODELNAME", http_client=httpx.Client(verify=False))
evaluation_llm = AzureChatOpenAI(deployment_name="MYMODELNAME",   http_client=httpx.Client(verify=False),  temperature=0.1)

dataset = []

dataset.append(
        {
            "user_input": "When was the first super bowl?",
            "retrieved_contexts": [
            "The First AFLNFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
        ],
            "response": "The first superbowl was held on Jan 15, 1967"
        }
    )

evaluation_dataset = EvaluationDataset.from_list(dataset)
result = evaluate(
    llm=evaluation_llm,
    embeddings= LangchainEmbeddingsWrapper(embeddings=embeddings),
    dataset= evaluation_dataset,
    metrics=[
        Faithfulness(llm=evaluation_llm)
    ],
)

print(result)
```

**Error trace**

```
Evaluating:   0%|                                                                                                                 | 0/2 [00:00<?, ?it/s]Exception raised in Job[1]: APIConnectionError(Connection error.)
Evaluating:  50%|████████████████████████████████████████████████████                                                    | 1/2 [02:36<02:36, 156.27s/it]
{'faithfulness': nan}
```
**Expected behavior**

Expected scores for all the metrics thats part of evaluate

**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: Found Typo in code

https://github.com/explodinggradients/ragas/blob/46ef849108caad21da65c10b0fd3d4a32f2e05b0/src/ragas/testset/persona.py#L94-L97

should be changed "filer" to "filter"
```
"No nodes that satisfied the given filter. Try changing the filter." 
```
Repository: explodinggradients/ragas
Issue: RAG Evaluation Metrics and Recommended Thresholds
GitHub labels: question

module-metrics

Are there any links to refer to recommended thresholds for each metric listed below for RAG evaluation?

 I applied these metrics to the RAG project. These are the metrics for RAG Evaluation that range from scores of 0-1 (0%-100%):
 RAGAS metric
 1.Correctness 
2.Semantic Similarity 
3.Faithfulness
4. Response Relevancy 
5. LLM Context Recall 


Nvidia Metrics: 
1.Answer Accuracy 
2.Context Relevance Responsto
3.Groundedness 

For example, I used 4o-mini to synthesize 100 questions about company policy and evaluated them using 4o-mini, anthropic.claude-3-5-haiku, and gemini-2.0. The summary scores are shown in the table shown in  below.

metric | mean_score

nv_accuracy | 0.74
nv_context_relevance | 0.96
nv_response_groundedness | 0.98
answer_correctness | 0.65
semantic_similarity | 0.95
faithfulness | 0.93
answer_relevancy | 0.97
context_recall | 0.95
llm_context_precision_with_reference | 0.95
llm_context_precision_without_reference | 0.98


 <p>&nbsp;</p>
</div>
Repository: explodinggradients/ragas
Issue: Evaluate a simple RAG system : RAG() does not initialize any instance
Root cause: Documentation Deficiency

Hi,
This is from the website: https://docs.ragas.io/en/stable/getstarted/rag_eval/

The line:
# Initialize RAG instance
rag = RAG()

is not working.

Opened PR #2292 to address this. Updates RAG example to use embed_text/embed_texts and adds a note about explicit
langchain-core/langchain-openai installs. 

The code given in the [doc](https://docs.ragas.io/en/stable/getstarted/rag_eval/) works as expected with these two changes.
Repository: explodinggradients/ragas
Issue: question about src.ragas.metrics._nv_metrics.ContextRelevance._single_turn_ascore
Root cause: Algorithmic Error

[ yes] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
template_relevance1 is used twice but template_relevance2 is not used. I think this is a bug.

**Code Examples**

**Additional context**