zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: StatementGeneratorPrompt's instruction mentions "statements", but these are not part of inputs anymore
Root cause: Documentation Deficiency

[X ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
A clear and concise description of what the bug is.

This is probably a minor one. I am just opening for awareness, and it could impact the results when using less capable LLMs.

The instruction for StatementGeneratorPrompt start with "Given a question, an answer, and sentences from the answer [...]", but sentences are not part of the inputs anymore. Just question and answer. I believe this was changed in a recent PR.

https://github.com/explodinggradients/ragas/blob/2605de5224a7c0e90f000d74f561d2dae770c18f/src/ragas/metrics/_faithfulness.py#L37

Ragas version: main branch in this repo
Python version: not applicable

**Code to Reproduce**

Not applicable

**Error trace**

Not applicable

**Expected behavior**

Instruction should reflect the fact that simple statements are extracted from the full answer now, and not the answer statements.


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hi @diegodebrito, 

Thank you for pointing that out. I’ve raised a PR to address and fix this issue.
Repository: explodinggradients/ragas
Issue: Context Precision Calculation
GitHub labels: question

module-metrics

[✓ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I wanted to get some clarification on how Context Precision is calculated - I thought it compares the retrieved context chunks to the user input to determine the precision / relevance of the retrieved context. However, the documentation suggests that the context chunks are compared to the response generated by the llm - so is it calculating whether the retrieved context is used by the llm in generating a response?

**Documentation**
LLM Based Context Precision
The following metrics uses LLM to identify if a retrieved context is relevant or not.
Context Precision without reference
LLMContextPrecisionWithoutReference metric can be used when you have both retrieved contexts and also reference contexts associated with a user_input. To estimate if a retrieved contexts is relevant or not this method uses the LLM to compare each of the retrieved context or chunk present in retrieved_contexts with response.

**Follow-up**
Is there a metric that checks to see the relevance of the retrieved context based on the user input?


Hi @ishachinniah-hds, 

> I wanted to get some clarification on how Context Precision is calculated - I thought it compares the retrieved context chunks to the user input to determine the precision / relevance of the retrieved context. However, the documentation suggests that the context chunks are compared to the response generated by the LLM - so is it calculating whether the retrieved context is used by the LLM in generating a response?

Yes, you are absolutely right. The Context Precision metric compares the retrieved context chunks with the response generated by the LLM to check if they were useful in forming the response.

> Is there a metric that checks to see the relevance of the retrieved context based on the user input?

If you're looking to check the relevance of the retrieved context based on the user input, you can create a [custom metric](https://docs.ragas.io/en/stable/howtos/customizations/metrics/_write_your_own_metric/). For a binary metric, you could use `Aspect Critic` (Simple Criteria Scoring). If you're looking for something more detailed, you can consider using `RubricsScore` (Rubric-based Metrics). 

Let me know how it works for you.


Hi @sahusiddhart,

Thank you for the clarification. I will consider creating a custom metric for use if necessary.
Repository: explodinggradients/ragas
Issue: ragas evaluate method for the same metric with different parameters gives scores for only one metric
Root cause: Algorithmic Error

I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When I evaluate the same metric but with different parameters the resulting pandas dataframe contains the results of the only last one. For example, I would like to measure factual_correctness in different modes precision and recall, I got the results for recall only.

Ragas version: 0.2.12
Python version: 3.10

**Code to Reproduce**
```py
scorer1 = FactualCorrectness(llm = evaluator_llm, mode="precision")
scorer2 = FactualCorrectness(llm = evaluator_llm, mode="recall")
evaluate(dataset=dataset, metrics[scorer1, scorer2]).to_pandas()
```
**Expected behaviour**
I would like to see scores for both factual_correctness for precision and recall in the resulting dataframe.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->
Repository: explodinggradients/ragas
Issue: Wrong cache hits for LlamaIndexEmbeddingsWrapper
Root cause: Algorithmic Error

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
When generating embeddings with the `LlamaIndexEmbeddingsWrapper` a wrong cache hit leads to all embeddings being the same.
This is due to the [`_generate_cache_key` function](https://github.com/explodinggradients/ragas/blob/cb63a8266b80fcb5715b06fc97d1f9165bc49f93/src/ragas/cache.py#L150) removing the first argument for methods. However it seems `self` is not passed here as an argument, hence valid arguments are sliced away [here](https://github.com/explodinggradients/ragas/blob/cb63a8266b80fcb5715b06fc97d1f9165bc49f93/src/ragas/cache.py#L151) and every cache key becomes the same.

Ragas version: 0.2.13
Python version: 3.11.2

**Code to Reproduce**
[see this gist](https://gist.github.com/triedel42/5afae2c2e43913355508a159b5956a62)

**Error trace**
N/A (see assertion in example code)

**Expected behavior**
Distinct `embed_text` arguments need to result in distinct vectors.
Distinct `embed_text` arguments need to be cached under distinct cache keys.

**Additional context**
We noticed this bug when running metrics and receiving implausible results (e.g. SemanticSimilarity always being 1).

[This test case](https://github.com/explodinggradients/ragas/blob/cb63a8266b80fcb5715b06fc97d1f9165bc49f93/tests/unit/test_cache.py#L29) did not catch it either, since it's only tested with functions, not methods.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Should be fixed with #1977 and will be released in next week release.
If still occurs, feel free to reopen. Thanks for bringing it up @triedel42 🙌🏼
Repository: explodinggradients/ragas
Issue: Use compression when saving knowledge graphs
Root cause: Unimplemented Feature Gap

**Describe the Feature**
I found that ragas directly dumps `node` and `relationships` directly when saving the knowledge graph, and the `relationships` holds all the information about the relevant `node`, with a very large amount of redundancy, especially in the `summary_embedding` field. So when saving `relationships`, wouldn't it be better to just save the ID of the related node instead?

Also, there are some compression methods that can be used when saving embeddings. For instances, some RAG pipelines use base64 encoding to save these embeddings.

**Why is the feature important for you?**
Generating test sets takes a lot of time, and it is necessary to save the knowledge graphs generated in between to be able to load the knowledge graphs directly in case of an anomaly instead of starting all over again.

<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->


https://github.com/explodinggradients/ragas/blob/cb63a8266b80fcb5715b06fc97d1f9165bc49f93/src/ragas/testset/graph.py#L88

It would be fine to add this code to class `Relationship`

```python
    @field_serializer("source", "target")
    def serialize_node(self, node: Node):
        return node.id
```

thanks again @lryan599 for the fix 🙌🏽 

> thanks again [@lryan599](https://github.com/lryan599) for the fix 🙌🏽

Glad to contribute.
Repository: explodinggradients/ragas
Issue: SingleHopQuerySynthesizer::prepare_combinations()
Root cause: Algorithmic Error

[ x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

https://github.com/explodinggradients/ragas/blob/cb63a8266b80fcb5715b06fc97d1f9165bc49f93/src/ragas/testset/synthesizers/single_hop/base.py#L51

**Your Question**
```python
    def prepare_combinations(
        self,
        node: Node,
        terms: t.List[str],
        personas: t.List[Persona],
        persona_concepts: t.Dict[str, t.List[str]],
    ) -> t.List[t.Dict[str, t.Any]]:

        sample = {"terms": terms, "node": node}
        valid_personas = []
        persona_list = PersonaList(personas=personas)
        for persona, concepts in persona_concepts.items():
            concepts = [concept.lower() for concept in concepts]
            if any(term.lower() in concepts for term in terms):
                if persona_list[persona]:
                    valid_personas.append(persona_list[persona])
            sample["personas"] = valid_personas
            sample["styles"] = list(QueryStyle)
            sample["lengths"] = list(QueryLength)

        return [sample]
```

I think it should be

```python
    def prepare_combinations(
        self,
        node: Node,
        terms: t.List[str],
        personas: t.List[Persona],
        persona_concepts: t.Dict[str, t.List[str]],
    ) -> t.List[t.Dict[str, t.Any]]:

        sample = {"terms": terms, "node": node}
        valid_personas = []
        persona_list = PersonaList(personas=personas)
        for persona, concepts in persona_concepts.items():
            concepts = [concept.lower() for concept in concepts]
            if any(term.lower() in concepts for term in terms):
                if persona_list[persona]:
                    valid_personas.append(persona_list[persona])
        // these codes should be outside
        sample["personas"] = valid_personas
        sample["styles"] = list(QueryStyle)
        sample["lengths"] = list(QueryLength)

        return [sample]
```
Repository: explodinggradients/ragas
Issue: Fix Documentation 

I was going through docs and noticed a few typos. Sharing them here.

[Evaluation Document](https://github.com/explodinggradients/ragas/blob/main/docs/getstarted/evaluation.md?plain=1#L31-L34):
> question: list[str] - These are the questions you**r** RAG pipeline will be evaluated on.
> answer: list[str] - The answer generated from the RAG pipeline and give**n** to the user.
> contexts: list[list[str]] - The contexts which **were** passed into the LLM to answer the question.
> ground_truths: list[list[str]] - The ground truth answer to the questions. (only required if you are using context_recall)
 

hey @mmaysami Nice catch. Would you like to raise a PR on this?
Repository: explodinggradients/ragas
Issue: How to use TongYi's text-embedding-v1 embeddings  model
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
May l ask if this is the method of calling

**Code Examples**
embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings(
            openai_api_key=DASHSCOPE_API_KEY,
            openai_api_base=openai_api_base,
            model=model_name
))
AuthenticationError(Error code: 401 - {'error': {'message': 'Incorrect API key provided. ', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}, 'request_id': '59de6b83-e2aa-97df-9b03-c95116cd245c'})




Hi @LillyChen,

You can use the any embedding supported in Langchain using the `LangchainEmbeddingsWrapper`.

```py
from langchain_community.embeddings import DashScopeEmbeddings
from ragas.embeddings import LangchainEmbeddingsWrapper

embeddings = DashScopeEmbeddings(dashscope_api_key="my-api-key")
evaluator_embedding = LangchainEmbeddingsWrapper(embeddings)
```

```
AuthenticationError(Error code: 401 - {'error': {'message': 'Incorrect API key provided. ', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}, 'request_id': '59de6b83-e2aa-97df-9b03-c95116cd245c'})
```

The error you are facing above is due to incorrect api key provided, can please check it again.


I'll give it a try. Thank you！

I also want to ask if there is a more common way to only pass openai_api_key, openai_api_base, and model , etc?

Because l want to quickly switch between different embeddings models.

I'll give it a try. Thank you！

I also want to ask if there is a more common way to only pass openai_api_key, openai_api_base, and model , etc?

Because l want to quickly switch between different embeddings models.



> Hi [@LillyChen](https://github.com/LillyChen),
> 
> You can use the any embedding supported in Langchain using the `LangchainEmbeddingsWrapper`.
> 
> from langchain_community.embeddings import DashScopeEmbeddings
> from ragas.embeddings import LangchainEmbeddingsWrapper
> 
> embeddings = DashScopeEmbeddings(dashscope_api_key="my-api-key")
> evaluator_embedding = LangchainEmbeddingsWrapper(embeddings)
> ```
> AuthenticationError(Error code: 401 - {'error': {'message': 'Incorrect API key provided. ', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}, 'request_id': '59de6b83-e2aa-97df-9b03-c95116cd245c'})
> ```
> 
> The error you are facing above is due to incorrect api key provided, can please check it again.
Repository: explodinggradients/ragas
Issue: Issues with await for language adapt
Root cause: Documentation Deficiency

[ y] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**

I run the code flow with the help documents bellow,but one error happened.
https://docs.ragas.io/en/latest/howtos/customizations/testgenerator/_language_adaptation/

```py
from ragas.testset.synthesizers.single_hop.specific import (
    SingleHopSpecificQuerySynthesizer,
)

distribution = [
    (SingleHopSpecificQuerySynthesizer(llm=generator_llm), 1.0),
]

for query, _ in distribution:
    prompts = await query.adapt_prompts("spanish", llm=generator_llm)
    query.set_prompts(**prompts)
```

Output
```
==========================
results:

    prompts = await query.adapt_prompts("spanish", llm=generator_llm)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: 'await' outside function



Ragas version: 0.2.13
Python version: 3.10.12

**Code to Reproduce**
from ragas.testset.synthesizers.single_hop.specific import (
    SingleHopSpecificQuerySynthesizer,
)

distribution = [
    (SingleHopSpecificQuerySynthesizer(llm=generator_llm), 1.0),
]

for query, _ in distribution:
    prompts = await query.adapt_prompts("spanish", llm=generator_llm)
    query.set_prompts(**prompts)


**Error trace**

    prompts = await query.adapt_prompts("spanish", llm=generator_llm)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: 'await' outside function
```


Hi @shawn-maxiao,

I couldn't reproduce the error on my end. I'm using Python 3.10.14 and Ragas 0.2.13. Could you try reinstalling the Ragas package and see if that helps?

```py
from ragas.testset.synthesizers.single_hop.specific import (
    SingleHopSpecificQuerySynthesizer,
)

distribution = [
    (SingleHopSpecificQuerySynthesizer(llm=generator_llm), 1.0),
]

for query, _ in distribution:
    prompts = await query.adapt_prompts("spanish", llm=generator_llm)
    query.set_prompts(**prompts)
```


The code needs to be in inside a`async def` function.  See this pull request for an example https://github.com/explodinggradients/ragas/pull/1983
Repository: explodinggradients/ragas
Issue: [question] About input schema
GitHub labels: question

answered

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
What's the difference between reference and gournd_truth? 


Hi @wangzhen38,

Both terms convey the same idea, but ground_truth was used in previous versions and is now deprecated. The current version uses reference instead.

It seems the issue was answered, closing this now.
Repository: explodinggradients/ragas
Issue: Bug: context relevancy
Root cause: Algorithmic Error

- [ ] Redo prompt to select only absolutely required sentences
- [ ] Use sent_tokenize instead of new line split [here](https://github.com/explodinggradients/ragas/blob/3ae0f04af0c9ee8159f868aa8d1acd488532de65/src/ragas/metrics/context_precision.py#L164)
Repository: explodinggradients/ragas
Issue: Unable to get  scores for all the examples in the DataFrame. Instead getting the combined score for the Whole dataframe.Need help to get the individual scores as well.
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
Unable to get scores for all the examples in the DataFrame. Instead getting the combined score for the Whole dataframe.Need help to get the individual scores as well.

Even though I have 400 samples i could able to access only final score..I have a need to get individual scores.
When I am inferencing one by one sample it is taking lot of time to generate the scores.

**Code Examples**
llm=LangChainBedrock(model_id="anthropic.claude-v2")
ragas_result = ragas.evaluation.evaluate(
        Dataset.from_pandas(ragas_df),
        metrics=metrics,
        llm=llm,
        embeddings=LangChainBedrockEmbed(model_id="cohere.embed-multilingual-v3"),
        column_map=column_map,
    )
ragas_result

{'answer_relevancy': 0.8665, 'faithfulness': 0.9537}
**Additional context**
Anything else you want to share with us? 

![Image](https://github.com/user-attachments/assets/c28d0037-afe0-408c-a3b0-76cb8269514d)

Hi @Rohith-pamidimarri,

To get individual scores, you can use the following line of code.

```py
ragas_result.to_pandas()
```
If you need help with tracing of evals, you can the [app.ragas.io](http://app.ragas.io/) for more information on it you can refer to these [docs](https://docs.ragas.io/en/stable/howtos/applications/_metrics_llm_calls/).

Thank you @sahusiddharth
Repository: explodinggradients/ragas
Issue: `default_transforms()` Missing cosine_similarity Transform for Docs with Token Count 101–500
Root cause: Validation Gap

At:  
[[default.py#L157](https://github.com/explodinggradients/ragas/blob/2bc29a2b8358ddb6b167fdf7ab0518ad9371463c/src/ragas/testset/transforms/default.py#L157)](https://github.com/explodinggradients/ragas/blob/2bc29a2b8358ddb6b167fdf7ab0518ad9371463c/src/ragas/testset/transforms/default.py#L157),  

you forgot to add the `cosine_sim_builder` transform to the list of default transforms. While you instantiated the `cosine_sim_builder` object, you did not include it in `Parallel(cosine_sim_builder, ner_overlap_sim)`, as you did in:  

[[default.py#L126](https://github.com/explodinggradients/ragas/blob/2bc29a2b8358ddb6b167fdf7ab0518ad9371463c/src/ragas/testset/transforms/default.py#L126C13-L126C59)](https://github.com/explodinggradients/ragas/blob/2bc29a2b8358ddb6b167fdf7ab0518ad9371463c/src/ragas/testset/transforms/default.py#L126C13-L126C59).  

This omission might impact the number of relationships created in the knowledge graph.  

**Ragas version:** `ragas==0.2.13`  
**Python version:** `Python 3.10.12`  


Can you share the problem you are facing?


The `default_ transforms`  function defined at  `src/ragas/testset/transforms/default.py` has a problem with handling transforms for documents with 101-500 tokens.

The code divides the `transforms` configurations based on the document's token count.  Several transforms are instantiated when the ["101-500" token count bins the first quartile (Q1](https://github.com/explodinggradients/ragas/blob/2bc29a2b8358ddb6b167fdf7ab0518ad9371463c/src/ragas/testset/transforms/default.py#L128), among them the `cosine_sim_builder`.  While `cosine_sim_builder` is correctly **instantiated** (line 139), it's then **not included** in the list of transforms that are actually returned (line 153).

It appears that `cosine_sim_builder` was likely unintentionally omitted from the returned transforms list.  The intended behavior should probably mirror how `ner_overlap_sim` is handled (line 120), where `cosine_sim_builder` is instantiated and added to the returned list.  The current code effectively instantiates `cosine_sim_builder` but then discards it. This omission might impact the number of relationships created in the knowledge graph. 

Hoping this can help you understand the problem I'm facing!

Accha got this, 
In `src/ragas/testset/transforms/default.py` 
Instead of line 157 to be  `ner_overlap_sim`, 
this should be `Parallel(cosine_sim_builder, ner_overlap_sim)`.

This makes sense. 

@rgrizzo-linksmt I think you should raise a PR for this.
Repository: explodinggradients/ragas
Issue: Can I  generate multi-hop reference_contexts Only? (without the question and reference)
GitHub labels: question

module-testsetgen

**Your Question**
I was wondering if the API support only generating the reference_contexts and not the whole flow?
In addition, where can I find the actual links the context is based on?

**Code Examples**
Currently I am doing:
 ```
query_distribution = [(MultiHopSpecificQuerySynthesizer(llm=self.generator_llm),1)]
            generator = TestsetGenerator(llm=self.generator_llm, embedding_model=self.generator_embeddings)
            dataset = generator.generate_with_langchain_docs(docs, testset_size= self.n_generations, query_distribution=query_distribution)
            chunks = []
            for sample in dataset.samples:
                # Check if the sample has any reference contexts.
                if sample.eval_sample.reference_contexts:
                    for context in sample.eval_sample.reference_contexts:
                        # Create a simple chunk object with page_content and metadata.
                        chunk = {
                                 "page_content": context,
                                 "metadata": {"Links": "TODO"}
                                }
                        chunks.append(chunk)

```
Thanks in advance.



Hi @GraderYuval,

The API doesn't support only `reference_contexts` generation; the testset sample is created by the `_generate_sample` function, which returns:

```py
return SingleTurnSample(
    user_input=response.query,
    reference=response.answer,
    reference_contexts=reference_context,
)
```

Could you clarify what you mean by "the actual links the context is based on"? 

For context, the current testset generation flow works like this: the knowledge graph (KG) is constructed, clusters are identified from the KG, scenarios are generated based on those clusters, and then the testset sample is created.

Hi, thank you very much for the reply. 
Regarding the "actual links...", I meant that if I use URLs to obtain the context, I want to know which specific URL each context hop came from. For example, if we are crawling a webpage and creating a multi-hop context from it, can I retrieve from the dataloader ('docs') the actual metadata of the URLs from which the context was extracted?